rtcApp.py in ISCE v2.3.3

Added by Helen Baldwin 4 months ago

Hi,

I'm trying out the new release of ISCE, with updated rtcApp.py (so cool!), and have a few questions. I read the other posts about rtcApp and wasn't able to find these answers, my apologies if I missed something!

My setup:
  1. successful scons install of ISCE v2.3.3 from GitHub
  2. example dataset is S1A_IW_GRDH_1SDV_20181221T225104_20181221T225129_025130_02C664_B46C (I acquired the same image as is as shown in the example xml in the GitHub ReadMe) [final rtcApp.xml attached]
  3. OS: Ubuntu 18.04.4 LTS
Questions:
  1. Although isce installed 'correctly' (isce imports in python), rtcApp.py doesn't exist in the install/isce/applications folder but only in the originally downloaded folder isce2-2.3.3/applications/rtcApp.py. Is this right or does this mean that my scons install was actually unsuccessful? If this is right, should I update my PATHs in a specific way?
  2. When I ran the command "dir/isce2-2.3.3/applications/rtcApp.py dir/rtcApp.xml", I got the error: "ModuleNotFoundError: No module named 'isceobj.RtcProc.runNormalize.'" While the file runNormalize.py doesn't exist in the installation folder dir/install/isce/components/isceobj/RtcProc, it does exist in the downloaded folder dir/isce2-2.3.3/components/isceobj/RtcProc/runNormalize.py. I copied and pasted this file into the installation folder above. Is this an appropriate work around or am I breaking things further? [Screenshot_ModuleNotFound attached]
  3. With the 'fix' from above, and after deleting the 'orbit' line from the xml, I tested running the command "dir/isce2-2.3.3/applications/rtcApp.py dir/rtcApp.xml" again. This time I got two pieces of output that troubled me [Output_NoneType-and-IndexErrors attached]:
    • "Could not extract polarization VV, Generated error: argument of type 'NoneType' is not iterable"
    • in the runPreprocessor step, "IndexError: list index out of range"

Please let me know if there is additional information that I can provide. Thank you in advance for your help!

Best,
Helen


Replies (51)

RE: rtcApp.py in ISCE v2.3.3 - Added by Piyush Agram 4 months ago

Looks like all the features are there but they are not installed as it was not included in the SConscript files in their corresponding folders.
You have polarization instead of polarizations as expected by rtcApp.py in your input file - https://github.com/isce-framework/isce2/blob/74f6473d4a164b899ebaef22eeb8ced5f73c644f/applications/rtcApp.py#L157

Happy to take PRs.

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 4 months ago

Thank you, Piyush!

I implemented those changes (pull request submitted), and now rtcApp.py is being properly included in the isce v2.3.3 installation and the polarization error mentioned is gone. I now have a new error: the attribute corresponding to the key “polarizations” is not present in the object “<class ‘isceobj.sensor.grd.sentinel1.sentinel1” (screenshot attached). I took a look at the isceobj/sensor/grd/sentinel1 file, but I'm not exactly sure what I'm looking for. Could it be a mistake in the formatting of the polarizations as [VV, VH] in my xml (also attached)?

Please let me know if there's any additional information I can provide.

Best,
Helen

RE: rtcApp.py in ISCE v2.3.3 - Added by Piyush Agram 4 months ago

Oh ... I believe the problem is because the polarizations should be outside the <master> component. runPreprocessor loops over these by setting a single polarization in the reader and parses the imagery.

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 4 months ago

Hi Piyush,

Do you mean that the polarizations should be in another xml file or within the same xml file under a new component or something else? I've tried a couple combinations and they haven't worked (errors like IndexError: list index out of range, ElementTree.ParseError: not well-formed (invalid token)) and reread the Component Configuration section of the ISCE2 Readme, but I'm not sure how to apply it to rtcApp.py I've attached an example xml from one of my tries, this one threw and IndexError: list index out of range during the runPreprocessor step, also the polarizations were unable to be extracted, ("argument of type 'NoneType' is not iterable" as before), so I'm definitely missing something on this xml.

Thank you for your help!

Best,
Helen

rtcApp.xml Magnifier (588 Bytes)

RE: rtcApp.py in ISCE v2.3.3 - Added by Piyush Agram 4 months ago

Outside master but within rtcApp. At the same level as posting.

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 4 months ago

I tried putting the "polarizations" before and after the master component, but still within rtcApp component. I'm getting "Could not extract polarization VV" then "TypeError: argument of type 'NoneType' is not iterable" and finally, " "Exception: User requested polarization VV but not found in the input data." I've attached a copy of one of the versions of the xml and screenshot of the error it produced.

RE: rtcApp.py in ISCE v2.3.3 - Added by Piyush Agram 4 months ago

The error clearly says that no orbit files have been provided. It fails to identify if orbit file is POEORB / RESORB. Provide an orbit folder / file.

RE: rtcApp.py in ISCE v2.3.3 - Added by Simon Kraatz 3 months ago

Hi Helen!

The error message about polarization VV not found can be misleading. I suggest you double check that all the files (orbit file and the sentinel grd files) are in the correct location. Don't worry, I've gotten this error myself sometimes, but it had to do with ISCE not finding the file it was looking for. And as Piyush pointed out there appears to besome mismatch between 'polarization' vs 'polarizations', I'll have to look into that.

Best, Simon

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 3 months ago

Thank you, Simon and Piyush,

I had removed the orbit property after a previous run when the message "Exception('No suitable orbit file found. If you want to process anyway - unset the orbitdir parameter" was produced and confused that message to mean I didn't need it. I've set it up (hopefully correctly this time) and will try again. I'll try with singular "polarization" in the xml as well. Thanks!

RE: rtcApp.py in ISCE v2.3.3 - Added by Simon Kraatz 3 months ago

Hi Helen,

I just set it up from scratch and it worked right away. I'm using an older version of ISCE (2.3.1), but I think it the workflow should work just as well in 2.3.3. I attached some images of my folder structure and settings.

Some pointers:

*Put the Sentinel-1 data file as downloaded from the Alaska Satellite Facility API tool or Copernicus in the working directory, as .zip
*Obtain orbit files. I used the precise orbit files, but the estimated ones would also be fine. You can get them from https://qc.sentinel1.eo.esa.int/aux_poeorb/. You only needed the orbit file encompassing the dates/times of the input file, for me it is: S1B_OPER_AUX_POEORB_OPOD_20190110T110700_V20181220T225942_20181222T005942.EOF. It doesn't hurt to have the adjacent ones. ISCE will find the appropriate file.
*Then, run "rtcApp.py rtcApp_sentinel.xml"
*The rtc gamma0 file is for example gamma_VH.10alks_10rlks.vrt. You can just drag the vrt into QGIS or ArcGIS to view it. Likewise, you can drag the beta_VH.img.vrt into either program, to compare how the rtc steps modified the dataset. The gamma0 file is not geocoded yet - I just haven't added the geocoding step to the workflow. I'm not sure when it will be done, but the geocoding is straightforward and can readily be done outside of this workflow. For example, you may use geolocation arrays: https://github.com/isce-framework/isce2-docs/blob/master/Notebooks/GDAL/03_RasterProjection.ipynb

Hope this helps!

workdir.png (129.3 kB)

rtcApp_sentinel.xml.png (65.8 kB)

output_dir.png (189.9 kB)

orbit_files.png (100.1 kB)

basedir.png (54.5 kB)

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 3 months ago

Hi Simon,

That's fantastic!
Thank you so much, this is extremely useful.

For some reason, my run has been getting 'stuck' on the earthdata step (screenshot attached), after 'runPreprocessor'. This seems strange, I had run stripmapApp a couple of days ago and DEMs were successfully downloaded. I tried rerunning a couple times throughout the day, thinking that maybe it was my internet connection. Then I double checked my .netrc, signed into earthdata to make sure I had the username/password correct, and that I had allowed the specified LP DAAC applications, and reran with the same result. I've let it run for several hours, and it hasn't moved beyond this step. Do you have any ideas of what this could be? Should I try reinstalling isce? I'm running on Ubuntu 18.04.4 LTS. I've attached screenshots of the outputs I've been able to produce, so at least I think my data is being read in correctly :) I just want to make sure I'm not missing something.

Thank you for all of your help!

Best,
Helen

RE: rtcApp.py in ISCE v2.3.3 - Added by Piyush Agram 3 months ago

I would check the .netrc setup with dem.py

dem.py -b 40 41 -76 -74 -s 1 -c -f

You can change the snwe limits. If this doesnt work there is something else wrong with the network setup.

Piyush

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 3 months ago

I'll try that, thanks!

RE: rtcApp.py in ISCE v2.3.3 - Added by Simon Kraatz 3 months ago

Hi Helen,

I don't think you need to reinstall anything. Sometimes it is possible the server is down, but I was able to download the DEM just fine today. Piyush's suggestion is a good check. You also have an url printed to command line, but it is cut off from the screenshot. You can copy/paste it (or most of it except the last dirs) in a browser to see if that gets you to a website where you can select the files manually. If that works, click a DEM file and see if it downloads. You could have a firewall, or maybe one of the internet hops connecting you to that server is out and you just cant reach it.

Simon

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 3 months ago

I was able to download the SRTMs with dem.py (following the link printed to the command line also worked). Does this mean my .netrc is okay? Should I download SRTM separately and then specify in the xml?

RE: rtcApp.py in ISCE v2.3.3 - Added by Simon Kraatz 3 months ago

Helen,

It is important to note that SRTM only is available for -60 S to +60 N latitudes. If any coordinate is outside of those bounds, it can't find the proper SRTM file. You will then need to use a different approach for obtaining the DEM. The workflow as provided assumes the user is working within a +-60 latitude range. I can help downloading/preparing the DEM for input into ISCE. But I'm not sure if there is an easy way to modify the rtcApp.py workflow for this case, maybe Piyush has a good workaround.

Simon

RE: rtcApp.py in ISCE v2.3.3 - Added by Piyush Agram 3 months ago

if dem.py works, other workflows should also work as they use the same mechanism
You can always specify input DEM as an option. Else the workflow downloads it based on the dataset being used.

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 3 months ago

Tried again and it completed! Thank you both so much! I will work on the geocoding step next.
For posterity, I've attached a screenshot of my final xml, and output gamma_VH.10alks_10rlks.img.

Thanks again!

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 3 months ago

Hi Simon,

Your resource for geocoding was very helpful. I used the geolocation arrays, geocoding a full image portion of the notebook, specifically the "geocodeUsingGdalWarp" function. I’ve attached a screenshot for a sanity check, and links to the geocoded geotiff and python script. I have a few follow up questions.
  • the original example used lat.rdr.full.vrt and lon.rdr.full.vrt. Since these weren't created in rtcApp.py I used lat.rdr.vrt and lon.rdr.vrt instead. Is it okay to make this change?
  • I didn’t create an xml file. I imagine that this would impact the output .vrt and not the output geotiff, but wanted to confirm.
  • When calling the geocodeUsingGdalWarp the spacing is specified as "spacing=[0.0005, 0.0005]". What's this referring to? Would it need to be changed for Sentinel, since the example is ALOS?
We’re planning to scale up our RTC processing using ISCE. I have a few questions related to that as well.
  • Do you have a timeline for incorporating the geocoding step into the workflow or a suggestion of where I should start with doing that on our side?
  • Does the current rtcApp.py allow for batch processing?
  • It’s been taking ~3 hours for one scene to process on my VM (2 processors, 10794MB memory), do you have any information on speed/efficiency on other setups?

Link to geocoded tiff: https://drive.google.com/open?id=1JTNDtQZsMTCAB6SYY73USx2xCvbPY7dJ

Thanks again for all of your help!

Best,
Helen

RE: rtcApp.py in ISCE v2.3.3 - Added by Simon Kraatz 3 months ago

Hi Helen,

As to your questions:

  • the original example used lat.rdr.full.vrt and lon.rdr.full.vrt. Since these weren't created in rtcApp.py I used lat.rdr.vrt and lon.rdr.vrt instead. Is it okay to make this change?
    That's fine.
  • I didn’t create an xml file. I imagine that this would impact the output .vrt and not the output geotiff, but wanted to confirm
    The xml is not needed to make a geotiff, just the .vrt is enough. As to if the xml impacts the vrt, I don't know.
  • When calling the geocodeUsingGdalWarp the spacing is specified as "spacing=[0.0005, 0.0005]". What's this referring to? Would it need to be changed for Sentinel, since the example is ALOS?
    This spacing is in latitude longitude values. Since GDAL is using the ISCE geolocation arrays, it would be the same for all the platforms. But because the output projection is lat/lon (EPSG:4326), it needs to be given in lat/lon units.
    But my workflow update is only for UTM, so it needs to be given in meters.
  • Do you have a timeline for incorporating the geocoding step into the workflow or a suggestion of where I should start with doing that on our side?
    You can already try it out if you want, just using the code in my pull request. I updated the ISCE rtc workflow (but only tested it with Sentinel1), to see the differences from before, see here: https://github.com/isce-framework/isce2/pull/128/files A copy of the description in the pull request:
    Added the geocoding step. It just uses gdal and the geolocation arrays as per the data in the geometry folder. This step comes with a few additional required input parameters.
    Quick explanation...
    "posting" : just for multilooking, integer multiple of spacing. Putting a large value speeds up processing if highest resolutions are not needed.
    "epsg id": need to put epsg code for utm. I only set up geocoding for utm.
    "geocode spacing": this is the input parameter for gdal, and sets the x and y spacing. It may be desireable to have this different from posting value, so I left this option.
    "geocode interpolation method": any of the gdal supported ones such as 'near' and 'bilinear'
    By the way, these are all required inputs, or it won't work.
  • Does the current rtcApp.py allow for batch processing?
    Sorry I haven't tried; not sure.
  • It’s been taking ~3 hours for one scene to process on my VM (2 processors, 10794MB memory), do you have any information on speed/efficiency on other setups?
    I also remember that example running really slow...it will speed up if you change the spacing in the example to larger values. Anyway, rtcApp.py finished in about 2 minutes on my 6 year old 6-core system, using the settings in the updated Readme.md settings (see my pull request; in particular, posting and geocode spacing set to 100).

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 3 months ago

Thanks, Simon!

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 3 months ago

Hi Simon & Piyush,

Thanks so much for the updates to the rtcApp.py on GitHub.

I'm trying to put together a "simple" workflow for my team, so that we can easily compare differences in steps with other processing software like SNAP. Is there a particular doc that I should be referring to? I've been looking at the scripts, the rtcApp.py --help output and the isce2 docs GitHub. There's a lot of info there, so I may have missed the answers to my questions there. A few questions that I've run into so far are:
  • In the rtcApp.py output printed on the terminal, it says that “no noise correction requested,” is there a way to request this?
  • Is there a calibration step?
  • It looks like there's a masking step happening, which step is this in?
And then if you have time to provide any input, these are the current steps:
  • Extract orbit, image
  • No noise correction requested
  • Preprocessor for both frames → extract slant range
  • Verify DEM → download if not provided
  • Topo → map radar to geo coordinates, DEM interpolation “Biquintic”, orbit interpolation “legendre”
  • Multilook (based on ‘posting’ input by user)
  • Normalize → create gamm_0
  • Geocode

Thanks again!

Best,
Helen

RE: rtcApp.py in ISCE v2.3.3 - Added by Simon Kraatz 3 months ago

Hi Helen, I'll answer what I can:

1) I'm also not sure about how to set the flag for noise correction, or whether that's a supported option currently.
2) There is a calibration factor applied in the extractImage part of the S1 GRD reader (see Piyush's comment in http://earthdef.caltech.edu/boards/4/topics/3093?r=3168#message-3168), to make the data beta0.
3) I think you refer to the shadowmask, that is an output from the topy.py step, and used as input for the RTC to avoid applying RTC in shadowed areas
About your steps, the multilook just after the DEM step and ahead of topo.py. That way topo.py can do calculations on the smaller, multilooked dataset to speed things up and reduce space requirements. You can see the order of the steps in rtcApp.py.

Best, Simon

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 3 months ago

Thanks, Simon!

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin about 1 month ago

Hi team,

Thank you so much for the updates to the rtcApp SConscript that allows for rtcApp install, and the addition of geocoding. Will these updates be made to the v2.3.3 release package and/or the Anaconda conda-forge packages?
Thanks!

Best,
Helen

1 2 3 Next » (1-25/51)