rtcApp.py in ISCE v2.3.3

Added by Helen Baldwin 4 months ago

Hi,

I'm trying out the new release of ISCE, with updated rtcApp.py (so cool!), and have a few questions. I read the other posts about rtcApp and wasn't able to find these answers, my apologies if I missed something!

My setup:
  1. successful scons install of ISCE v2.3.3 from GitHub
  2. example dataset is S1A_IW_GRDH_1SDV_20181221T225104_20181221T225129_025130_02C664_B46C (I acquired the same image as is as shown in the example xml in the GitHub ReadMe) [final rtcApp.xml attached]
  3. OS: Ubuntu 18.04.4 LTS
Questions:
  1. Although isce installed 'correctly' (isce imports in python), rtcApp.py doesn't exist in the install/isce/applications folder but only in the originally downloaded folder isce2-2.3.3/applications/rtcApp.py. Is this right or does this mean that my scons install was actually unsuccessful? If this is right, should I update my PATHs in a specific way?
  2. When I ran the command "dir/isce2-2.3.3/applications/rtcApp.py dir/rtcApp.xml", I got the error: "ModuleNotFoundError: No module named 'isceobj.RtcProc.runNormalize.'" While the file runNormalize.py doesn't exist in the installation folder dir/install/isce/components/isceobj/RtcProc, it does exist in the downloaded folder dir/isce2-2.3.3/components/isceobj/RtcProc/runNormalize.py. I copied and pasted this file into the installation folder above. Is this an appropriate work around or am I breaking things further? [Screenshot_ModuleNotFound attached]
  3. With the 'fix' from above, and after deleting the 'orbit' line from the xml, I tested running the command "dir/isce2-2.3.3/applications/rtcApp.py dir/rtcApp.xml" again. This time I got two pieces of output that troubled me [Output_NoneType-and-IndexErrors attached]:
    • "Could not extract polarization VV, Generated error: argument of type 'NoneType' is not iterable"
    • in the runPreprocessor step, "IndexError: list index out of range"

Please let me know if there is additional information that I can provide. Thank you in advance for your help!

Best,
Helen


Replies (51)

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin about 1 month ago

Hi team,

Will the updates to rtcApp.py (pull request #120 https://github.com/isce-framework/isce2/pull/128/files - which includes addition of geocoding and SConscript edits that allow for rtcApp install) be included in a new release package and conda-forge package? It would be great if the RTC functionality were more accessible to those who don't need to edit the code. Thanks!

RE: rtcApp.py in ISCE v2.3.3 - Added by Piyush Agram about 1 month ago

Hi Helen,
conda packages can only be updated with release versions and not the live github version. Since, these changes are purely at the python level copying over the folder from the live repo can be easily baked into any build script/ dockerfile.

When a new release is made by the JPL team, we can update the conda package.

Piyush

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin about 1 month ago

Ok, thanks, Piyush.

Although I was able to install ISCE and make the required edits for RTC to install on my Ubuntu VM, I've been having trouble getting this ISCE installed on CentOS.

RE: rtcApp.py in ISCE v2.3.3 - Added by Piyush Agram about 1 month ago

Centos unfortunately tends to be atleast 2 years behind. If you use Centos8, it will be lot better than Centos7 which still enforces backward compatibility and its compilers dont work well with software thats not from its official repos. If you are still forced to work with Centos 7, use conda compilers instead of the ones from the system.

From the next release on, hopefully you will be able to build isce with cmake which seems to have lot fewer issues than scons.

Piyush

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin about 1 month ago

Awesome, thanks!

Best,
Helen

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 28 days ago

Hi Piyush and team,

I ended up going back to Ubuntu (rather than CentOS) and was able to install ISCE.

I'm trying to run rtcApp.py again, this time using v2.3.3 release updated only with the #120 pull request.

I'm getting a TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType (full error in screenshot attached). Does this seem like an error related to my input rtcApp.xml (screenshot attached), a typo in the #120 PR update I made, or potentially something else? I've looked at runPreprocessor.py but still not quite sure what path is being referred to that's incorrect.

I'll keep working on this, but any guidance would be appreciated. Thanks!

Best,
Helen

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 27 days ago

I realized that the line throwing this error (29 in runPreprocessor.py) has been changed in the 'live' isce2
Screenshot of change attached, linked here: https://github.com/isce-framework/isce2/commit/23f2ffaa1eeac93bb039f7259817810c4c3c0b12#diff-2ba78cef2f8b03e21a4b71d22f64c8ec

Not sure if this has anything to do with it as I was able to run rtcApp.py previously by making only the SConscript update to the v2.3.3 release.

RE: rtcApp.py in ISCE v2.3.3 - Added by Piyush Agram 27 days ago

Looks like you have two entries for orbits instead of one for output.
That line change shouldnt matter.

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 27 days ago

Good catch! Fixed that.

Good to know about that line change. Thanks!

Probably something else in my inputs then. I'll check them again, looks like I'm throwing the same error even after the xml updated.

RE: rtcApp.py in ISCE v2.3.3 - Added by Piyush Agram 27 days ago

The xml file still looks the same and the error says "output" was not provided.

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 23 days ago

Yikes! You are right. Fixed it properly and now seems to be running. Thanks!

RE: rtcApp.py in ISCE v2.3.3 - Added by Heming Liao 23 days ago

Hi Helen and Piyush,

I am trying to do some time series amplitude analysis with the rtcApp.py module but got an error. I am wondering whether you have any idea about this?

(base) heming@heming-Latitude-7400:~$ rtcApp.py
This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
Traceback (most recent call last):
File "/home/heming/anaconda3/lib/python3.7/site-packages/isce/applications/rtcApp.py", line 34, in <module>
from isce import logging
ImportError: cannot import name 'logging' from 'isce' (/home/heming/anaconda3/lib/python3.7/site-packages/isce/__init__.py)

I installed isce with conda. I downloaded the rtcApp.py file from git and then placed it in the conda isce applications folder (~/anaconda3/lib/python3.7/site-packages/isce/applications/rtcApp.py).

This might be an easy error, but I haven't got an idea how to fix. I would appreciate if you could shed some light on this.

Heming

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 23 days ago

Hi Heming,

Piyush is the expert, but I think your issue may be that updating the rtcApp.py after using conda install isn't enough to give you the full RTC package installed.

Here are the instructions/notes for how I installed: https://docs.google.com/document/d/1PoanwXrAU4t6a4VKS20o7GAgjZWEhyPKvoWaaTgSpC8/edit?usp=sharing

Basically, I updated the v2.3.2 stable release with needed updates to SConscript files, etc (These are from a pull request after the stable version was released: https://github.com/isce-framework/isce2/pull/128/files) Updating these files allows you to do geocoding and ensures that the rtcApp package is completely installed). Then I installed ISCE using the scons install instructions from the README (rather than conda).

Best,
Helen

RE: rtcApp.py in ISCE v2.3.3 - Added by Heming Liao 23 days ago

Hi Helen,

Thank you very much for your quick reply. I also feel the problem might be related with my software installation. I will wait for a little bit to see whether there are other easier way to twist it so that I can still use the rtcApp.py with my current conda installation, instead of uninstall the conda isce and install it with SConscript manually. Anywary, thank you for sharing your notes.

Best,
Heming

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 22 days ago

Hi Piyush and team,

Two questions:

1. I'm trying to use a different DEM instead of the automatically downloaded SRTM in rtcApp.py and am getting an XML parse error (screenshot attached). I believe this is related to the xml file associated with the DEM, as I get the same error with mdx.py and the xml for this DEM (produced automatically by Gamma) has a very different format from the xml file associated the SRTM. I apologize if I missed this elsewhere in the forum, but is there guidance on how to create an xml for your own DEM?

2. When I run rtcApp.py with the automatically downloaded SRTM, I am able to run through rtcApp.py except for the geocoding step. The message "Too many points failed to transform, Unable to compute source region for output window" appears and the geocoded project isn't created. Do you have any suggestions for why this might be?

DEM_mismatchedtagError_7-14-2020.JPG - XML Parse error, probably related to DEM xml file (119.1 kB)

rtcAppxml_usingdifferentDEM_7-14-2020.JPG - rtcApp.xml for using a different DEM (67.1 kB)

GeocodeError_7-14-2020.JPG - Geocoding error (157.7 kB)

RE: rtcApp.py in ISCE v2.3.3 - Added by Piyush Agram 21 days ago

ISCE uses a flat binary file for DEM. Write out your DEM as an ENVI file using gdal_translate. Then run gdal2isce_xml.py on the ENVI file to generate the xml file for it.
Note that a major PR was just merged and all references to "master" need to be replaced with "reference".

We will be making a release soon with these changes.

Piyush

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 21 days ago

Thanks, Piyush!

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 16 days ago

Hi Piyush,

Unfortunately, I don't have access to Envi. QGIS has a GDAL translate tool, which I used to create dem.vrt. However, throws the "Attribute Error: 'Nonetype' object has no attribute 'RasterXSize". Earlier the error message says that "...vrt not recognised as a supported file format".This dem.vrt file is readable in QGIS and ArcMap. Since you mentioned "ENVI file" I also tried using ArcMap to make a .dat file, however gdal2isce_xml.py failed for the dem.dat with the same error. I used the -h to see if there was a list of supported file types, but there aren't any. Am I using the right file type? Is there a work around not reliant on Envi software access? Thanks!

Dem-dat-nonetypeError_7-20-2020.JPG - Can't open .dat (error the same for .vrt file) (105 kB)

RE: rtcApp.py in ISCE v2.3.3 - Added by Eric Fielding 16 days ago

Hi Helen,

You don't need the ENVI software, you only need to save the DEM in ENVI format. In QGIS, they call this "ENVI .hdr labelled" and you can export directly to that format. I don't think you can use the ".vrt" file exported from QGIS or GDAL in ISCE.

You can make the DEM file have a ".dem" extension when you export it, but I am not sure that is required.

++Eric

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 16 days ago

Thanks, Eric!

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 9 days ago

Hi Eric,

Is "ENVI .hdr labelled" in a specific tool in QGIS? I've googled and haven't been able to find which tool this would be. So far I had been using gdal_translate tool within QGIS to convert from geotiff to other formats.

I see that the automatically downloaded SRTM DEM is in the .dem and .vrt formats, but gdal2isce_xml.py failed for .vrt (and you mention that this isn't usable) and .dem has a "size on disk and size computed from metadata for file do not match" error during rtcApp.py

I thought I was on the right track with using dem as.bip file, since the default interleaving scheme for gdal2isce_xml.py is BIP, but while it doesn't throw an error it gets "stuck" on the "verifyDEM" step (screenshot attached).

Sorry if that's a bit jumbled, thanks for any suggestions!

Best,
Helen

gdal2isce_xml_7-27-2020.JPG - gdal2isce_xml.py default interleaving scheme is BIP (143.7 kB)

dem_sizeondisknotmatch_7-27-2020.JPG - .dem size on disk and metadata don't match (42 kB)

WithOwnDEM_testbipandothers_keepgettingstuckhere_7-26-2020.JPG - rtcApp.py gets 'stuck' on verifyDEM step using .bip file (72.8 kB)

RE: rtcApp.py in ISCE v2.3.3 - Added by Eric Fielding 9 days ago

Hi Helen,

In QGIS 3.10 (I don't have QGIS 2.x anymore, but it was similar), I can right-click on a layer and chose "Export->Save As". This brings up the "Save Raster Layer As" dialog and you can chose a number of output formats including "ENVI .hdr labeled". If you set your output file name to "mytopo.dem", then that should create the binary image file (mytopo.dem) in the simple ENVI format (BIP or BIL is the same for only one band) that ISCE can read and an ENVI metadata file (mytopo.dem.hdr). You also could do this directly with "gdal_translate -of ENVI mytopo.tif mytopo.dem".

Then running the "gdal2isce_xml.py" script on the ENVI-format mytopo.dem file will generate the .xml and .vrt metadata files that ISCE requires from the mytopo.dem.hdr file.

One thing to remember is that ".vrt" is not a file format. It is a virtual file that contains information about how to read the actual data file (or multiple data files). ISCE uses a specific type of .vrt file and expects the actual binary image to be in the simple ENVI format.

++Eric

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 8 days ago

Hi Eric,

Thanks so much for the step by step instructions. I was able to create the "mytopo.dem" with right click and export (rather than using the gdal_translate tool). Using gdal2isce_xml.py worked as well.

However, when running through rtcApp.py, I am still getting stuck on the step run VerifyDEM where it is sampling Geoid at grid points (no error is thrown, just sit on this step for much longer than rtcApp.py took to process when using the automatically downloaded DEM and expected outputs not created). Do you have any suggestions on why this might be? If there are additional details I can provide, please let me know.

Thanks again for all of your help!

Best,
Helen

SamplingGeoid_7-28-2020.JPG - Stuck at this point in rtcApp.py (66.6 kB)

RE: rtcApp.py in ISCE v2.3.3 - Added by Eric Fielding 8 days ago

Helen,

What is the map projection (CRS) of your DEM? It should be equally spaced in latitude and longitude (CRS "EPSG:4326 - WGS 84 - Geographic") for ISCE. If the DEM is in another projection such as a UTM projection and you already have your DEM loaded in QGIS, you can use Raster->Projection->Warp(reproject) and choose the output CRS as EPSG:4326. This can also be done on the command line with "gdalwarp".

++Eric

RE: rtcApp.py in ISCE v2.3.3 - Added by Helen Baldwin 7 days ago

Hi Eric,

I think that worked! Still running, but I got past that particular step. Thanks so much!

Best,
Helen

« Previous 1 2 3 Next » (26-50/51)