Earthdef::ISCEForum

Can you send us a SSARA command to download the dataset that you are working with?

Piyush

Not sure about the SSARA command, since the data I was using is in south america (not in ASF, or supersite). However, I tried for a pair that is freely accessible through the supersite (http://eo-virtual-archive4.esa.int):

https://eo-virtual-archive4.esa.int/supersites/ASA_IMS_1PNDPA20100430_141134_000000162089_00053_42692_1181.N1
https://eo-virtual-archive4.esa.int/supersites/ASA_IMS_1PNUPA20100326_141137_000000162088_00053_42191_0673.N1

ISCE runs fine with this data. But if I download the same frames via their new "On-The-Fly" service with Eolisa, I get the following files, which produce the same error I mentioned in the previous post:

ASA_IMS_1PNESA20100326_141135_000000182088_00053_42191_0000.N1
ASA_IMS_1PNESA20100430_141132_000000182089_00053_42692_0000.N1

So it seems to be an issue with a change in file format. I didn't come across anything obvious on their website:
https://earth.esa.int/web/guest/missions/esa-operational-eo-missions/envisat/news/-/article/new-service-open-to-asar-data-users-esa-s-envisat-asar-image-mode-archive-released-for-on-the-fly-data-download

Cheers,
Scott

Can you send us some of the new OTF data by large file transfer for testing? I tried the reader with another pair of old SLCs and everything seems to work fine.

Piyush

Ok, I've sent a link via email to the ASA_IMS_1PNESA* files mentioned in the previous post. I am able to open both version in the latest version of SNAP. Posting a screenshot comparing metadata in case it's helpful. I'm guessing there is something different with the header structure so readGeoParams() in EnviSAT_SLC.py isn't getting the correct value for "timeToFirstSample"...

http://envisat.esa.int/handbooks/asar/CNTR6-2-9.html

Looks like processing parameter record length and the number of geoocoding records have changed between versions.
We could update the readers to start making all this flexible. The easier solution might be to just use pyepr :

https://github.com/avalentino/pyepr
https://github.com/avalentino/pyepr/blob/master/doc/interactive_use.txt

This had no problem dealing with both sets of data. Using this would make life easy as all the fields are read in and are easily available within python but would also introduce a dependency.

Piyush

For now, you can change procParamLength to 10069 and geoParamLength to 521*13 to get this to work.
You can get these numbers by using pyepr on the product files.

Piyush

Make the following changes to isceobj/Sensor/Envisat_SLC.py

@@ -544,13 +544,14 @@
         BaseEnvisatFile.__init__(self)
         self.fileName = fileName
         self.sqLength = 170
-        self.procParamLength = 2009
+        self.procParamLength = None
         self.doppParamLength = 55
         self.chirpParamLength = 1483
-        self.geoParamLength = 521 * 12
+        self.geoParamLength = None

     def parse(self):

+        import pprint
         imageryDict = {}
         try:
             self.fp = open(self.fileName, 'rb')
@@ -561,6 +562,20 @@

         self.readMPH()
         self.readSPH()
+
+        ####Handling software version change in 6.02
+        ver = float(self.mph['SOFTWARE_VER'].strip()[-4:])
+
+        if ver < 6.02:
+            print('Old ESA Software version: ', ver)
+            self.procParamLength = 2009
+            self.geoParamLength = 521*12
+        else:
+            print('New ESA Software version: ', ver)
+            self.procParamLength = 10069
+            self.geoParamLength = 521*13
+
+
         procDict = self.readProcParams()
         doppDict = self.readDopplerParams()
         geoDict = self.readGeoParams()

Piyush

Excellent! Seems to be working. Thanks for the fix Piyush.

I get a crash with the same error that Scott reported above. I am processing ERS OTF data in SLC format, downloaded today, using isce2.2.0 and the ERS_Envisat_SLC sensor with stripmapApp.

2018-12-20 17:54:08,585 - isce.make_raw - INFO - Calculating Squint Angle
Traceback (most recent call last):  
 File "/home/gareth/pkgs/isce/isce/applications/stripmapApp.py", line 959, in <module>
  status = insar.run()
 File "/home/gareth/pkgs/isce/isce/components/iscesys/Component/Application.py", line 129, in run
  exitStatus = self._processSteps() 
 File "/home/gareth/pkgs/isce/isce/components/iscesys/Component/Application.py", line 392, in _processSteps
  result = func(*pargs, **kwargs) 
 File "/home/gareth/pkgs/isce/isce/components/isceobj/StripmapProc/Factories.py", line 46, in __call__
  return self.method(self.other, *args, **kwargs)
 File "/home/gareth/pkgs/isce/isce/components/isceobj/StripmapProc/runPreprocessor.py", line 57, in runPreprocessor
  master = make_raw(sensor, masterdop)
 File "/home/gareth/pkgs/isce/isce/components/isceobj/StripmapProc/runPreprocessor.py", line 197, in make_raw
  objMakeRaw(sensor=sensor, doppler=doppler) 
 File "/home/gareth/pkgs/isce/isce/components/iscesys/Component/Component.py", line 166, in __call__
  return getattr(self, self.__class__.__name__.lower())(*args)
 File "/home/gareth/pkgs/isce-2.2.0/isce/applications/make_raw.py", line 330, in make_raw
  self.calculateSquint()
 File "/home/gareth/pkgs/isce-2.2.0/isce/applications/make_raw.py", line 195, in calculateSquint
  "larger than starting Range (%s)") % (h-z, startingRange))
ValueError: Spacecraft Height - Terrain Height  (784723.6919871317) larger than starting Range (0.0)

When I looked at isceobj/Sensor/ERS_Envisat_SLC.py I note that the code changes suggested above are present in the script, but commented out. Even when I uncomment them it doesn't make any difference. Did they change the file definition again or something?

I'm not sure how to use pyepr to estimate the size of the file. But I did install it...

You should be able to point pyepr to the downloaded file and query through the fields to see what the starting range is. Looks like the reader is currently setting this to 0 which is wrong.
They did change the format to change the size of one of the metadata fields. The envisat_slc reader should have been updated with this change.
I think this had to with variables named paramLength. This has now been made variable and should be read in from the file. I see that these are still hardcoded in the ERS reader. Just using the same snippet of code from Envisat_SLC reader should solve the issue hopefully.

I copied in the snippets from the EnviSAT_SLC.py and I get the same crashes. One more thing I noticed, though, that may be relevant?

It seems that stripmapApp thinks my data are raw?

Running step preprocess
DOPPLER:  ERS_EnviSAT_SLC useDEFAULT
Master data is in RAW format. Adding _raw to output name.
2018-12-21 13:14:38,089 - isce.make_raw - INFO - Extracting image
ESA Software Version:  6.0
2018-12-21 13:14:38,132 - isce.sensor.ers_envisat_slc - INFO - Using Delft Orbits  
1993-10-04 18:34:22.520603
<isceobj.Orbit.ODR.Arclist object at 0x7f020dfca240

To update sensor interpretation - see isceobj.StripmapProc.Factories. ERS_Envisat_SLC probably doesn't even show up in the current list.
There are functions to label sensor as raw/slc and zero doppler / native doppler. Both will need to be updated.

The readers for ERS and Envisat can be completely simplified using pyepr and reading the parameters using their simple interface.
One wont need to hardcode numbers and write parsers like we have been doing for a long time. pyepr should take care of all the headers etc.
There should be and almost one-to-one correspondence in names between parameters read in the current reader and pyepr.

Here is an example from their documentation:
http://avalentino.github.io/pyepr/html/interactive_use.html

pyepr is an external package like numpy / scipy /opencv. We dont plan to distribute pyepr with ISCE. I dont see an issue if we are just importing and using an external package.

Hi Gareth,

I have tried to use ERS_ENVISAT_SLC with stripmapApp.py, and I have found a similar problem regarding ERS SLC interpreted as RAW data. At least for that problem I found a workaround: add the sensor string 'ers_envisat_slc' in the file isce/components/isceobj/StripmapProc/Factories.py at lines 55 and 66 (functions isRawSensor() and isZeroDopplerSLC()).

Regards,
Jose Uribe

Jose,

Thanks for the tip... stripmapApp now will recognise my data as SLCs. It still crashes immediately after, but it's a start!

Piyush,

I have been able to open my data files in pyepr, and print out summary information like this:

epr.Product(SAR_IMS_1PNESA19931004_183407_00000018C098_00442) 6 datasets, 6 bands

epr.Dataset(MDS1_SQ_ADS) 1 records
epr.Dataset(MAIN_PROCESSING_PARAMS_ADS) 1 records
epr.Dataset(DOP_CENTROID_COEFFS_ADS) 1 records
epr.Dataset(CHIRP_PARAMS_ADS) 400 records
epr.Dataset(GEOLOCATION_GRID_ADS) 13 records
epr.Dataset(MDS1) 30927 records

epr.Band(slant_range_time) of epr.Product(SAR_IMS_1PNESA19931004_183407_00000018C098_00442)
epr.Band(incident_angle) of epr.Product(SAR_IMS_1PNESA19931004_183407_00000018C098_00442)
epr.Band(latitude) of epr.Product(SAR_IMS_1PNESA19931004_183407_00000018C098_00442)
epr.Band(longitude) of epr.Product(SAR_IMS_1PNESA19931004_183407_00000018C098_00442)
epr.Band(i) of epr.Product(SAR_IMS_1PNESA19931004_183407_00000018C098_00442)
epr.Band(q) of epr.Product(SAR_IMS_1PNESA19931004_183407_00000018C098_00442)

However, I am not too sure what to do next. How do I convert this information to procParamLength, geoParamLength and find the time to first sample, etc? How did you arrive at 10069 and 521*13? I can query individual records and fields, but am not apparently finding anything familiar (like a one-way delay).

Hi Gareth, I remember that I was able to run stripmapApp.py with ERS_ENVISAT_SLC and those changes, without crash. Could you share your stripmapApp output?

Jose Uribe

Here it is:

Running step preprocess
DOPPLER:  ers_envisat_slc useDEFAULT
Master data is in SLC format. Adding _slc to output name. 
2019-01-17 18:56:30,800 - isce.make_raw - INFO - Extracting image
Old ESA Software version:  6.0
2019-01-17 18:56:30,844 - isce.sensor.ers_envisat_slc - INFO - Using Delft Orbits
1993-10-04 18:34:07.437319
 <isceobj.Orbit.ODR.Arclist object at 0x7ff9d0f5df60> 
2019-01-17 18:56:32,724 - isce.sensor.ers_envisat_slc - DEBUG - Extracting orbits between 1993-10-04 18:29:07.437319 and 1993-10-04 18:39:25.846722
Extracted line: 1000
Extracted line: 2000   
. . .
Extracted line: 29000
Extracted line: 30000
API open (R): primary_slc/primary.slc
API close:  primary_slc/primary.slc
Using default doppler values for sensor: NoneType
Doppler Fit:  [1.94456612e-17 1.94456638e-17 2.57576643e-24 3.81690727e-31  5.21065041e-38]
2019-01-17 18:56:37,542 - isce.make_raw - INFO - Calculating Spacecraft Velocity
Orbit.calculateHeight: self.position =  [-2500585.4314116216, -5343853.408751463, 4052965.7926059784]
Orbit.calculateHeight: ellipsoid.a, ellipsoid.e2 =  6378137.0 0.0066943799901
Orbit.calculateHeight: self.position =  [-2531144.7497161725, -5370337.951443445, 3996429.3466763115]
Orbit.calculateHeight: ellipsoid.a, ellipsoid.e2 =  6378137.0 0.0066943799901
2019-01-17 18:56:37,569 - isce.make_raw - INFO - Calculating Squint Angle
Traceback (most recent call last):
   File "/home/gareth/pkgs/isce/isce/applications/stripmapApp.py", line 959, in <module>
    status = insar.run()
  File "/home/gareth/pkgs/isce/isce/components/iscesys/Component/Application.py", line 129, in run
    exitStatus = self._processSteps()
  File "/home/gareth/pkgs/isce/isce/components/iscesys/Component/Application.py", line 392, in _processSteps 
    result = func(*pargs, **kwargs)
  File "/home/gareth/pkgs/isce/isce/components/isceobj/StripmapProc/Factories.py", line 46, in __call
    return self.method(self.other, *args, **kwargs)
  File "/home/gareth/pkgs/isce/isce/components/isceobj/StripmapProc/runPreprocessor.py", line 78, in runPreprocessor 
    master = make_raw(sensor, masterdop)
  File "/home/gareth/pkgs/isce/isce/components/isceobj/StripmapProc/runPreprocessor.py", line 197, in make_raw 
    objMakeRaw(sensor=sensor, doppler=doppler)
  File "/home/gareth/pkgs/isce/isce/components/iscesys/Component/Component.py", line 166, in __call
    return getattr(self, self.__class__.__name__.lower())(*args)
  File "/home/gareth/pkgs/isce-2.2.0/isce/applications/make_raw.py", line 330, in make_raw 
    self.calculateSquint()
  File "/home/gareth/pkgs/isce-2.2.0/isce/applications/make_raw.py", line 195, in calculateSquint 
    "larger than starting Range (%s)") % (h-z, startingRange))
ValueError: Spacecraft Height - Terrain Height  (786687.67478877) larger than starting Range (0.0)   

Any suggestions gratefully received!

This page details the structure of the Envisat format SLC files:

http://envisat.esa.int/handbooks/asar/CNTR6-2-9.html

I just used the number of records and the fixed record length to determine byte offsets - 13 records of 521 bytes. Scroll down to bottom of each Dataset type in link above to see total number of bytes.

If the sensor object were updated to replace all the dicts - propDict, doppDict, geoDict, mph, sph etc to use pyepr Dataset directly, the theory is that one wouldn't have to deal with different versions of processor software / format updates since EPR is I believe an ESA funded effort to read all forms of Envisat data. Right now, we are parsing these dicts with manually keyed in byte offsets - which are done by epr/pyepr probably in a more correct / general manner.

Piyush

Hi Gareth, I tried other ERS SAR_IMS_1P products from ESA, and all work OK for me. It looks like something is different in your SAR_IMS_1P file. The error "Terrain Height (786687.67478877) larger than starting Range (0.0)" comes from isce/applications/make_raw.py, and means the variable startingRange=0 (and should not be 0). If you look at isce/components/isceobj/Sensor/ERS_EnviSAT_SLC.py, you will see that "startingRange" is calculated at line 188 from "timeToFirstSample", which is obtained at line 598 with readGeoParams() function (using hardcode file offset numbers as Piyush said).

Following Piyush's recommendation, you can get that value (and all product data) using pyepr. Here a python example to print "timeToFirstSample":

import epr

prod = epr.open('SAR_IMS_1PNESA19960214_142741_00000017A008_00425_04287_0000.E2')
rec = prod.get_dataset('GEOLOCATION_GRID_ADS').read_record(0)
print('timeToFirstSample:', rec.get_field('first_line_tie_points.slant_range_times').get_elem(0))

Regards,
Jose

Hi,

I have this error message once I try to run unpackFrame_ENV.py -i /mnt/l/CNRS-L/Data/Envisat/ -o /mnt/l/CNRS-L/Data/Envisat/SLC/, could you please help me in order to fix this issue.

message:

New ESA Software version: 6.03
sens: 2003-01-19 07:46:21.536694
Extracted line: 1000
Extracted line: 2000
Extracted line: 3000
Extracted line: 4000
Extracted line: 5000
Extracted line: 6000
Extracted line: 7000
Extracted line: 8000
Extracted line: 9000
Extracted line: 10000
Extracted line: 11000
Extracted line: 12000
Extracted line: 13000
Extracted line: 14000
Extracted line: 15000
Extracted line: 16000
Extracted line: 17000
Extracted line: 18000
Extracted line: 19000
Extracted line: 20000
Extracted line: 21000
Extracted line: 22000
Extracted line: 23000
Extracted line: 24000
Extracted line: 25000
Extracted line: 26000
Extracted line: 27000
Extracted line: 28000
Extracted line: 29000
Extracted line: 30000
API open (R): /mnt/l/CNRS-L/Data/Envisat/SLC/SLC.slc
API close: /mnt/l/CNRS-L/Data/Envisat/SLC/SLC.slc
Traceback (most recent call last):
File "/opt/isce-2.2.0/contrib/stack/stripmapStack/unpackFrame_ENV.py", line 95, in <module>
unpack(inps.h5dir, inps.slcdir)
File "/opt/isce-2.2.0/contrib/stack/stripmapStack/unpackFrame_ENV.py", line 50, in unpack
pc = obj._dopplerCoeffs[::-1]
AttributeError: 'EnviSAT_SLC' object has no attribute '_dopplerCoeffs'

Looks like this unpack script was not updated to the latest version. Replace the part below the numpy manipulation with what is in unpackFrame_S1.py . All of the doppler manipulation has been moved to extractDoppler() method.

Piyush

    obj.extractDoppler()

    pickName = os.path.join(slcname, 'data')
    with shelve.open(pickName) as db:
        db['frame'] = obj.frame

Yes. Read the instructions for the stack processors.

Hi Gareth and Jose,

I also realized that stripmapApp thinks the ERS SLC data are raw. I refined "Factories.py" as suggested by Jose:

add the sensor string 'ers_envisat_slc' in the file isce/components/isceobj/StripmapProc/Factories.py at lines 55 and 66.

Just curious, how did you guys set your .xml file (stripmapApp.xmp, master.xml)?

I found Scott's suggestion from here (http://earthdef.caltech.edu/boards/4/topics/1958?r=1967#message-1967). But with the new ISCE 2.2.0, the "doppler method" in the stripmapApp.xml will result in error:

Error. The attribute corresponding to the key "dopplermethod" is not present in the object "<class '__main__.Insar'>".

Would you please share your .xml file with me? Just want to make sure I have no problem with the xml files.

Thanks,
Han

Han,

This is what I used. It is very similar to Scott's although I commented out the "doppler method" line, to stop it crashing!

Gareth

Gareth,

Thanks you very much for the confirmation. I just finished a entire workflow with newly downloaded ERS SLC data (two acquisitions of one frame) and the 2019-Jan ISCE release.

I did the following steps one by one with no problem: 'preprocess', 'formslc', 'verifyDEM', 'topo', 'geo2rdr', 'coarse_resample', 'misregistration', 'refined_resample', 'dense_offsets', 'interferogram', 'filter', 'unwrap', 'geocode', 'geocodeoffsets', 'endup'

I skipped: 'cropraw', 'cropslc', 'rubber_sheet', 'fine_resample', 'split_range_spectrum', 'sub_band_resample', 'sub_band_interferogram', 'filter_low_band', 'filter_high_band', 'unwrap_low_band', 'unwrap_high_band', 'ionosphere'

However, I can't use 'mdx' or 'gdal' to open or display any mid-product, for example, '19970217.slc', 'geometry/lat.rdr.full' (product of 'topo'), or 'interferogram/topophase.flat' (product of 'filter'). I can't even use 'gdalinfo' to check their information. The errors are:

[hbrandon@centos 19970109_slc]$ gdalinfo 19970109.slc
gdalinfo failed - unable to open '19970109.slc'.

or

[hbrandon@centos 19970109_slc]$ mdx 19970109.slc

<< mdx Version 194.0 29-Mar-2013 >>
<< graphx Version 79.0 >>

Number of columns not specified
STOP

Do you have any idea? Maybe I did something totally wrong...

Thank you very much for all of your help. I really appreciate it.
Han