topsOffsetApp.py running issue

Added by Jianbao Sun over 4 years ago

Hi, all,
Is there anybody here successfully running the topsOffsetApp.py code?
I tried once with it, after successful running of topsApp.py, but it complains some properties in the master.xml/fine_coreg.xml are missing or wrongly spelling.
It actually stops at "_, minBurst, maxBurst = master.getCommonBurstLimits(coreg)", when calling runMergeSLCs.py.
Some bugs with it?
Thanks!
Jianbao


Replies (47)

RE: topsOffsetApp.py running issue - Added by Piyush Agram over 4 years ago

It works fine. You should have run topsApp with steps. If not, topsOffsetApp doesnt have enough information to work.

RE: topsOffsetApp.py running issue - Added by Jianbao Sun over 4 years ago

Thanks, Piyush, I actually run through to produce geocoded interferograms. Not sure why it always complains some attributes not found, such as 'output directory', 'orbit directory' etc. The attributes are in master.xml and it was fine for topsApp.py. The running stopped at LoadProduct function (for master.xml) of runMergedSLCs.py. I may need to debug the function. Jianbao

RE: topsOffsetApp.py running issue - Added by Joshua Cohen over 4 years ago

Hi, to clarify what Piyush said and to ask a further question, are you running topsApp at least to the 'fineresamp' step as specified in the example XML file/documentation for topsOffsetApp? If terminated before this step, topsApp does not generate the correct metadata in the internal objects to be running topsOffsetApp.

Best,
--Josh

RE: topsOffsetApp.py running issue - Added by Jianbao Sun over 4 years ago

Hi,Josh, yes, I run several times to confirm I got nice interferograms. For topsOffsetApp.py, I again to run it until fineresamp. I didn't see any errors and warnings. I believe it could be a minor error somewhere,I will print some variables to check what happened. Cheers, Jianbao

RE: topsOffsetApp.py running issue - Added by Jianbao Sun over 4 years ago

Dear Josh, I checked the code and found the calling sequence for topsOffsets.py as follows:
topsOffsetApp.py-->runMergeSLCs.py (error occure for 'load' master.xml)-->ProductManager.py('LoadProduct')-->Configurable.py ('load' at line 1390)-->Configurable.py (self.initProperties(self.catalog), at line 1415)

Before the line 1415 of Configurable.py, I print self.catalog and see something like: {'safe': ['/media/S1A_IW_SLC__1SDV.zip'], 'orbit directory': '/home/isce20160908/isce/orb_res', 'auxiliary data directory': '/home/isce20160908/isce/aux_files', '__const__': {}, 'output directory': 'masterdir', 'swath number': 1}

But when self.initProperties(self.catalog) is called at line 1415, it shows that some attributes cannot be found:
Error. The attribute corresponding to the key "safe" is not present in the object "<class 'iscesys.Component.ProductManager.ProductManager'>".

I don't know why the 'self.catalog' can't be processed by 'self.initProperties here'. A lot of parameters are in the function "initProperties", so I was lost...

Please provide further solutions. Not sure if I am clear for the issue.

Thanks!

RE: topsOffsetApp.py running issue - Added by Eric Fielding over 4 years ago

Jianbao,

Did you run "topsApp.py" with the "--steps" flag? That is necessary to save all the pickle files.

++Eric

RE: topsOffsetApp.py running issue - Added by Jianbao Sun over 4 years ago

Hi, Eric, yes, I tried. This is the same as before (say, using --start=xxx --end=xxx).
topsApp.py will read PICKLE/*.xml for processing, but topsOffsetApp.py is using master.xml/fine_coreg.xml for runMergeSLCs.py processing in my case. Not sure if this is the problem for my side.

print('\nMerging master and slave SLC bursts...')
master = self._insar.loadProduct(self._insar.masterSlcProduct + '.xml') #read master.xml, raise an error I reported here.
coreg = self._insar.loadProduct(self._insar.fineCoregDirname + '.xml')
_, minBurst, maxBurst = master.getCommonBurstLimits(coreg)
print('\nMerging bursts %02d through %02d.' % (minBurst,maxBurst))

Best,
Jianbao

RE: topsOffsetApp.py running issue - Added by Piyush Agram over 4 years ago

Are you running it in the correct folder? You might be running it from within the merged folder by mistake. You should run the script in the same folder as topsApp.py .

RE: topsOffsetApp.py running issue - Added by Piyush Agram over 4 years ago

Try this:

Add the following part to your topsApp

<component name="topsproc">
 <property name="master slc product">output_directory_of_master</property>
<property name="slave slc product">output_directory_of_slave</property>
</component>

Try running topsOffsetApp again.

Piyush

RE: topsOffsetApp.py running issue - Added by Jianbao Sun over 4 years ago

Hi,Piyush, I run all of the programs (topsApp.py and topsOffsetApp.py) at the same location, with all of the xml files kept the same.
Should the master.xml be read by the topsOffsetApp.py, or those xml files inside PICKLE be read for processing (with runMergeSLCs.py)?

RE: topsOffsetApp.py running issue - Added by Piyush Agram over 4 years ago

Please try the suggestion I posted earlier. It might be confusing master.xml and your outputdirectory.xml. By default, it may always be looking for master.xml instead of reading the "masterslcproduct" in the pickle files.

RE: topsOffsetApp.py running issue - Added by Jianbao Sun over 4 years ago

Hi, Piyush, I reprocessed the Italy quake data following your suggestions from start of topsApp.py (--steps flag). The topsOffsetApp.py errors are kept the same as before, though the interferograms are quite good. I attached all of the inputs and the screen output from "topsOffsetApp.py --end=mergeSLCs" here. Please take a look and let me known what's wrong with it. Thanks! Jianbao

RE: topsOffsetApp.py running issue - Added by Jianbao Sun over 4 years ago

Sorry, the attachment is missing.

topsOffsetApp.zip (12.3 kB)

RE: topsOffsetApp.py running issue - Added by Piyush Agram over 4 years ago

Hi,
Please follow instructions from numerous other posts about usage or insarproc / topsproc. topsproc should be contained within topsinsar.
You only need to rerun offsetapp with the changes.

Piyush

RE: topsOffsetApp.py running issue - Added by Jianbao Sun over 4 years ago

Hi, Piyushu,it works now, thanks!

RE: topsOffsetApp.py running issue - Added by Jianbao Sun over 4 years ago

Hi, all,
I had some tests with the topsOffsetsApp.py, and it works very fast and gives basically quite good results (with default parameters).
One of the problems I found is that some gaps occurred at the burst overlap regions in the dense_offsets.bil and related products (with value= -10000 ). The problem originated from the master.slc.full and slave.slc.full products. It seems that the runMergebursts.py missed some lines in the deburst process.
Does anyone else has the same problem?

Best,

2.jpg (378.8 kB)

RE: topsOffsetApp.py running issue - Added by Piyush Agram over 4 years ago

Use larger window sizes and smaller skips.
Please try to spend some time understanding what is happening in the different steps, what are implications of burst stitching, filtering, masking etc.

Piyush

RE: topsOffsetApp.py running issue - Added by Heming Liao about 2 years ago

Hi All,

I encountered the same problem here running topsOffsetApp.py and I tried to understand the conversation above but still cannot solve this problem. I use "topsApp.py" with the "--steps" flag to 'fineresamp', then run topsOffsetApp.py, I got the following error.
[hliao@u02 20171117_20171111_offset]$ topsOffsetApp.py
2019-01-14 00:35:11,274 - isce.insar - INFO - ISCE VERSION = 2.2.0, RELEASE_SVN_REVISION = 2497,RELEASE_DATE = 20180714, CURRENT_SVN_REVISION = exported
ISCE VERSION = 2.2.0, RELEASE_SVN_REVISION = 2497,RELEASE_DATE = 20180714, CURRENT_SVN_REVISION = exported

Merging master and slave SLC bursts...
Error. The attribute corresponding to the key "orbit directory" is not present in the object "<class 'iscesys.Component.ProductManager.ProductManager'>".
Possible causes are the definition in the xml file of such attribute that is no longer defined
in the object "<class 'iscesys.Component.ProductManager.ProductManager'>" or a spelling error

I added the sentences Piyush suggested to topsApp.xml:
<component name="topsproc">
<property name="master slc product">masterdir</property>
<property name="slave slc product">slavedir</property>
</component>
This does not solve the issue.

Could anyone provide some guidance here? I attached all my input xml files below.

Thanks in advance.

Heming

topsApp.xml Magnifier (1.4 kB)

master_S1.xml Magnifier (435 Bytes)

slave_S1.xml Magnifier (350 Bytes)

topsOffsetApp.xml Magnifier (2 kB)

RE: topsOffsetApp.py running issue - Added by Heming Liao about 2 years ago

by the way, I can run topsApp.py to the end and generate offsets maps without any problem. range and azimuth offset maps attached.

I can see clear offsets jump at the boundary of swaths, is there any setting in the topsApp.py process I may not be aware that could fix this?

Thanks.

azimuth_offset.jpg (251.2 kB)

range_offset.jpg (230.7 kB)

RE: topsOffsetApp.py running issue - Added by Piyush Agram about 2 years ago

topsOffsetApp is deprecated as indicated in the release notes of a previous ISCE release. Offsets can be generated directly using topsApp.xml itself.
This dataset has been processed numerous times - for UNAVCO tutorials in 2018 as well as for sample products for ARIA on aria-products.jpl.nasa.gov
Which version of ISCE are you using? Do you see the swath boundaries when using the CPU offset code? The GPU ampcor code that is part of the public release is experimental.

Piyush

RE: topsOffsetApp.py running issue - Added by Heming Liao about 2 years ago

Hi Piyush,

Thank you for your reply. I am using the latest version, ISCE v 2.2.0. I turned off the 'use GPU' property in the topsApp.xml, and my offsets map still have the discontinuity at sub-swath boundaries. I checked the UNAVCO tutorials, and found only one sub-swath was processed in the tutorial, I guess that's probably the reason why we didn't see discontinuity there.

Can you confirm that the offsets does not have jump at the sub-swath boundaries following the standard processing using topsApp.py?

Thank you.

Heming

RE: topsOffsetApp.py running issue - Added by Piyush Agram about 2 years ago

The sample product on the ARIA page includes ionospheric corrections for this pair and that we know that this one was definitely impacted. Do you see the sub-swath offsets when you draw profile across the subswaths? What is the magnitude of these offsets? If there are actual offsets, we would probably see it in the phase as well.

We will try to reprocess this pair to see if we see discontinuities.

Piyush

RE: topsOffsetApp.py running issue - Added by Heming Liao about 2 years ago

Hi Piyush,

There is an offset jump of ~0.03 pixel (pixel location in range direction ~675 and pixel ~1350) at the sub-swath boundaries in the azimuth offsets map. Please see the profile figure attached.

it looks like there are still some other artifacts in it azimuth offsets map (the offset value appears at specific values, which I would expect to be more smooth). Could this be related to the oversampling factor we used in the cross-correlation?

Thanks.
Heming

offsets_profile.jpg (382.1 kB)

RE: topsOffsetApp.py running issue - Added by Jose Uribe about 2 years ago

Hi,

I also got offset jumps among swaths (and also among bursts) when I enabled GPU with topsApp.py (using step denseoffsets of topsApp.py, not topsOffsetApp.py). However, if I disabled GPU support and run topsApp.py, and then run just denseoffsets with GPU enabled, the jumps disappear.
Now I understand that GPU support is experimental, probably is not working fine during coregistration (probably GPU Geo2rdr)?

Regards,
Jose Uribe

RE: topsOffsetApp.py running issue - Added by Piyush Agram about 2 years ago

Thanks Jose for narrowing down the issue. Will try to replicate the same behavior here. It is possible that we fixed some bug in runGeo2rdrCPU but did not propagate it to runGeo2rdrGPU in runFineOffsets. Will try to track it down.

The 0.03 step has to do with the quantization of 1/32 from the oversampling factor.

Piyush

1 2 Next » (1-25/47)