topsOffsetApp.py running issue

Added by Jianbao Sun over 4 years ago

Hi, all,
Is there anybody here successfully running the topsOffsetApp.py code?
I tried once with it, after successful running of topsApp.py, but it complains some properties in the master.xml/fine_coreg.xml are missing or wrongly spelling.
It actually stops at "_, minBurst, maxBurst = master.getCommonBurstLimits(coreg)", when calling runMergeSLCs.py.
Some bugs with it?
Thanks!
Jianbao


Replies (47)

RE: topsOffsetApp.py running issue - Added by Piyush Agram about 2 years ago

I have not been able to reproduce this issue. The numbers coming out of the CPU and GPU modules are the same. The only difference would be in the python run scripts that I haven't been able to track down yet.

RE: topsOffsetApp.py running issue - Added by Jose Uribe about 2 years ago

Hi Piyush,

I tried again to reproduce the jumps problems, and my issue was just GPU memory: GPU TOPO failed with "Sync kernel error: out of memory". I didn't realize about this before, just disabled GPU support and all was fine.
I am using a Geforce GTX 1060 with 6 GB, probably too small. However, I changed the GPU architecture for PyCuAmpcor (from sm_35 to sm_61) and seems to work OK for denseoffsets.

Jose

RE: topsOffsetApp.py running issue - Added by Piyush Agram about 2 years ago

Great. Yes. The GPU code in ISCE2 doesnt have all the checks and whistles that we have slowly been adding.
I have been testing everything on a K80 and did not see any issues. Thats the lowest chip that we are designing the code for.
I suspect this might be the case for Heming as well.

RE: topsOffsetApp.py running issue - Added by Heming Liao about 2 years ago

Thanks Piyush and Jose for tracking this. I delete my previous processing and rerun the processing with and without using GPU setting. I still get the very tiny offset jump at the swath boundary.

I didn't see the error message (out of memory) like Jose's processing in my processing.

I haven't go into details of the code, I am not sure whether this jumps could be related to the parameters I am using in dense registration (I literally use all default setting).

I attached my xml file and related results below. Do you see any obvious mistake in my xml file?

Thank you for looking at this.

Heming

RE: topsOffsetApp.py running issue - Added by Heming Liao about 2 years ago

Hi All,

I get a related question here.

I am using the offsets for glacier velocity estimate. I calculate the range and azimuth offset map and they looks kind of noisy, so I am trying to use offset SNR or coherence file as a mask to mask out those unreasonable estimate or estimate over water area. However, I found the SNR value over the ocean area (left bottom part in the geocoded image) has a larger SNR than those areas even with good offset estimate. Also, the coherence map (phsig.cor.geo) over the ocean area has very similar value (~0.3) to area with pretty good offset estimate. These doesn't look right to me, as I suspect SNR or coherence value for the ocean area should be smaller to the area with good offset. Does any one have any explanation for this? why does ocean area has larger snr or similar coherence value to areas with good offset estimate? I know DEM can be mask for water area, but I thought snr or coherence could be an effective mask too.

I attached my xml file and related offset, snr, cor result attached. I appreciate any help.

Thanks

Heming

RE: topsOffsetApp.py running issue - Added by Eric Fielding about 2 years ago

Hi Heming,

I think you are misinterpreting the output of the ISCE denseampcor program used to measure pixel offsets. One of the outputs of the denseampcor program is an estimate of the "covariance" (actually it is standard deviation) or error in the ampcor matches. The units are in pixels. High values mean the matches are poor. The ampcor output called "SNR" is something like the signal-to-noise ratio of the ampcor matches, but it is not very reliable. The "coherence" map phsig.cor.geo is an effective coherence estimated from the phase sigma (standard deviation in small windows) of the interferogram phase after the filtering, and has no relation to the pixel offset estimates. After filtering, the effective coherence of 0.3 is close to the minimum possible.

The denseampcor program only looks at the amplitude of the two images to calculate matches and offsets, so it is not related to the interferogram phase.

++Eric F.

RE: topsOffsetApp.py running issue - Added by Heming Liao about 2 years ago

Hi Eric,

Thank you very much for your reply. Yes, I was looking for some metric to quantify the offset measurements quality. You mentioned some estimate of "covariance" (actually it is standard deviation) or error in the ampcor matches, is that the dense_offset_snr file you are talking about? or there is another file I am not aware?

Thanks.
Heming

RE: topsOffsetApp.py running issue - Added by Eric Fielding about 2 years ago

Hi Heming,

Sorry, I was wrong about the outputs of the denseampcor function in stripmapApp and topsApp (I have only used the stripmapApp version, but I think they are the same). I was thinking about the original ampcor program that calculated a "covariance" value in addition to the SNR estimate. For whatever reason, the developers that added the denseampcor function into stripmapApp and topsApp decided to only write out the SNR estimates. I am looking at the SNR estimates for a stripmapApp denseoffset calculation and I see that the SNR is strongly affected by the overall amplitude of the radar image. That makes it a poor estimate of the matching accuracy.

I never used the SNR values with the old ampcor program because I knew they were not reliable, as confirmed by one of the ampcor authors, Scott Hensley. I am not sure when things were changed to write out only the SNR. It is that way in DenseAmpcor.py also.

++Eric

RE: topsOffsetApp.py running issue - Added by Piyush Agram about 2 years ago

Hi Heming,
The question is did you perform all the steps with GPU turned off. The issue Jose found was that one of the geometry modules generated an error. If this was the case, only switching ampcor step to GPU / noGPU will have very little effect. The default data blocksizes are set to what we expect on a K40/K80. If you are using a different GPU with lower memory, its entirely possible that the errors are not caught. The right thing to do is to start from the topo step with noGPU turned on and then look at the result. We still cant reproduce what you are seeing over Kurdistan. We will get back to you again on this.

Regarding the other dataset, what does the interferogram look like over the ocean region? Its unlikely that you will see high phase sigma correlation without clear fringes. There are regions over sea ice where coherence in some pairs are high but the latitude seems too low for that. It is also a function of adaptive filter strength.

Piyush

RE: topsOffsetApp.py running issue - Added by Piyush Agram about 2 years ago

We are happy to take contributions - covariance is still computed. Users who find that useful can probably contribute the minor updates to output it as well.

Piyush

RE: topsOffsetApp.py running issue - Added by Eric Fielding about 2 years ago

Good to hear the covariance values are still computed. There should be three values. Maybe somebody decided it was better to keep only the single SNR instead of the three covariance values. I found these lines in ampcor.F that save the values to an array, but I didn't find where the arrays are written out.

                     numRowTable = numRowTable + 1
                     i_centerxiArr(numRowTable) = i_centerxi
                     i_centeryiArr(numRowTable) = i_centeryi
                     r_shftxoscArr(numRowTable) = r_shftxosc
                     r_shftyoscArr(numRowTable) = r_shftyosc
                     r_snrArr(numRowTable) = r_snr
                     r_cov1Arr(numRowTable) = r_cov(1)
                     r_cov2Arr(numRowTable) = r_cov(2)
                     r_cov3Arr(numRowTable) = r_cov(3)
!c                     write(15,151) i_centerxi,r_shftxosc,i_centeryi,r_shftyosc,
!c     &                    r_snr,r_cov(1),r_cov(2),r_cov(3)

How is your Python level, Heming?

++Eric

RE: topsOffsetApp.py running issue - Added by Piyush Agram about 2 years ago

The bulk of the work is already one and is available in Ampcor.py . Should be trivial to have denseampcor gather it too.

RE: topsOffsetApp.py running issue - Added by Heming Liao about 2 years ago

Hi Eric,

Thank you very much for checking this. I just transfer from matlab to python, so I am still at the beginning level for python. I will take a look at this.

Best,
Heming

Eric Fielding wrote:

Good to hear the covariance values are still computed. There should be three values. Maybe somebody decided it was better to keep only the single SNR instead of the three covariance values. I found these lines in ampcor.F that save the values to an array, but I didn't find where the arrays are written out.
[...]

How is your Python level, Heming?

++Eric

RE: topsOffsetApp.py running issue - Added by Heming Liao about 2 years ago

Hi Piyush and Eric,

Sorry to get back to you late. Please see the attached file for the range and azimuth offset, SNR, Interferometric phase, Phase correlation map. As you can see for land area, where we have reasonable good offsets estimate, the phase is decorrelated and the phase coherence is very low. Phase coherence value for the land area is similar to that in the ocean at the left bottom area. This means the phase correlation is not a good metric to remove outliers for offsets estimate. For the SNR, the estimate is a little bit strange, as we get very high SNR at both area with good interferometric coherence and ocean area. other areas with relatively good offsets the SNR is very low. This doesn't seem right, so I think another product from the registration processing will be needed to quantify the offsets quality. I will take a look at the ampcor.py as piyush suggested.

Piyush, for the GPU, I set <property name="use GPU">True</property> in the topsApp.xml file and run the topsApp with steps setting --end=denseoffsets. I guess this setting will apply GPU on for all steps, right?
For the run without gpu, I just comment out <property name="use GPU">True</property>. Will this turn offset GPU for all steps?

Below is my gpu informaton:

lspci -vnn | grep VGA -A 12
06:00.0 VGA compatible controller [0300]: Matrox Graphics, Inc. G200eR2 [102b:0534] (prog-if 00 [VGA controller])
Subsystem: Dell Device [1028:04f8]
Flags: bus master, medium devsel, latency 64, IRQ 11
Memory at d8000000 (32-bit, prefetchable) [size=16M]
Memory at deffc000 (32-bit, non-prefetchable) [size=16K]
Memory at de000000 (32-bit, non-prefetchable) [size=8M]
Expansion ROM at <unassigned> [disabled]
Capabilities: <access denied>

08:00.0 Ethernet controller [0200]: Broadcom Corporation NetXtreme II BCM57810 10 Gigabit Ethernet [14e4:168e] (rev 10)
Subsystem: Broadcom Corporation Device [14e4:1006]
Flags: bus master, fast devsel, latency 0, IRQ 48
Memory at d9000000 (64-bit, prefetchable) [size=8M]

Thank you very much for tracking this.

Best,
Heming

Piyush Agram wrote:

Hi Heming,
The question is did you perform all the steps with GPU turned off. The issue Jose found was that one of the geometry modules generated an error. If this was the case, only switching ampcor step to GPU / noGPU will have very little effect. The default data blocksizes are set to what we expect on a K40/K80. If you are using a different GPU with lower memory, its entirely possible that the errors are not caught. The right thing to do is to start from the topo step with noGPU turned on and then look at the result. We still cant reproduce what you are seeing over Kurdistan. We will get back to you again on this.

Regarding the other dataset, what does the interferogram look like over the ocean region? Its unlikely that you will see high phase sigma correlation without clear fringes. There are regions over sea ice where coherence in some pairs are high but the latitude seems too low for that. It is also a function of adaptive filter strength.

Piyush

RE: topsOffsetApp.py running issue - Added by Eric Fielding about 2 years ago

Hi Heming,

The phsig.cor values near 0.3 are essentially zero coherence in the ocean and on the land. There is always a large bias in the phase-sigma correlation estimates in areas of very low coherence, so you need to consider that in looking at it. The original correlation topophase.cor will be somewhat less biased, but will still be far from zero over water.

++Eric

RE: topsOffsetApp.py running issue - Added by Heming Liao about 2 years ago

Hi Eric,

How could I use the covariance as a metric for the offsets quality?

I just checked Ampcor.py, it does have information of the three component of the variance:

offset.setCovariance(sigx,sigy,sigxy)

I used to use GAMMA software, which provide either a SNR or correlation coeffcient to for registration quality index for registration point. I am not so clear about the relation of the covariance with the SNR or correlation coefficient used in GAMMA.

Thanks.

Heming

RE: topsOffsetApp.py running issue - Added by Eric Fielding about 2 years ago

Heming,

The covariance estimates calculated by ampcor are a measurement of the sharpness of the cross-correlation peak, converted to an estimate of the x, y and x-y covariance errors on the offset measurement. I forget now whether you need to take the square root, but the units are either pixels or pixels^2. This is a direct estimate of the offset quality. I typically use a threshold where I discard the offset measurements where the errors are larger than about 0.5 or 1 pixel because that means the cross-correlation match is bad. I don't know what the GAMMA software calculates.

++Eric

RE: topsOffsetApp.py running issue - Added by Heming Liao about 2 years ago

Hi Eric,

Thank you very much for your explanation. This is really helpful.

Best,
Heming

RE: topsOffsetApp.py running issue - Added by Piyush Agram about 2 years ago

Hi Heming,
Just to confirm that you only see the offset in the azimuth offsets for the Kurdistan pair and no visible shift in range offsets. Is that correct?

I think in an earlier post - http://earthdef.caltech.edu/boards/4/topics/1432?r=2535#message-2535
The filenames are flipped (might be due to labeling in the jupyter notebook).

I have been checking range offsets based on the above post and there are no discontinuities in the processing.
If the offsets are only in azimuth - this could be ionosphere. Will get back to you on this.

Piyush

RE: topsOffsetApp.py running issue - Added by Heming Liao about 2 years ago

Hi Piyush,

I double checked it. Yes, it only appeared the azimuth offset.

Heming

RE: topsOffsetApp.py running issue - Added by Piyush Agram about 2 years ago

Hi Heming,
Having cross checked with Heresh Fattahi and David Bekaert, this azimuth offset pattern is definitely due to ionosphere. Attached are interferograms before and after ionosphere correction generated by David using ionosphere estimation module contributed by Cunren Liang (see https://eartharxiv.org/atxr7/ ) . The seamless range offsets suggests that there is nothing wrong with the processing itself.

This module is part of the next release of ISCE. The azimuth offsets are not adjusted. The corrections in this module adjust the phase.

Piyush

With_iono_correction.png - With ionosphere correction (997.8 kB)

Original_without_ionocorrection.png - Original interferogram (994.3 kB)

iono_phase.png - Ionosphere phase (151.1 kB)

RE: topsOffsetApp.py running issue - Added by Heming Liao about 2 years ago

Hi Piyush,

Thank you for tracking this. I am glad to see this has been implemented for Sentinle-1 data. I will be glad to try this.

Do you have any information when will the next version of ISCE release?

Heming

Piyush Agram wrote:

Hi Heming,
Having cross checked with Heresh Fattahi and David Bekaert, this azimuth offset pattern is definitely due to ionosphere. Attached are interferograms before and after ionosphere correction generated by David using ionosphere estimation module contributed by Cunren Liang (see https://eartharxiv.org/atxr7/ ) . The seamless range offsets suggests that there is nothing wrong with the processing itself.

This module is part of the next release of ISCE. The azimuth offsets are not adjusted. The corrections in this module adjust the phase.

Piyush

« Previous 1 2 (26-47/47)