Model differencing

We just described a differencing method that computes an implied transformation given a range \(r\). This will produce a different result for each \(r\), but in reality, there's only a single true transformation, and the solutions at different ranges are different estimates of it.

If we just need to compare two different representations of the same lens, then we don't care about the implied transformation itself. The correct thing to do here would be to set \(r\) to the intended working distance of the system: for me this is generally infinity. Looking at a single \(r\), these implied-transformation fits will always overfit a little bit, but from experience, this doesn't affect the resulting model-difference output.

However, in some applications (extrinsics differencing for instance), we do want the true implied transform. And when we look at this computed transform more deeply, we see that with some \(r\) we get a result that is clearly wrong. For instance:

$ mrcal-show-projection-diff \
    --no-uncertainties       \
    --unset key              \
    --distance 1000          \
    opencv8.cameramodel      \
    splined.cameramodel

Transformation cam1 <-- cam0:  rotation: 0.316 degrees, translation: [-2.67878241 -0.832383    1.8063095 ] m
## WARNING: fitted camera moved by 3.336m. This is probably aphysically high, and something is wrong. Pass both a high and a low --distance?

More on --no-uncertainties later; it's here to speed things up, and isn't important. We asked for a diff at 1000m out, and the solver said that the optimal transform moves the camera coordinate system back by 1.8m and to the right by 2.7m. This is looking at the same data as before: comparing two solves from the tour of mrcal. Nothing moved. The camera coordinate system could have shifted inside the camera housing a tiny bit, but the solved shifts are huge, and clearly aren't inside the housing anymore. This needs a deeper investigation on how to do it "properly", but for practical use I have a working solution: solve using points at two ranges at the same time, a near range and a far range:

$ mrcal-show-projection-diff \
    --no-uncertainties       \
    --unset key              \
    --distance 1,1000        \
    opencv8.cameramodel      \
    splined.cameramodel

Transformation cam1 <-- cam0:  rotation: 0.316 degrees, translation: [2.90213542e-05 2.59871714e-06 1.73397909e-03] m

Much better. This is analogous to the multiple ranges-to-the camera we need when we compute a surveyed calibration and the multiple ranges we get when we tilt the chessboard during calibration.

When we use a fit to compute the implied transformation, we minimize the reprojection error. We hope that the main contributions to this error come from geometric misalignment. If this was the case, minimizing this reprojection error would produce the correct implied \(\left(R,t\right)\). Applying this transformation would correct the misalignment, leaving a very small residual. It is possible to have other error contributions though, which would break this method of finding the implied \(\left(R,t\right)\). This is common when intrinsics differences are significant; let's demonstrate that.

We just saw a difference result from the tour of mrcal, showing that the two models are similar but not identical. Once again:

mrcal-show-projection-diff \
  --unset key              \
  opencv8.cameramodel      \
  splined.cameramodel

This uses the default data-selection behavior of mrcal-show-projection-diff:

• uncertainties are used to weight the sampled points
• points are sampled from the whole imager (without uncertainties the default behavior is to use a limited region in the center instead)

What if we used all the data in the imager, but weighed them all equally?

mrcal-show-projection-diff \
  --unset key              \
  --no-uncertainties       \
  --radius 1e6             \
  --cbmax 50               \
  opencv8.cameramodel      \
  splined.cameramodel

Note the completely different color-bar scaling: this difference is huge! The reason this doesn't work is that projections of the two models don't just differ due to an implied transformation: there's also a significant difference in intrinsics. This difference is small near the center, and huge in the corners, so even the optimal \(\left(R,t\right)\) fits badly. We can solve this problem by either

• Giving the bad points low weights during the solve by using the default behavior, without --no-uncertainties. This is described in the next section
• Excluding the bad points entirely by passing a --radius

Let's demonstrate the effect of --radius. I re-ran the calibration from the tour of mrcal using LENSMODEL_OPENCV4. This model fits the wide lens even worse than the LENSMODEL_OPENCV8 fits we studied, and is not expected to work at all. But it is very good for this demo. If we run this solve, the outlier rejection logic kicks in, and makes the solve converge despite the model not fitting the data well. The resulting model is available here. Using an insufficient model like this isn't a good way to run calibrations: the outlier rejection will throw away the clearly-ill-fitting measurements, but marginal measurements will make it through, which will slightly poison the result. I'm only doing this for this demo. If you really do want to use an insufficient model, and you really do care only about points near the center, you should use the mrcal-cull-corners tool to throw away the data you don't care about, prior to calibrating.

Let's compute the diff between the narrow-only LENSMODEL_OPENCV4 lens model and the mostly-good-everywhere LENSMODEL_OPENCV8 model, using an expanding radius of points. We expect this to work well when using a small radius, and we expect the difference to degrade as we use more and more data away from the center.

# This is a zsh loop
for r (200 600 800 1000) {
  mrcal-show-projection-diff \
    --no-uncertainties       \
    --distance 1             \
    --radius $r              \
    --cbmax 15               \
    --unset key              \
    --extratitle "radius $r" \
    opencv[48].cameramodel
}

All of these radii are equally "right", and there's a trade-off in picking one. The models agree very well in a small region at the center, but they can also agree decently well in a much larger region, if we are willing to accept a higher level of error in the middle.

For the purposes of computing the implied transform you don't need a lot of data, so I generally use a small region in the center, where I have reasonable confidence that the intrinsics match. For a camera like this one --radius 500 is usually plenty.

Clearly the LENSMODEL_OPENCV4 solve does agree with the LENSMODEL_OPENCV8 solve well, but only in the center of the imager. The issue from a tooling standpoint is that in order for the tool to tell us that, we needed to tell the tool to only look at the center. That is unideal.

The issue we observed is that some regions of the imager have unreliable behavior, which poisons the fit. But we know where the fit is reliable: in the areas where the projection uncertainty is low. So we can weigh the fit by the inverse of the projection uncertainty, and we will then favor the "good" regions. Without requiring the user to specify the good-projection region.

This works, but with a big caveat. As described on the projection uncertainty page, lean models report overly-optimistic uncertainties. Thus when used as weights for the fit, areas that actually are unreliable will be weighted too highly, and will still poison the fit. We see that here, when comparing the LENSMODEL_OPENCV4 and LENSMODEL_OPENCV8 results. The above plots show that the LENSMODEL_OPENCV4 result is only reliable within a few 100s of pixels around the center. However, LENSMODEL_OPENCV4 is a very lean model, so its uncertainty at 1m out (near the sweet spot, where the chessboards were) looks far better than that:

mrcal-show-projection-uncertainty \
    --distance 1                  \
    --unset key                   \
    opencv4.cameramodel

And the diff using that uncertainty as a weight without specifying a radius looks poor:

mrcal-show-projection-diff \
  --distance 1             \
  --unset key              \
  opencv[48].cameramodel

Where this technique does work well is when using splined models, which produce realistic uncertainty estimates. To demonstrate, let's produce a splined-model calibration that is only reliable in a particular region of the imager. We do this by culling the tour of mrcal calibration data to throw out all points outside of a circle at the center, calibrate off that data, and run a diff on those results:

< corners.vnl \
  mrcal-cull-corners --imagersize 6000 3376 --cull-rad-off-center 1500 \
> /tmp/raw.vnl &&
  vnl-join --vnl-sort - -j filename /tmp/raw.vnl \
     <(< /tmp/raw.vnl vnl-filter -p filename --has level |
                       vnl-uniq -c |
                       vnl-filter 'count > 20' -p filename ) \
  > corners-rad1500.vnl

mrcal-calibrate-cameras                                                         \
  --corners-cache corners-rad1500.vnl                                           \
  --lensmodel LENSMODEL_SPLINED_STEREOGRAPHIC_order=3_Nx=30_Ny=18_fov_x_deg=150 \
  --focal 1900                                                                  \
  --object-spacing 58.8e-3                                                      \
  --object-width-n 14                                                           \
  '*.JPG'

mrcal-show-projection-uncertainty \
  --distance 1                    \
  --unset key                     \
  splined-rad1500.cameramodel

mrcal-show-projection-diff \
  --distance 1             \
  --unset key              \
  splined{,-rad1500}.cameramodel

The cut-down corners are here and the resulting model is here. The uncertainty of this model looks like this:

and the diff like this:

This is yet another reason to use only splined models for real-world lens modeling.

We just demonstrated that projection uncertainties provide a working method for selecting the points used to fit the implied transformation. From a practical standpoint, there's a downside: computing uncertainties makes the differencing method much slower. So while the uncertainty-aware method is available in mrcal-show-projection-diff, most of the time I run something like mrcal-show-projection-diff --no-uncertainties --radius 500. This falls back on simply using the points in a 500-pixel-radius circle at the center. This computes much faster, and with good-coverage-low-error calibrations usually produces a very similar result. If something looks odd, I will run the uncertainty-aware diff to debug. But my usual default is --no-uncertainties.