mrcal roadmap

• The input noise should be characterized better. Currently we use the distribution from the optimal residuals. This feels right, but the empirical distribution isn't entirely gaussian. Why? There's an attempt to quantify the input noise directly in mrgingham. Does it work? Does that estimate agree with what the residuals tell us? If not, which is right? If a better method is found, the observed_pixel_uncertainty should come back as something the user passes in.
• Can I quantify heteroscedasticity to detect model errors? In the tour of mrcal the human observer can clearly see patterns in the residuals. Can these patterns be detected automatically to flag these issues, especially when they're small and not entirely obvious? Do I want a "white test"?
• As desired, we currently report high uncertainties in imager regions with no chessboards. When using a splined model, the projection in those regions is controlled entirely by the regularization terms, so we report high uncertainties there only because of the moving extrinsics. This isn't a great thing to rely on, and could break if I have some kind of surveyed calibration (known chessboard and/or camera poses).

Fitting of the implied transformation is key to computing a diff, and various details about how this is done could be improved. Currently mrcal computes this from a fit. The default behavior of mrcal-show-projection-diff is to use the whole imager, using the uncertainties as weights. This has two problems:

• If using a splined model, this is slow
• If using a lean model, the overly-optimistic uncertainties you get from lean models tend to poison the fit, as seen in the documentation.

• Currently I have a routine to compute projection uncertainty. And a separate routine to compute triangulation uncertainty. It would be nice to have a generic monocular uncertainty routine that is applicable to those and more cases. Should I be computing the uncertainty of a stabilized, normalized stereographic projection of \(\mathrm{unproject}\left(\vec q\right)\)? Then I could do monocular tracking with uncertainties. Can I derive the existing uncertainty methods from that one?
• As noted on the triangulation page, some distributions become non-gaussian when looking at infinity. Is this a problem? When is it a problem? Should it be fixed? How?

• It's currently not clear how to choose the spline order (the order configuration parameter) and the spline density (the Nx and Ny parameters). There's some trade-off here: a quadratic spline needs denser knots. An initial study of the effects of spline spacings appears here. Can this be used to select the best spline configuration? We see that the uncertainty oscillates, with peaks at the knots. The causes and implications of this need to be understood better
• The current regularization scheme is iffy. More or less mrcal is using simple L2 regularization. Something is required to tell the solver what to do in regions of no data. The transition between "data" and "no-data" regions is currently aphysical, as described in the documentation. Changing the regularization scheme to pull towards the mean, and not towards 0 could possibly fix this. An earlier attempt to do that was reverted because any planar splined surface would have "perfect" regularization, and that was breaking things (crazy focal lengths would be picked). But now that I'm locking down the intrinsics core when optimizing splined models, this isn't a problem anymore, so maybe that approach should be revisited.

• The current outlier-rejection scheme is simplistic. A smarter approach is available in libdogleg (Cook's D and Dima's variations on that). Bringing those in could be good
• Outlier rejection is currently only enabled for chessboard observations. It should be enabled for discrete points as well

• A pre-filter should be added to the mrcal-stereo tool to enhance the edges prior to stereo matching. A patch to add an early, untested prototype:

  diff --git a/mrcal/stereo.py b/mrcal/stereo.py
  index 6ba3549..7a6eabc 100644
  --- a/mrcal/stereo.py
  +++ b/mrcal/stereo.py
  @@ -1276,5 +1276,22 @@ data_tuples, plot_options. The plot can then be made with gp.plot(*data_tuples,
                  q0[ 0,-1],
                  q0[-1,-1] )
  
  +    image1 = image1.astype(np.float32)
  +    image1 -= \
  +        cv2.boxFilter(image1,
  +                      ddepth     = -1,
  +                      ksize      = tuple(template_size1),
  +                      normalize  = True,
  +                      borderType = cv2.BORDER_REPLICATE)
  +    template_size0 = (round(np.max(q0[...,1]) - np.min(q0[...,1])),
  +                      round(np.max(q0[...,0]) - np.min(q0[...,0])))
  +    # I don't need to mean-0 the entire image0. Just the template will do
  +    image0 = image0.astype(np.float32)
  +    image0 -= \
  +        cv2.boxFilter(image0,
  +                      ddepth     = -1,
  +                      ksize      = template_size0,
  +                      normalize  = True,
  +                      borderType = cv2.BORDER_REPLICATE)
       image0_template = mrcal.transform_image(image0, q0)
  
  

• Currently a stereo pair arranged axially (one camera in front of the other) cause mrcal to fail. But it could work: the rectified images are similar to a polar transform of the input.

An early stage of a calibration run generates a rough estimate of the chessboard geometry. Internally this is currently assuming a pinhole model, which is wrong, and currently requires an ugly hack. This does appear to work fairly well, but it should be fixed

• The mrcal-stereo tool should be able to estimate the field of view automatically: the user should not be required to pass --az-fov-deg and --el-fov-deg

• Currently mrcal.triangulate() broadcasts nicely, while mrcal.projection_uncertainty() does not. It would be nice if it did and if its API resembled that of mrcal.triangulate()

• mrcal-show-geometry tool: the mrcal-stereo tool produces a field-of-view visualization. This should be made available in the Python API and in the mrcal-show-geometry tool
• dance-study.py: if asked for chessboards that are too close, the tool goes into an infinite loop as it searches for chessboard poses that are fully visible by the camera. Something smarter than an infinite loop should happen
• Warnings in mrcal.c: there are a number of warnings in mrcal.c tagged with // WARNING that should eventually be addressed. This has never been urgent-enough to deal with. But someday
• viz tools should accept --vectorfield and --vector-field