A tour of mrcal: calibration

Next, let's examine the residuals more closely. We have an overall RMS reprojection error value from above, but let's look at the full distribution of errors for all the images and cameras:

mrcal-show-residuals    \
  --histogram           \
  --set 'xrange [-2:2]' \
  --unset key           \
  --binwidth 0.1        \
  opencv8.cameramodel

We would like to see a normal distribution (since that's what the noise model assumes), and we roughly do see that here. If the images were captured with varying illumination (which happens with different board orientations and a directional light source), the center cluster or the tails would get over-populated. That would be a violation of the noise model, but things still appear to work OK even in that case.

Let's look deeper. If there's anything really wrong with our data, then we should see it in the worst-fitting images. Let's ask the tool to show us the worst one:

mrcal-show-residuals-board-observation \
  --from-worst                         \
  --vectorscale 200                    \
  --circlescale 0.5                    \
  --set 'cbrange [0:2]'                \
  opencv8.cameramodel                  \
  0

The residual vector for each chessboard corner in this observation is shown, scaled by a factor of 200 for legibility (the actual errors are tiny!) The circle color also indicates the magnitude of the errors. The size of each circle represents the weight given to that point. The weight is reduced for points that were detected at a lower resolution by the chessboard detector. So even high reprojection errors (shown as the vectors) could result in low measurements if the weight was low. Points thrown out as outliers are not shown at all.

Residual plots such as this one are a good way to identify common data-gathering issues such as:

• out-of focus images
• images with motion blur
• rolling shutter effects
• camera synchronization errors
• chessboard detector failures
• insufficiently-rich models (of the lens or of the chessboard shape or anything else)

See the how-to-calibrate page for practical details. Back to this image. In absolute terms, even this worst-fitting image fits really well. The RMS reprojection error in this image is 0.70 pixels. The residuals in this image look mostly reasonable, but there is an issue: if the model fit this data well, the residuals would sample independent, random noise. Thus the residuals would all be uncorrelated with each other, but here there's a pattern: we can see that the errors are mostly radial in the chessboard (they point to/from the center). A unmodeled chessboard flex could cause this kind of problem, for instance. This is a source of bias in this solution. Let's keep going, keeping this in mind.

One issue with lean models such as LENSMODEL_OPENCV8 (used here) is that the radial distortion is never quite right, especially as we move further and further away from the optical axis: this is the last point in the common-errors list above. The result of these radial distortion errors is high residuals in the corners and near the edges of the image. We can clearly see this here in the 10th-worst image (10th worst and not the worst because the really poor-fitting points are thrown out as outliers):

mrcal-show-residuals-board-observation \
  --from-worst                         \
  --vectorscale 200                    \
  --circlescale 0.5                    \
  --set 'cbrange [0:2]'                \
  opencv8.cameramodel                  \
  10

Clearly this image is a problem: the points near the corners have poor residuals, and the entire column of points at the edge was thrown out as outliers. We note that this is observation 47, so that we can come back to it later.

And we can see this same high-error-in-the-corners effect in the residual magnitudes of all the observations:

mrcal-show-residuals                   \
  --magnitudes                         \
  --set 'cbrange [0:1.5]'              \
  opencv8.cameramodel

In addition to the expected high errors at the edges, this plot also shows an anomalous ring of poorly-fitting observations at the center. This could maybe be caused by the solver sacrificing accuracy at the center of the image to get a better fit in the much larger areas further out? We shall see in a bit. Let's look at the systematic errors in another way: let's look at all the residuals over all the observations, color-coded by their direction, ignoring the magnitudes:

mrcal-show-residuals    \
  --directions          \
  --unset key           \
  opencv8.cameramodel

As before, if the model fit the observations, the errors would represent random noise, and no color pattern would be discernible in these dots. But here we can clearly see the colors clustered together. This is not random noise, and is a very clear indication that this lens model is not able to fit this data.

It would be good to have a quantitative measure of these systematic patterns. At this time mrcal doesn't provide an automated way to do that. This will be added in the future.

Clearly there're unmodeled errors in this solve. As we have seen, the errors here are all fairly small, but they become important when doing precision work like, for instance, long-range stereo.

Let's fix it.

The base of a splined stereographic model is a stereographic projection. In this projection, a point that lies an angle \(\theta\) off the camera's optical axis projects to \(\left|\vec q - \vec q_\mathrm{center}\right| = 2 f \tan \frac{\theta}{2}\) pixels from the imager center, where \(f\) is the focal length. Note that this representation supports projections behind the camera (\(\theta > 90^\circ\)) with a single singularity directly behind the camera. This is unlike the pinhole model, which has \(\left|\vec q - \vec q_\mathrm{center}\right| = f \tan \theta\), and projects to infinity as \(\theta \rightarrow 90^\circ\). The pinhole model can not represent projections behind the imager.

Basing the new model on a stereographic projection lifts the inherent forward-view-only limitation of LENSMODEL_OPENCV8.

Let \(\vec p\) be the camera-coordinate system point being projected. The angle off the optical axis is

\[ \theta \equiv \tan^{-1} \frac{\left| \vec p_{xy} \right|}{p_z} \]

The normalized stereographic projection is

\[ \vec u \equiv \frac{\vec p_{xy}}{\left| \vec p_{xy} \right|} 2 \tan\frac{\theta}{2} \]

This initial projection operation unambiguously collapses the 3D point \(\vec p\) into a 2D vector \(\vec u\). We then use \(\vec u\) to look-up an adjustment factor \(\Delta \vec u\) using two splined surfaces: one for each of the two elements:

\[ \Delta \vec u \equiv \left[ \begin{aligned} \Delta u_x \left( \vec u \right) \\ \Delta u_y \left( \vec u \right) \end{aligned} \right] \]

We can then define the rest of the projection function:

\[\vec q = \left[ \begin{aligned} f_x \left( u_x + \Delta u_x \right) + c_x \\ f_y \left( u_y + \Delta u_y \right) + c_y \end{aligned} \right] \]

The parameters we can optimize are the spline control points and \(f_x\), \(f_y\), \(c_x\) and \(c_y\), the usual focal-length-in-pixels and imager-center values.

Let's run the same exact calibration as before, but using the richer model to specify the lens:

mrcal-calibrate-cameras                                                         \
  --corners-cache corners.vnl                                                   \
  --lensmodel LENSMODEL_SPLINED_STEREOGRAPHIC_order=3_Nx=30_Ny=18_fov_x_deg=150 \
  --focal 1900                                                                  \
  --object-spacing 0.0588                                                       \
  --object-width-n 14                                                           \
  '*.JPG'

Reported diagnostics:

## initial solve: geometry only
## RMS error: 9.523144801739077

## initial solve: geometry and LENSMODEL_STEREOGRAPHIC core only
=================== optimizing everything except board warp from seeded intrinsics
mrcal.c(5351): Threw out some outliers (have a total of 28 now); going again
## final, full optimization
## RMS error: 0.24540598163794722
RMS reprojection error: 0.2 pixels
Worst residual (by measurement): 1.3 pixels
Noutliers: 28 out of 16464 total points: 0.2% of the data
calobject_warp = [-1.26851438e-04 -8.03269701e-05]

Wrote /tmp/camera-0.cameramodel

The resulting model is available here.

The requested lens model LENSMODEL_SPLINED_STEREOGRAPHIC_order=3_Nx=30_Ny=18_fov_x_deg=150 is the only difference in the command. Unlike LENSMODEL_OPENCV8, this model has some configuration parameters: the spline order (we use cubic splines here), the spline density (here each spline surface has 30 x 18 knots), and the rough horizontal field-of-view we support (we specify about 150 degrees horizontal field of view). These parameters are fixed in the model, and are not subject to optimization.

There're over 1000 lens parameters here, but the problem is sparse, so we can still process this in a reasonable amount of time.

The previous LENSMODEL_OPENCV8 solve had 564 points that fit so poorly, the solver threw them away as outliers; here we have only 28. The RMS reprojection error dropped from 0.4 pixels to 0.2 pixels. The estimated chessboard shape stayed roughly the same: flat. These are all what we expect and hope to see.

Let's look at the residual distribution in this solve, compared to before:

mrcal-show-residuals    \
  --histogram           \
  --set 'xrange [-2:2]' \
  --unset key           \
  --binwidth 0.1        \
  splined.cameramodel

This still has the nice bell curve, but the residuals are lower: the data fits better than before.

Let's look at the worst-fitting single image:

mrcal-show-residuals-board-observation \
  --from-worst                         \
  --vectorscale 200                    \
  --circlescale 0.5                    \
  --set 'cbrange [0:2]'                \
  splined.cameramodel                  \
  0

Interestingly, the worst observation here is the same one we saw with LENSMODEL_OPENCV8, so we can do a nice side-by-side comparison:

All the errors are significantly smaller. The previous radial pattern is still there, but is much less pronounced.

What happens when we look at the image that showed a poor fit in the corner previously? It was observation 47.

mrcal-show-residuals-board-observation \
  --vectorscale 200                    \
  --circlescale 0.5                    \
  --set 'cbrange [0:2]'                \
  splined.cameramodel                  \
  47

And the residual magnitudes of all the observations:

mrcal-show-residuals                   \
  --magnitudes                         \
  --set 'cbrange [0:1.5]'              \
  splined.cameramodel

Neat! The model fits the data at the edges now. And what about the residual directions?

mrcal-show-residuals    \
  --directions          \
  --unset key           \
  splined.cameramodel

Much better than before: the color clustering is gone. The poorly-fitting ring in the center is still there, but it is being modeled by this splined model far better than how LENSMODEL_OPENCV8 did (error magnitudes are much better). We do still see a clear pattern in the directions inside the ring, so it could still be improved: a denser spline spacing would do better. This is a peculiar image processing artifact of the Sony Alpha 7 III camera: from experience, other cameras using this lens do not exhibit this effect. See the tour of mrcal 2.2 for instance; this used the same lens with a different camera.

We can also visualize the magnitude of the vector field defined by the splined surfaces \(\left| \Delta \vec u \right|\):

mrcal-show-splined-model-correction   \
  --set 'cbrange [0:0.1]'             \
  --unset grid                        \
  --set 'xrange [:] noextend'         \
  --set 'yrange [:] noextend reverse' \
  --set 'key opaque box'              \
  splined.cameramodel

Each X in the plot is a "knot" of the spline surface, a point where a control point value is defined. We're looking at the spline domain, so the axes of the plot are the normalized stereographic projection coordinates \(u_x\) and \(u_y\), and the knots are arranged in a regular grid. The region where the spline surface is well-defined begins at the 2nd knot from the edges; its boundary is shown as a thin green line. The valid-intrinsics region (the area where the intrinsics are confident because we had sufficient chessboard observations there) is shown as a thick, green curve. Since each \(\vec u\) projects to a pixel coordinate \(\vec q\) in some nonlinear way, this curve is not straight.

We want the valid-intrinsics region to lie entirely within the spline-in-bounds region, and we get that here nicely. If some observations did lie outside the spline-in-bounds regions, the projection behavior there would be less flexible than the rest of the model, resulting in less realistic uncertainties, and a larger-fov splined model would be more appropriate. See the lensmodel documentation for more detail.

Alternately, I can look at the spline surface as a function of the pixel coordinates:

mrcal-show-splined-model-correction   \
  --set 'cbrange [0:0.1]'             \
  --imager-domain                     \
  --unset grid                        \
  --set 'xrange [:] noextend'         \
  --set 'yrange [:] noextend reverse' \
  --set 'key opaque box'              \
  splined.cameramodel

Now the valid-intrinsics region is straight, but the spline-in-bounds region is a curve. Projection at the edges is poorly-defined, so the boundary of the spline-in-bounds region appears irregular in this view.

I can also look at the correction vector field:

mrcal-show-splined-model-correction \
  --vectorfield                     \
  --imager-domain                   \
  --unset grid                      \
  --set 'xrange [-300:6300]'        \
  --set 'yrange [3676:-300]'        \
  --set 'key opaque box'            \
  --gridn 40 30                     \
  splined.cameramodel

This doesn't show anything noteworthy in this solve, but seeing it can be informative with other lenses.