Projection uncertainty: mean-frames

The state vector \(\vec b\) is a random variable, and we know its distribution. To evaluate the projection uncertainty we want to project a fixed point, to see how this projection \(\vec q\) moves around as the chessboards and cameras and intrinsics shift due to the uncertainty in \(\vec b\). In other words, we want to project a point defined in the coordinate system of the camera housing, as the origin of the mathematical camera moves around inside this housing:

How do we operate on points in a fixed coordinate system when all the coordinate systems we have are floating random variables? We use the most fixed thing we have: chessboards. As with the camera housing, the chessboards themselves are fixed in space. We have noisy camera observations of the chessboards that implicitly produce estimates of the fixed transformation \(T_{\mathrm{cf}_i}\) for each chessboard \(i\). The explicit transformations that we actually have in \(\vec b\) all relate to a floating reference coordinate system: \(T_\mathrm{cr}\) and \(T_\mathrm{rf}\). That coordinate system doesn't have any physical meaning, and it's useless in producing our fixed point.

Thus if we project points from a chessboard frame, we would be unaffected by the untethered reference coordinate system. So points in a chessboard frame are somewhat "fixed" for our purposes.

To begin, let's focus on just one chessboard frame: frame 0. We want to know the uncertainty at a pixel coordinate \(\vec q\), so let's unproject and transform \(\vec q\) out to frame 0:

\[ \vec p_{\mathrm{frame}_0} = T_{\mathrm{f}_0\mathrm{r}} T_\mathrm{rc} \mathrm{unproject}\left( \vec q \right) \]

We then transform and project \(\vec p_{\mathrm{frame}_0}\) back to the imager to get \(\vec q^+\). But here we take into account the uncertainties of each transformation to get the desired projection uncertainty \(\mathrm{Var}\left(\vec q^+ - \vec q\right)\). The full data flow looks like this, with all the perturbed quantities marked with a \(+\) superscript.

\[ \xymatrix{ \vec q^+ & & \vec p^+_\mathrm{camera} \ar[ll]_-{\vec b_\mathrm{intrinsics}^+} & \vec p^+_{\mathrm{reference}_0} \ar[l]_-{T^+_\mathrm{cr}} & \vec p_{\mathrm{frame}_0} \ar[l]_-{T^+_{\mathrm{rf}_0}} & \vec p_\mathrm{reference} \ar[l]_-{T_\mathrm{fr}} & \vec p_\mathrm{camera} \ar[l]_-{T_\mathrm{rc}} & & \vec q \ar[ll]_-{\vec b_\mathrm{intrinsics}} } \]

This works, but it depends on \(\vec p_{\mathrm{frame}_0}\) being "fixed". We can do better. We're observing more than one chessboard, and in aggregate all the chessboard frames can represent an even-more "fixed" frame. Currently we take a very simple approach towards combinining the frames: we compute the mean of all the \(\vec p^+_\mathrm{reference}\) estimates from each frame. The full data flow then looks like this:

\[ \xymatrix{ & & & & \vec p^+_{\mathrm{reference}_0} \ar[dl]_{\mathrm{mean}} & \vec p^+_{\mathrm{frame}_0} \ar[l]_-{T^+_{\mathrm{rf}_0}} \\ \vec q^+ & & \vec p^+_\mathrm{camera} \ar[ll]_-{\vec b_\mathrm{intrinsics}^+} & \vec p^+_{\mathrm{reference}} \ar[l]_-{T^+_\mathrm{cr}} & \vec p^+_{\mathrm{reference}_1} \ar[l]_{\mathrm{mean}} & \vec p^+_{\mathrm{frame}_1} \ar[l]_-{T^+_{\mathrm{rf}_1}} & \vec p_\mathrm{reference} \ar[l]_-{T_\mathrm{f_1 r}} \ar[lu]_-{T_\mathrm{f_0 r}} \ar[ld]^-{T_\mathrm{f_2 r}} & \vec p_\mathrm{camera} \ar[l]_-{T_\mathrm{rc}} & & \vec q \ar[ll]_-{\vec b_\mathrm{intrinsics}} \\ & & & & \vec p^+_{\mathrm{reference}_2} \ar[ul]^{\mathrm{mean}} & \vec p^+_{\mathrm{frame}_2} \ar[l]_-{T^+_{\mathrm{rf}_2}} } \]

So to summarize, to compute the projection uncertainty at a pixel \(\vec q\) we

1. Unproject \(\vec q\) and transform to each chessboard coordinate system to obtain \(\vec p_{\mathrm{frame}_i}\)
2. Transform and project back to \(\vec q^+\), using the mean of all the \(\vec p_{\mathrm{reference}_i}\) and taking into account uncertainties

We have \(\vec q^+\left(\vec b\right) = \mathrm{project}\left( T_\mathrm{cr} \, \mathrm{mean}_i \left( T_{\mathrm{rf}_i} \vec p_{\mathrm{frame}_i} \right) \right)\) where the transformations \(T\) and the intrinsics used in \(\mathrm{project}()\) come directly from the optimization state vector \(\vec b\). This function can be used in the higher-level projection uncertainty propagation method to compute \(\mathrm{Var}\left( \vec q \right)\).

This "mean-frames" uncertainty method works well, but has several issues:

analyses/dancing/dance-study.py           \
    --scan num_far_constant_Nframes_near  \
    --range 2,10                          \
    --Ncameras 1                          \
    --Nframes-near 100                    \
    --observed-pixel-uncertainty 2        \
    --ymax 4                              \
    --uncertainty-at-range-sampled-max 35 \
    --Nscan-samples 4                     \
    --method mean-frames                  \
    opencv8.cameramodel

analyses/dancing/dance-study.py           \
    --scan num_far_constant_Nframes_near  \
    --range 2,10                          \
    --Ncameras 1                          \
    --Nframes-near 100                    \
    --observed-pixel-uncertainty 2        \
    --ymax 4                              \
    --uncertainty-at-range-sampled-max 35 \
    --Nscan-samples 4                     \
    --method cross-reprojection--rrp-Jfp  \
    opencv8.cameramodel