Projection uncertainty: mean-frames

The state vector $\overrightarrow{b}$ is a random variable, and we know its distribution. To evaluate the projection uncertainty we want to project a fixed point, to see how this projection $\overrightarrow{q}$ moves around as the chessboards and cameras and intrinsics shift due to the uncertainty in $\overrightarrow{b}$. In other words, we want to project a point defined in the coordinate system of the camera housing, as the origin of the mathematical camera moves around inside this housing:

How do we operate on points in a fixed coordinate system when all the coordinate systems we have are floating random variables? We use the most fixed thing we have: chessboards. As with the camera housing, the chessboards themselves are fixed in space. We have noisy camera observations of the chessboards that implicitly produce estimates of the fixed transformation $T_{{\mathrm{cf}}_i}$ for each chessboard $i$. The explicit transformations that we actually have in $\overrightarrow{b}$ all relate to a floating reference coordinate system: $T_{\mathrm{cr}}$ and $T_{\mathrm{rf}}$. That coordinate system doesn't have any physical meaning, and it's useless in producing our fixed point.

Thus if we project points from a chessboard frame, we would be unaffected by the untethered reference coordinate system. So points in a chessboard frame are somewhat "fixed" for our purposes.

To begin, let's focus on just one chessboard frame: frame 0. We want to know the uncertainty at a pixel coordinate $\overrightarrow{q}$, so let's unproject and transform $\overrightarrow{q}$ out to frame 0:

$${\overrightarrow{p}}_{{\mathrm{frame}}_0}=T_{{\mathrm{f}}_0\mathrm{r}}{T}_{\mathrm{rc}}\mathrm{unproject}\left(\overrightarrow{q}\right)$$

We then transform and project ${\overrightarrow{p}}_{{\mathrm{frame}}_0}$ back to the imager to get ${\overrightarrow{q}}^{+}$. But here we take into account the uncertainties of each transformation to get the desired projection uncertainty $\mathrm{Var}({\overrightarrow{q}}^{+}-\overrightarrow{q})$. The full data flow looks like this, with all the perturbed quantities marked with a $+$ superscript.

$${\overrightarrow{q}}^{+}{\overrightarrow{p}}_{\mathrm{camera}}^{+}{\overrightarrow{b}}_{\mathrm{intrinsics}}^{+}{\overrightarrow{p}}_{{\mathrm{reference}}_0}^{+}{T}_{\mathrm{cr}}^{+}{\overrightarrow{p}}_{{\mathrm{frame}}_0}{T}_{{\mathrm{rf}}_0}^{+}{\overrightarrow{p}}_{\mathrm{reference}}{T}_{\mathrm{fr}}{\overrightarrow{p}}_{\mathrm{camera}}{T}_{\mathrm{rc}}\overrightarrow{q}{\overrightarrow{b}}_{\mathrm{intrinsics}}$$

This works, but it depends on ${\overrightarrow{p}}_{{\mathrm{frame}}_0}$ being "fixed". We can do better. We're observing more than one chessboard, and in aggregate all the chessboard frames can represent an even-more "fixed" frame. Currently we take a very simple approach towards combinining the frames: we compute the mean of all the ${\overrightarrow{p}}_{\mathrm{reference}}^{+}$ estimates from each frame. The full data flow then looks like this:

$${\overrightarrow{p}}_{{\mathrm{reference}}_0}^{+}\mathrm{mean}{\overrightarrow{p}}_{{\mathrm{frame}}_0}^{+}{T}_{{\mathrm{rf}}_0}^{+}{\overrightarrow{q}}^{+}{\overrightarrow{p}}_{\mathrm{camera}}^{+}{\overrightarrow{b}}_{\mathrm{intrinsics}}^{+}{\overrightarrow{p}}_{\mathrm{reference}}^{+}{T}_{\mathrm{cr}}^{+}{\overrightarrow{p}}_{{\mathrm{reference}}_1}^{+}\mathrm{mean}{\overrightarrow{p}}_{{\mathrm{frame}}_1}^{+}{T}_{{\mathrm{rf}}_1}^{+}{\overrightarrow{p}}_{\mathrm{reference}}{T}_{{\mathrm{f}}_1\mathrm{r}}{T}_{{\mathrm{f}}_0\mathrm{r}}{T}_{{\mathrm{f}}_2\mathrm{r}}{\overrightarrow{p}}_{\mathrm{camera}}{T}_{\mathrm{rc}}\overrightarrow{q}{\overrightarrow{b}}_{\mathrm{intrinsics}}{\overrightarrow{p}}_{{\mathrm{reference}}_2}^{+}\mathrm{mean}{\overrightarrow{p}}_{{\mathrm{frame}}_2}^{+}{T}_{{\mathrm{rf}}_2}^{+}$$

So to summarize, to compute the projection uncertainty at a pixel $\overrightarrow{q}$ we

1. Unproject $\overrightarrow{q}$ and transform to each chessboard coordinate system to obtain ${\overrightarrow{p}}_{{\mathrm{frame}}_i}$
2. Transform and project back to ${\overrightarrow{q}}^{+}$, using the mean of all the ${\overrightarrow{p}}_{{\mathrm{reference}}_i}$ and taking into account uncertainties

We have ${\overrightarrow{q}}^{+}\left(\overrightarrow{b}\right)=\mathrm{project}\left(T_{\mathrm{cr}}{\mathrm{mean}}_i\left(T_{{\mathrm{rf}}_i}{\overrightarrow{p}}_{{\mathrm{frame}}_i}\right)\right)$ where the transformations $T$ and the intrinsics used in $\mathrm{project}()$ come directly from the optimization state vector $\overrightarrow{b}$. This function can be used in the higher-level projection uncertainty propagation method to compute $\mathrm{Var}\left(\overrightarrow{q}\right)$.

This "mean-frames" uncertainty method works well, but has several issues:

analyses/dancing/dance-study.py           \
    --scan num_far_constant_Nframes_near  \
    --range 2,10                          \
    --Ncameras 1                          \
    --Nframes-near 100                    \
    --observed-pixel-uncertainty 2        \
    --ymax 4                              \
    --uncertainty-at-range-sampled-max 35 \
    --Nscan-samples 4                     \
    --method mean-frames                  \
    opencv8.cameramodel

analyses/dancing/dance-study.py           \
    --scan num_far_constant_Nframes_near  \
    --range 2,10                          \
    --Ncameras 1                          \
    --Nframes-near 100                    \
    --observed-pixel-uncertainty 2        \
    --ymax 4                              \
    --uncertainty-at-range-sampled-max 35 \
    --Nscan-samples 4                     \
    --method cross-reprojection--rrp-Jfp  \
    opencv8.cameramodel