| | |
- R_from_quat(*args, **kwargs)
- Convert a rotation defined as a unit quaternion rotation to a rotation matrix
SYNOPSIS
s = np.sin(rotation_magnitude/2.)
c = np.cos(rotation_magnitude/2.)
quat = nps.glue( c, s*rotation_axis, axis = -1)
print(quat.shape)
===>
(4,)
R = mrcal.R_from_quat(quat)
print(R.shape)
===>
(3,3)
This is mostly for compatibility with some old stuff. mrcal doesn't use
quaternions anywhere. Test this thoroughly before using.
This function is broadcast-aware through numpysane.broadcast_define().
The expected inputs have input prototype:
((4,),)
and output prototype
(3, 3)
The first 1 positional arguments will broadcast. The trailing shape of
those arguments must match the input prototype; the leading shape must follow
the standard broadcasting rules. Positional arguments past the first 1 and
all the keyword arguments are passed through untouched.
- R_from_r(r, get_gradients=False, out=None)
- Compute a rotation matrix from a Rodrigues vector
SYNOPSIS
r = rotation_axis * rotation_magnitude
R = mrcal.R_from_r(r)
Given a rotation specified as a Rodrigues vector (a unit rotation axis scaled by
the rotation magnitude, in radians.), converts it to a rotation matrix.
By default this function returns the rotation matrices only. If we also want
gradients, pass get_gradients=True. Logic:
if not get_gradients: return R
else: return (R, dR/dr)
This function supports broadcasting fully.
ARGUMENTS
- r: array of shape (3,). The Rodrigues vector that defines the rotation. This is
a unit rotation axis scaled by the rotation magnitude, in radians
- get_gradients: optional boolean. By default (get_gradients=False) we return an
array of rotation matrices. Otherwise we return a tuple of arrays of rotation
matrices and their gradients.
- out: optional argument specifying the destination. By default, new numpy
array(s) are created and returned. To write the results into existing (and
possibly non-contiguous) arrays, specify them with the 'out' kwarg. If
get_gradients: 'out' is the one numpy array we will write into. Else: 'out' is
a tuple of all the output numpy arrays. If 'out' is given, we return the same
arrays passed in. This is the standard behavior provided by numpysane_pywrap.
RETURNED VALUE
If not get_gradients: we return an array of rotation matrices. Each broadcasted
slice has shape (3,3)
If get_gradients: we return a tuple of arrays containing the rotation matrices
and the gradient (R, dR/dr):
1. The rotation matrix. Each broadcasted slice has shape (3,3). This is a valid
rotation: matmult(R,transpose(R)) = I, det(R) = 1
2. The gradient dR/dr. Each broadcasted slice has shape (3,3,3). The first two
dimensions select the element of R, and the last dimension selects the
element of r
- Rt_from_rt(rt, get_gradients=False, out=None)
- Compute an Rt transformation from a rt transformation
SYNOPSIS
r = rotation_axis * rotation_magnitude
rt = nps.glue(r,t, axis=-1)
print(rt.shape)
===>
(6,)
Rt = mrcal.Rt_from_rt(rt)
print(Rt.shape)
===>
(4,3)
translation = Rt[3,:]
rotation_matrix = Rt[:3,:]
Converts an rt transformation to an Rt transformation. Both specify a rotation
and translation. An Rt transformation is a (4,3) array formed by nps.glue(R,t,
axis=-2) where R is a (3,3) rotation matrix and t is a (3,) translation vector.
An rt transformation is a (6,) array formed by nps.glue(r,t, axis=-1) where r is
a (3,) Rodrigues vector and t is a (3,) translation vector.
Applied to a point x the transformed result is rotate(x)+t. Given a matrix R,
the rotation is defined by a matrix multiplication. x and t are stored as a row
vector (that's how numpy stores 1-dimensional arrays), but the multiplication
works as if x was a column vector (to match linear algebra conventions). See the
docs for mrcal._transform_point_Rt() for more detail.
By default this function returns the Rt transformations only. If we also want
gradients, pass get_gradients=True. Logic:
if not get_gradients: return Rt
else: return (Rt, dR/dr)
Note that the translation gradient isn't returned: it is always the identity
This function supports broadcasting fully.
ARGUMENTS
- rt: array of shape (6,). This vector defines the input transformation. rt[:3]
is a rotation defined as a Rodrigues vector; rt[3:] is a translation.
- get_gradients: optional boolean. By default (get_gradients=False) we return an
array of Rt transformations. Otherwise we return a tuple of arrays of Rt
transformations and their gradients.
- out: optional argument specifying the destination. By default, new numpy
array(s) are created and returned. To write the results into existing (and
possibly non-contiguous) arrays, specify them with the 'out' kwarg. If
get_gradients: 'out' is the one numpy array we will write into. Else: 'out' is
a tuple of all the output numpy arrays. If 'out' is given, we return the same
arrays passed in. This is the standard behavior provided by numpysane_pywrap.
RETURNED VALUE
If not get_gradients: we return the Rt transformation. Each broadcasted slice
has shape (4,3). Rt[:3,:] is a rotation matrix; Rt[3,:] is a translation. The
matrix R is a valid rotation: matmult(R,transpose(R)) = I and det(R) = 1
If get_gradients: we return a tuple of arrays containing the Rt transformation
and the gradient (Rt, dR/dr):
1. The Rt transformation. Each broadcasted slice has shape (4,3,)
2. The gradient dR/dr. Each broadcasted slice has shape (3,3,3). The first two
dimensions select the element of R, and the last dimension selects the
element of r
- align_procrustes_points_Rt01(p0, p1, weights=None)
- Compute a rigid transformation to align two point clouds
SYNOPSIS
print(points0.shape)
===>
(100,3)
print(points1.shape)
===>
(100,3)
Rt01 = mrcal.align_procrustes_points_Rt01(points0, points1)
print( np.sum(nps.norm2(mrcal.transform_point_Rt(Rt01, points1) -
points0)) )
===>
[The fit error from applying the optimal transformation. If the two point
clouds match up, this will be small]
Given two sets of 3D points in numpy arrays of shape (N,3), we find the optimal
rotation, translation to align these sets of points. This is done with a
well-known direct method. See:
- https://en.wikipedia.org/wiki/Orthogonal_Procrustes_problem
- https://en.wikipedia.org/wiki/Kabsch_algorithm
We return a transformation that minimizes the sum 2-norm of the misalignment:
cost = sum( norm2( w[i] (a[i] - transform(b[i])) ))
We return an Rt transformation to map points in set 1 to points in set 0.
A similar computation can be performed to instead align a set of UNIT VECTORS to
compute an optimal rotation matrix R by calling align_procrustes_vectors_R01().
ARGUMENTS
- p0: an array of shape (..., N, 3). Each row is a point in the coordinate
system we're transforming TO
- p1: an array of shape (..., N, 3). Each row is a point in the coordinate
system we're transforming FROM
- weights: optional array of shape (..., N). Specifies the relative weight of
each point. If omitted, all the given points are weighted equally
RETURNED VALUES
The Rt transformation in an array of shape (4,3). We return the optimal
transformation to align the given point clouds. The transformation maps points
TO coord system 0 FROM coord system 1.
- align_procrustes_vectors_R01(v0, v1, weights=None)
- Compute a rotation to align two sets of direction vectors
SYNOPSIS
print(vectors0.shape)
===>
(100,3)
print(vectors1.shape)
===>
(100,3)
R01 = mrcal.align_procrustes_vectors_R01(vectors0, vectors1)
print( np.mean(1. - nps.inner(mrcal.rotate_point_R(R01, vectors1),
vectors0)) )
===>
[The fit error from applying the optimal rotation. If the two sets of
vectors match up, this will be small]
Given two sets of normalized direction vectors in 3D (stored in numpy arrays of
shape (N,3)), we find the optimal rotation to align them. This is done with a
well-known direct method. See:
- https://en.wikipedia.org/wiki/Orthogonal_Procrustes_problem
- https://en.wikipedia.org/wiki/Kabsch_algorithm
We return a rotation that minimizes the weighted sum of the cosine of the
misalignment:
cost = -sum( w[i] inner(a[i], rotate(b[i])) )
We return a rotation to map vectors in set 1 to vectors in set 0.
A similar computation can be performed to instead align a set of POINTS to
compute an optimal transformation Rt by calling align_procrustes_points_Rt01().
ARGUMENTS
- v0: an array of shape (..., N, 3). Each row is a vector in the coordinate
system we're transforming TO
- v1: an array of shape (..., N, 3). Each row is a vector in the coordinate
system we're transforming FROM
- weights: optional array of shape (..., N). Specifies the relative weight of
each vector. If omitted, everything is weighted equally
RETURNED VALUES
The rotation matrix in an array of shape (3,3). We return the optimal rotation
to align the given vector sets. The rotation maps vectors TO coord system 0 FROM
coord system 1.
- annotate_image__valid_intrinsics_region(image, model, color=(0, 0, 255))
- Annotate an image with a model's valid-intrinsics region
SYNOPSIS
model = mrcal.cameramodel('cam0.cameramodel')
image = cv2.imread('image.jpg')
mrcal.annotate_image__valid_intrinsics_region(image, model)
cv2.imwrite('image-annotated.jpg', image)
This function reads a valid-intrinsics region from a given camera model, and
draws it on top of a given image. This is useful to see what parts of a captured
image have reliable intrinsics.
This function modifies the input image.
If the given model has no valid-intrinsics region defined, an exception is
thrown. If the valid-intrinsics region is empty, a solid circle is drawn at the
center.
If we want an interactive plot instead of an annotated image, call
mrcal.show_valid_intrinsics_region() instead.
ARGUMENTS
- model: the mrcal.cameramodel object that contains the valid-intrinsics region
contour
- image: the numpy array containing the image we're annotating. This is both an
input and an output
- color: optional tuple of length 3 indicating the BGR color of the annotation.
Red by default
RETURNED VALUES
None. The input image array is modified
- apply_color_map(array, a_min=None, a_max=None)
- Color-code an array
SYNOPSIS
image = produce_data()
print( image.shape )
===>
(480, 640)
image_colorcoded = mrcal.apply_color_map(image)
print( image_colorcoded.shape )
===>
(480, 640, 3)
print( image_colorcoded.dtype )
===>
dtype('uint8')
cv2.imwrite('data.png', image_colorcoded)
This is very similar to cv2.applyColorMap(); more flexible in several important
ways. Differences:
- Supports arrays of any shape. Most of the time the input is 2-dimensional
images, but this isn't required
- Supports any input data type, NOT limited to 8-bit images like
cv2.applyColorMap()
- Does not support MATLAB color maps. At this time only one color map is
supported (the default Gnuplot color map). Other gnuplot color maps will be
supported in the future.
The color map is applied to each value in the input, each one producing an RGB
row of shape (3,). So output.shape is input.shape + (3,).
The output has dtype=numpy.uint8, so these arrays can be output as images, and
visualized using any image-viewing tools.
At this time only one color map is supported: the default color map used by
gnuplot. Support for others will be added in the future.
ARGUMENTS
- array: input numpy array
- a_min: optional value indicating the lower bound of the values we color map.
All input values outside of the range [a_min,a_max] are clipped. If omitted, I
use array.min()
- a_max: optional value indicating the upper bound of the values we color map.
All input values outside of the range [a_min,a_max] are clipped. If omitted, I
use array.max()
RETURNED VALUE
The color-mapped output array of shape array.shape + (3,) and containing 8-bit
unsigned integers. The last row is the RGB color-mapped values.
- apply_homography(...)
- Apply a homogeneous-coordinate homography to a set of 2D points
SYNOPSIS
print( H.shape )
===> (3,3)
print( q0.shape )
===> (100, 2)
q1 = mrcal.apply_homography(H10, q0)
print( q1.shape )
===> (100, 2)
A homography maps from pixel coordinates observed in one camera to pixel
coordinates in another. For points represented in homogeneous coordinates ((k*x,
k*y, k) to represent a pixel (x,y) for any k) a homography is a linear map H.
Since homogeneous coordinates are unique only up-to-scale, the homography matrix
H is also unique up to scale.
If two pinhole cameras are observing a planar surface, there exists a homography
that relates observations of the plane in the two cameras.
This function supports broadcasting fully.
ARGUMENTS
- H: an array of shape (..., 3,3). This is the homography matrix. This is unique
up-to-scale, so a homography H is functionally equivalent to k*H for any
non-zero scalar k
- q: an array of shape (..., 2). The pixel coordinates we are mapping
RETURNED VALUE
An array of shape (..., 2) containing the pixels q after the homography was
applied
- close_contour(c)
- Close a polyline, if it isn't already closed
SYNOPSIS
print( a.shape )
===>
(5, 2)
print( a[0] )
===>
[844 204]
print( a[-1] )
===>
[886 198]
b = mrcal.close_contour(a)
print( b.shape )
===>
(6, 2)
print( b[0] )
===>
[844 204]
print( b[-2:] )
===>
[[886 198]
[844 204]]
This function works with polylines represented as arrays of shape (N,2). The
polygon represented by such a polyline is "closed" if its first and last points
sit at the same location. This function ingests a polyline, and returns the
corresponding, closed polygon. If the first and last points of the input match,
the input is returned. Otherwise, the first point is appended to the end, and
this extended polyline is returned.
None is accepted as an empty polygon: we return None.
ARGUMENTS
- c: an array of shape (N,2) representing the polyline to be closed. None and
arrays of shape (0,2) are accepted as special cases ("unknown" and "empty"
regions, respectively)
RETURNED VALUE
An array of shape (N,2) representing the closed polygon. The input is returned
if the input was None or has shape (0,2)
- compose_Rt(*Rt, out=None)
- Compose Rt transformations
SYNOPSIS
Rt10 = nps.glue(rotation_matrix10,translation10, axis=-2)
Rt21 = nps.glue(rotation_matrix21,translation21, axis=-2)
Rt32 = nps.glue(rotation_matrix32,translation32, axis=-2)
print(Rt10.shape)
===>
(4,3)
Rt30 = mrcal.compose_Rt( Rt32, Rt21, Rt10 )
print(x0.shape)
===>
(3,)
print( mrcal.transform_point_Rt(Rt30, x0) -
mrcal.transform_point_Rt(Rt32,
mrcal.transform_point_Rt(Rt21,
mrcal.transform_point_Rt(Rt10, x0))) )
===>
0
Given some number (2 or more, presumably) of Rt transformations, returns
their composition. An Rt transformation is a (4,3) array formed by
nps.glue(R,t, axis=-2) where R is a (3,3) rotation matrix and t is a (3,)
translation vector. This transformation is defined by a matrix multiplication
and an addition. x and t are stored as a row vector (that's how numpy stores
1-dimensional arrays), but the multiplication works as if x was a column vector
(to match linear algebra conventions):
transform_point_Rt(Rt, x) = transpose( matmult(Rt[:3,:], transpose(x)) +
transpose(Rt[3,:]) ) =
= matmult(x, transpose(Rt[:3,:])) +
Rt[3,:]
This function supports broadcasting fully, so we can compose lots of
transformations at the same time.
ARGUMENTS
- *Rt: a list of transformations to compose. Usually we'll be composing two
transformations, but any number could be given here. Each broadcasted slice
has shape (4,3).
- out: optional argument specifying the destination. By default, a new numpy
array is created and returned. To write the results into an existing (and
possibly non-contiguous) array, specify it with the 'out' kwarg. If 'out' is
given, we return the same array passed in. This is the standard behavior
provided by numpysane_pywrap.
RETURNED VALUE
An array of composed Rt transformations. Each broadcasted slice has shape (4,3)
- compose_rt(*rt, get_gradients=False, out=None)
- Compose rt transformations
SYNOPSIS
r10 = rotation_axis10 * rotation_magnitude10
r21 = rotation_axis21 * rotation_magnitude21
r32 = rotation_axis32 * rotation_magnitude32
print(rt10.shape)
===>
(6,)
rt30 = mrcal.compose_rt( rt32, rt21, rt10 )
print(x0.shape)
===>
(3,)
print( mrcal.transform_point_rt(rt30, x0) -
mrcal.transform_point_rt(rt32,
mrcal.transform_point_rt(rt21,
mrcal.transform_point_rt(rt10, x0))) )
===>
0
print( [arr.shape for arr in mrcal.compose_rt(rt21,rt10,
get_gradients = True)] )
===>
[(6,), (6,6), (6,6)]
Given some number (2 or more, presumably) of rt transformations, returns their
composition. An rt transformation is a (6,) array formed by nps.glue(r,t,
axis=-1) where r is a (3,) Rodrigues vector and t is a (3,) translation vector.
This transformation is defined by a matrix multiplication and an addition. x and
t are stored as a row vector (that's how numpy stores 1-dimensional arrays), but
the multiplication works as if x was a column vector (to match linear algebra
conventions):
transform_point_rt(rt, x) = transpose( matmult(R_from_r(rt[:3]), transpose(x)) +
transpose(rt[3,:]) ) =
= matmult(x, transpose(R_from_r(rt[:3]))) +
rt[3:]
By default this function returns the composed transformations only. If we also
want gradients, pass get_gradients=True. This is supported ONLY if we have
EXACTLY 2 transformations to compose. Logic:
if not get_gradients: return rt=compose(rt0,rt1)
else: return (rt=compose(rt0,rt1), dr/drt0,dr/drt1)
Note that the poseutils C API returns only
- dr_dr0
- dr_dr1
- dt_dr0
- dt_dt1
because
- dr/dt0 is always 0
- dr/dt1 is always 0
- dt/dr1 is always 0
- dt/dt0 is always the identity matrix
This Python function, however fills in those constants to return the full (and
more convenient) arrays.
This function supports broadcasting fully, so we can compose lots of
transformations at the same time.
ARGUMENTS
- *rt: a list of transformations to compose. Usually we'll be composing two
transformations, but any number could be given here. Each broadcasted slice
has shape (6,)
- get_gradients: optional boolean. By default (get_gradients=False) we return an
array of composed transformations. Otherwise we return a tuple of arrays of
composed transformations and their gradients. Gradient reporting is only
supported when exactly two transformations are given
- out: optional argument specifying the destination. By default, new numpy
array(s) are created and returned. To write the results into existing (and
possibly non-contiguous) arrays, specify them with the 'out' kwarg. If
get_gradients: 'out' is the one numpy array we will write into. Else: 'out' is
a tuple of all the output numpy arrays. If 'out' is given, we return the same
arrays passed in. This is the standard behavior provided by numpysane_pywrap.
RETURNED VALUE
If not get_gradients: we return an array of composed rt transformations. Each
broadcasted slice has shape (4,3)
If get_gradients: we return a tuple of arrays containing the composed
transformations and the gradients (rt=compose(rt0,rt1),
drt/drt0,drt/drt1):
1. The composed transformation. Each broadcasted slice has shape (6,)
2. The gradient drt/dr0. Each broadcasted slice has shape (6,6). The first
dimension selects the element of rt, and the last dimension selects the
element of rt0
3. The gradient drt/drt1. Each broadcasted slice has shape (6,6). The first
dimension selects the element of rt, and the last dimension selects the
element of rt1
- compute_chessboard_corners(Nw, Nh, globs=('*',), corners_cache_vnl=None, jobs=1, exclude_images=set(), extracol='level')
- Compute the chessboard observations and returns them in a usable form
SYNOPSIS
observations, indices_frame_camera, paths = \
mrcal.compute_chessboard_corners(10, 10,
('frame*-cam0.jpg','frame*-cam1.jpg'),
"corners.vnl")
The input to a calibration problem is a set of images of a calibration object
from different angles and positions. This function ingests these images, and
outputs the detected chessboard corner coordinates in a form usable by the mrcal
optimization routines.
The "corners_cache_vnl" argument specifies a file containing cached results of
the chessboard detector. If this file already exists, we don't run the detector,
but just use the contents of the file. Otherwise, we run the detector, and store
the results here.
The "corner cache" file is a vnlog 3 or 4 columns. Each row describes a
chessboard corner. The first 3 columns are
# filename x y
If the 4th column is given, it usually is a 'level' or a 'weight'. It encodes
the confidence we have in that corner, and the exact interpretation is dependent
on the value of the 'extracol' argument. The output of this function is an array
with a weight for each point, so the logic serves to convert the extra column to
a weight.
if extracol == 'level': the 4th column is a decimation level of the detected
corner. If we needed to cut down the image resolution to detect a corner, its
coordinates are known less precisely, and we use that information to weight
the errors appropriately later. We set the output weight = 1/2^level. If the
4th column is '-' or <0, the given point was not detected, and should be
ignored: we set weight = -1
elif extracol == 'weight': the 4th column is already represented as a weight, so
I just copy it to the output. If the 4th column is '-' or <0, the given point
was not detected, and should be ignored: we set weight = -1
else: I hard-code the output weight to 1.0
ARGUMENTS
- Nw, Nh: the width and height of the point grid in the calibration object we're
using
- globs: a list of strings, one per camera, containing globs matching the image
filenames for that camera. The filenames are expected to encode the
instantaneous frame numbers, with identical frame numbers implying
synchronized images. A common scheme is to name an image taken by frame C at
time T "frameT-camC.jpg". Then images frame10-cam0.jpg and frame10-cam1.jpg
are assumed to have been captured at the same moment in time by cameras 0 and
1. With this scheme, if you wanted to calibrate these two cameras together,
you'd pass ('frame*-cam0.jpg','frame*-cam1.jpg') in the "globs" argument.
The "globs" argument may be omitted. In this case all images are mapped to the
same camera.
- corners_cache_vnl: the name of a file to use to read/write the detected
corners; or a python file object to read data from. If the given file exists
or a python file object is given, we read the detections from it, and do not
run the detector. If the given file does NOT exist (which is what happens the
first time), mrgingham will be invoked to compute the corners from the images,
and the results will be written to that file. So the same function call can be
used to both compute the corners initially, and to reuse the pre-computed
corners with subsequent calls. This exists to save time where re-analyzing the
same data multiple times.
- jobs: a GNU-Make style parallelization flag. Indicates how many parallel
processes should be invoked when computing the corners. If given, a numerical
argument is required. This flag does nothing if the corners-cache file already
exists
- exclude_images: a set of filenames to exclude from reported results
- extracol: an optional string, defaulting to 'level'. Selects the
interpretation of the 4th column describing each corner. Valid options are:
- 'level': the 4th column is a decimation level. Level-0 means
'full-resolution', level-1 means 'half-resolution' and so on. I set output
weight = 1/2^level. If the 4th column is '-' or <0, the given point was not
detected, and should be ignored: we set output weight = -1
- 'weight': the 4th column is already a weight; I copy it to the output. If
the 4th column is '-' or <0, the given point was not detected, and should be
ignored: we set output weight = -1
- '' the 4th column should be ignored, and I set the output weight to 1.0
RETURNED VALUES
This function returns a tuple (observations, indices_frame_camera, files_sorted)
- observations: an ordered (N,object_height_n,object_width_n,3) array describing
N board observations where the board has dimensions
(object_height_n,object_width_n) and each point is an (x,y,weight) pixel
observation. A weight<0 means "ignore this point". Incomplete chessboard
observations can be specified in this way.
- indices_frame_camera is an (N,2) array of contiguous, sorted integers where
each observation is (index_frame,index_camera)
- files_sorted is a list of paths of images corresponding to the observations
Note that this assumes we're solving a calibration problem (stationary cameras)
observing a moving object, so this returns indices_frame_camera. It is the
caller's job to convert this into indices_frame_camintrinsics_camextrinsics,
which mrcal.optimize() expects
- corresponding_icam_extrinsics(...)
- Return the icam_extrinsics corresponding to a given icam_intrinsics
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
icam_intrinsics = m.icam_intrinsics()
icam_extrinsics = \
mrcal.corresponding_icam_extrinsics(icam_intrinsics,
**optimization_inputs)
if icam_extrinsics >= 0:
extrinsics_rt_fromref_at_calibration_time = \
optimization_inputs['extrinsics_rt_fromref'][icam_extrinsics]
When calibrating cameras, each observations is associated with some camera
intrinsics (lens parameters) and some camera extrinsics (geometry). Those two
chunks of data live in different parts of the optimization vector, and are
indexed independently. If we have stationary cameras, then each set of camera
intrinsics is associated with exactly one set of camera extrinsics, and we can
use this function to query this correspondence. If we have moving cameras, then
a single physical camera would have one set of intrinsics but many different
extrinsics, and this function call will throw an exception.
Furthermore, it is possible that a camera's pose is used to define the reference
coordinate system of the optimization. In this case this camera has no explicit
extrinsics (they are an identity transfomration, by definition), and we return
-1, successfully.
In order to determine the camera mapping, we need quite a bit of context. If we
have the full set of inputs to the optimization function, we can pass in those
(as shown in the example above). Or we can pass the individual arguments that
are needed (see ARGUMENTS section for the full list). If the optimization inputs
and explicitly-given arguments conflict about the size of some array, the
explicit arguments take precedence. If any array size is not specified, it is
assumed to be 0. Thus most arguments are optional.
ARGUMENTS
- icam_intrinsics: an integer indicating which camera we're asking about
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- Ncameras_intrinsics
Ncameras_extrinsics
- Nobservations_board
- Nobservations_point
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
- indices_frame_camintrinsics_camextrinsics: array of dims (Nobservations_board,
3). For each observation these are an
(iframe,icam_intrinsics,icam_extrinsics) tuple. icam_extrinsics == -1
means this observation came from a camera in the reference coordinate system.
iframe indexes the "frames_rt_toref" array, icam_intrinsics indexes the
"intrinsics_data" array, icam_extrinsics indexes the "extrinsics_rt_fromref"
array
All of the indices are guaranteed to be monotonic. This array contains 32-bit
integers.
RETURNED VALUE
The integer reporting the index of the camera extrinsics in the optimization
vector. If this camera is at the reference of the coordinate system, return -1
- estimate_joint_frame_poses(calobject_Rt_camera_frame, extrinsics_Rt_fromref, indices_frame_camera, object_width_n, object_height_n, object_spacing)
- Estimate world-referenced poses of the calibration object
SYNOPSIS
print( calobject_Rt_camera_frame.shape )
===>
(123, 4,3)
print( extrinsics_Rt_fromref.shape )
===>
(2, 4,3)
# We have 3 cameras. The first one is at the reference coordinate system,
# the pose estimates of the other two are in this array
print( indices_frame_camera.shape )
===>
(123, 2)
frames_rt_toref = \
mrcal.estimate_joint_frame_poses(calobject_Rt_camera_frame,
extrinsics_Rt_fromref,
indices_frame_camera,
object_width_n, object_height_n,
object_spacing)
print( frames_rt_toref.shape )
===>
(87, 6)
# We have 123 observations of the calibration object by ANY camera. 87
# instances of time when the object was observed. Most of the time it was
# observed by multiple cameras simultaneously, hence 123 > 87
i_observation = 10
iframe,icam = indices_frame_camera[i_observation, :]
# The calibration object in its reference coordinate system
calobject = mrcal.ref_calibration_object(object_width_n,
object_height_n,
object_spacing)
# The estimated calibration object points in the reference coordinate
# system, for this one observation
pref = mrcal.transform_point_rt( frames_rt_toref[iframe],
calobject )
# The estimated calibration object points in the camera coord system. Camera
# 0 is at the reference
if icam >= 1:
pcam = mrcal.transform_point_Rt( extrinsics_Rt_fromref[icam-1],
pref )
else:
pcam = pref
# The pixel observations we would see if the pose estimates were correct
q = mrcal.project(pcam, *models[icam].intrinsics())
# The reprojection error, comparing these hypothesis pixel observations from
# what we actually observed. This should be small
err = q - observations[i_observation][:2]
print( np.linalg.norm(err) )
===>
[something small]
mrcal solves camera calibration problems by iteratively optimizing a nonlinear
least squares problem to bring the pixel observation predictions in line with
actual pixel observations. This requires an initial "seed", an initial estimate
of the solution. This function is a part of that computation. Since this is just
an initial estimate that will be refined, the results of this function do not
need to be exact.
This function ingests an estimate of the camera poses in respect to each other,
and the estimate of the calibration objects in respect to the observing camera.
Most of the time we have simultaneous calibration object observations from
multiple cameras, so this function consolidates all this information to produce
poses of the calibration object in the reference coordinate system, NOT the
observing-camera coordinate system poses we already have.
By convention, we have a "reference" coordinate system that ties the poses of
all the frames (calibration objects) and the cameras together. And by
convention, this "reference" coordinate system is the coordinate system of
camera 0. Thus the array of camera poses extrinsics_Rt_fromref holds Ncameras-1
transformations: the first camera has an identity transformation, by definition.
This function assumes we're observing a moving object from stationary cameras
(i.e. a vanilla camera calibration problem). The mrcal solver is more general,
and supports moving cameras, hence it uses a more general
indices_frame_camintrinsics_camextrinsics array instead of the
indices_frame_camera array used here.
ARGUMENTS
- calobject_Rt_camera_frame: an array of shape (Nobservations,4,3). Each slice
is an Rt transformation TO the observing camera coordinate system FROM the
calibration object coordinate system. This is returned by
estimate_monocular_calobject_poses_Rt_tocam()
- extrinsics_Rt_fromref: an array of shape (Ncameras-1,4,3). Each slice is an Rt
transformation TO the camera coordinate system FROM the reference coordinate
system. By convention camera 0 defines the reference coordinate system, so
that camera's extrinsics are the identity, by definition, and we don't store
that data in this array
- indices_frame_camera: an array of shape (Nobservations,2) and dtype
numpy.int32. Each row (iframe,icam) represents an observation at time
instant iframe of a calibration object by camera icam
- object_width_n: number of horizontal points in the calibration object grid
- object_height_n: number of vertical points in the calibration object grid
- object_spacing: the distance between adjacent points in the calibration
object. A square object is assumed, so the vertical and horizontal distances
are assumed to be identical
RETURNED VALUE
An array of shape (Nframes,6). Each slice represents the pose of the calibration
object at one instant in time: an rt transformation TO the reference coordinate
system FROM the calibration object coordinate system.
- estimate_monocular_calobject_poses_Rt_tocam(indices_frame_camera, observations, object_spacing, models_or_intrinsics)
- Estimate camera-referenced poses of the calibration object from monocular views
SYNOPSIS
print( indices_frame_camera.shape )
===>
(123, 2)
print( observations.shape )
===>
(123, 3)
models = [mrcal.cameramodel(f) for f in ("cam0.cameramodel",
"cam1.cameramodel")]
# Estimated poses of the calibration object from monocular observations
Rt_camera_frame = \
mrcal.estimate_monocular_calobject_poses_Rt_tocam( indices_frame_camera,
observations,
object_spacing,
models)
print( Rt_camera_frame.shape )
===>
(123, 4, 3)
i_observation = 10
icam = indices_frame_camera[i_observation,1]
# The calibration object in its reference coordinate system
calobject = mrcal.ref_calibration_object(object_width_n,
object_height_n,
object_spacing)
# The estimated calibration object points in the observing camera coordinate
# system
pcam = mrcal.transform_point_Rt( Rt_camera_frame[i_observation],
calobject )
# The pixel observations we would see if the calibration object pose was
# where it was estimated to be
q = mrcal.project(pcam, *models[icam].intrinsics())
# The reprojection error, comparing these hypothesis pixel observations from
# what we actually observed. We estimated the calibration object pose from
# the observations, so this should be small
err = q - observations[i_observation][:2]
print( np.linalg.norm(err) )
===>
[something small]
mrcal solves camera calibration problems by iteratively optimizing a nonlinear
least squares problem to bring the pixel observation predictions in line with
actual pixel observations. This requires an initial "seed", an initial estimate
of the solution. This function is a part of that computation. Since this is just
an initial estimate that will be refined, the results of this function do not
need to be exact.
We have pixel observations of a known calibration object, and we want to
estimate the pose of this object in the coordinate system of the camera that
produced these observations. This function ingests a number of such
observations, and solves this "PnP problem" separately for each one. The
observations may come from any lens model; everything is reprojected to a
pinhole model first to work with OpenCV. This function is a wrapper around the
solvePnP() openCV call, which does all the work.
ARGUMENTS
- indices_frame_camera: an array of shape (Nobservations,2) and dtype
numpy.int32. Each row (iframe,icam) represents an observation of a
calibration object by camera icam. iframe is not used by this function
- observations: an array of shape
(Nobservations,object_height_n,object_width_n,3). Each observation corresponds
to a row in indices_frame_camera, and contains a row of shape (3,) for each
point in the calibration object. Each row is (x,y,weight) where x,y are the
observed pixel coordinates. Any point where x<0 or y<0 or weight<0 is ignored.
This is the only use of the weight in this function.
- object_spacing: the distance between adjacent points in the calibration
object. A square object is assumed, so the vertical and horizontal distances
are assumed to be identical. Usually we need the object dimensions in the
object_height_n,object_width_n arguments, but here we get those from the shape
of the observations array
- models_or_intrinsics: either
- a list of mrcal.cameramodel objects from which we use the intrinsics
- a list of (lensmodel,intrinsics_data) tuples
These are indexed by icam from indices_frame_camera
RETURNED VALUE
An array of shape (Nobservations,4,3). Each slice is an Rt transformation TO the
camera coordinate system FROM the calibration object coordinate system.
- hypothesis_corner_positions(icam_intrinsics=None, idx_inliers=None, **optimization_inputs)
- Reports the 3D chessboard points observed by a camera at calibration time
SYNOPSIS
model = mrcal.cameramodel("xxx.cameramodel")
optimization_inputs = model.optimization_inputs()
indices = optimization_inputs['indices_frame_camintrinsics_camextrinsics']
# shape (Nobservations, Nheight, Nwidth, 3)
pcam = mrcal.hypothesis_corner_positions(**optimization_inputs)
# shape (Nobservations,1,1,Nintrinsics)
intrinsics = nps.mv(optimization_inputs['intrinsics'][indices[:,1]],-2,-4)
optimization_inputs['observations_board'][...,:2] = \
mrcal.project( p,
optimization_inputs['lensmodel'],
intrinsics )
# optimization_inputs now contains perfect, noiseless board observations
x = mrcal.optimizer_callback(**optimization_inputs)[1]
print(nps.norm2(x[:mrcal.num_measurements_boards(**optimization_inputs)]))
==>
0
The optimization routine generates hypothetical observations from a set of
parameters being evaluated, trying to match these hypothetical observations to
real observations. To facilitate analysis, this routine returns these
hypothetical coordinates of the chessboard corners being observed. This routine
reports the 3D points in the coordinate system of the observing camera.
The hypothetical points are constructed from
- The calibration object geometry
- The calibration object-reference transformation in
optimization_inputs['frames_rt_toref']
- The camera extrinsics (reference-camera transformation) in
optimization_inputs['extrinsics_rt_fromref']
- The table selecting the camera and calibration object frame for each
observation in
optimization_inputs['indices_frame_camintrinsics_camextrinsics']
This function knows to return 3 types of output:
- ALL the points observed by ALL cameras together (returned always)
- The points observed by only a specific camera, inliers only (returned if
icam_intrinsics is not None)
- The points observed by only a specific camera, outliers only (returned if
icam_intrinsics is not None)
ARGUMENTS
- icam_intrinsics: optional integer specifying which intrinsic camera in the
optimization_inputs we're looking at. If omitted (or None), I return a single
numpy array containing the points for all the cameras. Otherwise I return a
3-tuple with this array in the first element, and the camera-specific arrays
in the last two elements
- idx_inliers: optional numpy array of booleans of shape
(Nobservations,object_height,object_width) to select the outliers manually. If
omitted (or None), the outliers are selected automatically: idx_inliers =
observations_board[...,2] > 0. This argument is available to pick common
inliers from two separate solves.
- **optimization_inputs: a dict() of arguments passable to mrcal.optimize() and
mrcal.optimizer_callback(). We use the geometric data. This dict is obtainable
from a cameramodel object by calling cameramodel.optimization_inputs()
RETURNED VALUE
if icam_intrinsics is None: returns only the array containing ALL the points
observed by ALL cameras. Otherwise returns a tuple, with that array as the first
element:
- An array of shape (Nobservations, Nheight, Nwidth, 3) containing the
coordinates (in the coordinate system of each camera) of the chessboard
corners. These correspond to the observations in
optimization_inputs['observations_board'], which also have shape
(Nobservations, Nheight, Nwidth, 3)
- an (N,3) array containing camera-frame 3D points observed at calibration time,
and accepted by the solver as inliers. Returned only if icam_intrinsics is not
None
- an (N,3) array containing camera-frame 3D points observed at calibration time,
but rejected by the solver as outliers. Returned only if icam_intrinsics is
not None
- identity_R(...)
- Return an identity rotation matrix
SYNOPSIS
print( mrcal.identity_R() )
===>
[[1. 0. 0.]
[0. 1. 0.]
[0. 0. 1.]]
As with all the poseutils functions, the output can be written directly into a
(possibly-non-contiguous) array, by specifying the destination in the 'out'
kwarg
- identity_Rt(...)
- Return an identity Rt transformation
SYNOPSIS
print( mrcal.identity_Rt() )
===>
[[1. 0. 0.]
[0. 1. 0.]
[0. 0. 1.]
[0. 0. 0.]]
As with all the poseutils functions, the output can be written directly into a
(possibly-non-contiguous) array, by specifying the destination in the 'out'
kwarg
- identity_r(...)
- Return an identity Rodrigues rotation
SYNOPSIS
print( mrcal.identity_r() )
===>
[0. 0. 0.]
As with all the poseutils functions, the output can be written directly into a
(possibly-non-contiguous) array, by specifying the destination in the 'out'
kwarg
- identity_rt(...)
- Return an identity rt transformation
SYNOPSIS
print( mrcal.identity_rt() )
===>
[0. 0. 0. 0. 0. 0.]
As with all the poseutils functions, the output can be written directly into a
(possibly-non-contiguous) array, by specifying the destination in the 'out'
kwarg
- image_transformation_map(model_from, model_to, use_rotation=False, plane_n=None, plane_d=None)
- Compute a reprojection map between two models
SYNOPSIS
model_orig = mrcal.cameramodel("xxx.cameramodel")
image_orig = cv2.imread("image.jpg")
model_pinhole = mrcal.pinhole_model_for_reprojection(model_orig,
fit = "corners")
mapxy = mrcal.image_transformation_map(model_orig, model_pinhole)
image_undistorted = mrcal.transform_image(image_orig, mapxy)
# image_undistorted is now a pinhole-reprojected version of image_orig
Returns the transformation that describes a mapping
- from pixel coordinates of an image of a scene observed by model_to
- to pixel coordinates of an image of the same scene observed by model_from
This transformation can then be applied to a whole image by calling
mrcal.transform_image().
This function returns a transformation map in an (Nheight,Nwidth,2) array. The
image made by model_to will have shape (Nheight,Nwidth). Each pixel (x,y) in
this image corresponds to a pixel mapxy[y,x,:] in the image made by model_from.
This function has 3 modes of operation:
- intrinsics-only
This is the default. Selected if
- use_rotation = False
- plane_n = None
- plane_d = None
All of the extrinsics are ignored. If the two cameras have the same
orientation, then their observations of infinitely-far-away objects will line
up exactly
- rotation
This can be selected explicitly with
- use_rotation = True
- plane_n = None
- plane_d = None
Here we use the rotation component of the relative extrinsics. The relative
translation is impossible to use without knowing what we're looking at, so IT
IS ALWAYS IGNORED. If the relative orientation in the models matches reality,
then the two cameras' observations of infinitely-far-away objects will line up
exactly
- plane
This is selected if
- use_rotation = True
- plane_n is not None
- plane_d is not None
We map observations of a given plane in camera FROM coordinates
coordinates to where this plane would be observed by camera TO. This uses
ALL the intrinsics, extrinsics and the plane representation. If all of
these are correct, the observations of this plane would line up exactly in
the remapped-camera-fromimage and the camera-to image. The plane is
represented in camera-from coordinates by a normal vector plane_n, and the
distance to the normal plane_d. The plane is all points p such that
inner(p,plane_n) = plane_d. plane_n does not need to be normalized; any
scaling is compensated in plane_d.
ARGUMENTS
- model_from: the mrcal.cameramodel object describing the camera used to capture
the input image
- model_to: the mrcal.cameramodel object describing the camera that would have
captured the image we're producing
- use_rotation: optional boolean, defaulting to False. If True: we respect the
relative rotation in the extrinsics of the camera models.
- plane_n: optional numpy array of shape (3,); None by default. If given, we
produce a transformation to map observations of a given plane to the same
pixels in the source and target images. This argument describes the normal
vector in the coordinate system of model_from. The plane is all points p such
that inner(p,plane_n) = plane_d. plane_n does not need to be normalized; any
scaling is compensated in plane_d. If given, plane_d should be given also, and
use_rotation should be True. if given, we use the full intrinsics and
extrinsics of both camera models
- plane_d: optional floating-point valud; None by default. If given, we produce
a transformation to map observations of a given plane to the same pixels in
the source and target images. The plane is all points p such that
inner(p,plane_n) = plane_d. plane_n does not need to be normalized; any
scaling is compensated in plane_d. If given, plane_n should be given also, and
use_rotation should be True. if given, we use the full intrinsics and
extrinsics of both camera models
RETURNED VALUE
A numpy array of shape (Nheight,Nwidth,2) where Nheight and Nwidth represent the
imager dimensions of model_to. This array contains 32-bit floats, as required by
cv2.remap() (the function providing the internals of mrcal.transform_image()).
This array can be passed to mrcal.transform_image()
- imagergrid_using(imagersize, gridn_width, gridn_height=None)
- Get a 'using' expression for imager colormap plots
SYNOPSIS
import gnuplotlib as gp
import mrcal
...
Nwidth = 60
Nheight = 40
# shape (Nheight,Nwidth,3)
v,_ = \
mrcal.sample_imager_unproject(Nw, Nh,
*model.imagersize(),
*model.intrinsics())
# shape (Nheight,Nwidth)
f = interesting_quantity(v)
gp.plot(f,
tuplesize = 3,
ascii = True,
using = mrcal.imagergrid_using(model.imagersize, Nw, Nh),
square = True,
_with = 'image')
We often want to plot some quantity at every location on the imager (intrinsics
uncertainties for instance). This is done by gridding the imager, computing the
quantity at each grid point, and sending this to gnuplot. This involves a few
non-obvious plotting idioms, with the full usage summarized in the above
example.
Due to peculiarities of gnuplot, the 'using' expression produced by this
function can only be used in plots using ascii data commands (i.e. pass
'ascii=True' to gnuplotlib).
ARGUMENTS
- imagersize: a (width,height) tuple for the size of the imager. With a
mrcal.cameramodel object this is model.imagersize()
- gridn_width: how many points along the horizontal gridding dimension
- gridn_height: how many points along the vertical gridding dimension. If
omitted or None, we compute an integer gridn_height to maintain a square-ish
grid: gridn_height/gridn_width ~ imager_height/imager_width
RETURNED VALUE
The 'using' string.
- implied_Rt10__from_unprojections(q0, p0, v1, weights=None, atinfinity=True, focus_center=array([0., 0.]), focus_radius=100000000.0)
- Compute the implied-by-the-intrinsics transformation to fit two cameras' projections
SYNOPSIS
models = ( mrcal.cameramodel('cam0-dance0.cameramodel'),
mrcal.cameramodel('cam0-dance1.cameramodel') )
lensmodels = [model.intrinsics()[0] for model in models]
intrinsics_data = [model.intrinsics()[1] for model in models]
# v shape (...,Ncameras,Nheight,Nwidth,...)
# q0 shape (..., Nheight,Nwidth,...)
v,q0 = \
mrcal.sample_imager_unproject(60, None,
*models[0].imagersize(),
lensmodels, intrinsics_data,
normalize = True)
implied_Rt10 = \
mrcal.implied_Rt10__from_unprojections(q0, v[0,...], v[1,...])
q1 = mrcal.project( mrcal.transform_point_Rt(implied_Rt10, v[0,...]),
*models[1].intrinsics())
projection_diff = q1 - q0
When comparing projections from two lens models, it is usually necessary to
align the geometry of the two cameras, to cancel out any transformations implied
by the intrinsics of the lenses. This transformation is computed by this
function, used primarily by mrcal.show_projection_diff() and the
mrcal-show-projection-diff tool.
What are we comparing? We project the same world point into the two cameras, and
report the difference in projection. Usually, the lens intrinsics differ a bit,
and the implied origin of the camera coordinate systems and their orientation
differ also. These geometric uncertainties are baked into the intrinsics. So
when we project "the same world point" we must apply a geometric transformation
to compensate for the difference in the geometry of the two cameras. This
transformation is unknown, but we can estimate it by fitting projections across
the imager: the "right" transformation would result in apparent low projection
diffs in a wide area.
The primary inputs are unprojected gridded samples of the two imagers, obtained
with something like mrcal.sample_imager_unproject(). We grid the two imagers,
and produce normalized observation vectors for each grid point. We pass the
pixel grid from camera0 in q0, and the two unprojections in p0, v1. This
function then tries to find a transformation to minimize
norm2( project(camera1, transform(p0)) - q1 )
We return an Rt transformation to map points in the camera0 coordinate system to
the camera1 coordinate system. Some details about this general formulation are
significant:
- The subset of points we use for the optimization
- What kind of transformation we use
In most practical usages, we would not expect a good fit everywhere in the
imager: areas where no chessboards were observed will not fit well, for
instance. From the point of view of the fit we perform, those ill-fitting areas
should be treated as outliers, and they should NOT be a part of the solve. How
do we specify the well-fitting area? The best way is to use the model
uncertainties to pass the weights in the "weights" argument (see
show_projection_diff() for an implementation). If uncertainties aren't
available, or if we want a faster solve, the focus region can be passed in the
focus_center, focus_radius arguments. By default, these are set to encompass the
whole imager, since the uncertainties would take care of everything, but without
uncertainties (weights = None), these should be set more discriminately. It is
possible to pass both a focus region and weights, but it's probably not very
useful.
Unlike the projection operation, the diff operation is NOT invariant under
geometric scaling: if we look at the projection difference for two points at
different locations along a single observation ray, there will be a variation in
the observed diff. This is due to the geometric difference in the two cameras.
If the models differed only in their intrinsics parameters, then this would not
happen. Thus this function needs to know how far from the camera it should look.
By default (atinfinity = True) we look out to infinity. In this case, p0 is
expected to contain unit vectors. To use any other distance, pass atinfinity =
False, and pass POINTS in p0 instead of just observation directions. v1 should
always be normalized. Generally the most confident distance will be where the
chessboards were observed at calibration time.
Practically, it is very easy for the unprojection operation to produce nan or
inf values. And the weights could potentially have some invalid values also.
This function explicitly checks for such illegal data in p0, v1 and weights, and
ignores those points.
ARGUMENTS
- q0: an array of shape (Nh,Nw,2). Gridded pixel coordinates covering the imager
of both cameras
- p0: an array of shape (...,Nh,Nw,3). An unprojection of q0 from camera 0. If
atinfinity, this should contain unit vectors, else it should contain points in
space at the desired distance from the camera. This array may have leading
dimensions that are all used in the fit. These leading dimensions correspond
to those in the "weights" array
- v1: an array of shape (Nh,Nw,3). An unprojection of q0 from camera 1. This
should always contain unit vectors, regardless of the value of atinfinity
- weights: optional array of shape (...,Nh,Nw); None by default. If given, these
are used to weigh each fitted point differently. Usually we use the projection
uncertainties to apply a stronger weight to more confident points. If omitted
or None, we weigh each point equally. This array may have leading dimensions
that are all used in the fit. These leading dimensions correspond to those in
the "p0" array
- atinfinity: optional boolean; True by default. If True, we're looking out to
infinity, and I compute a rotation-only fit; a full Rt transformation is still
returned, but Rt[3,:] is 0; p0 should contain unit vectors. If False, I'm
looking out to a finite distance, and p0 should contain 3D points specifying
the positions of interest.
- focus_center: optional array of shape (2,); (0,0) by default. Used to indicate
that we're interested only in a subset of pixels q0, a distance focus_radius
from focus_center. By default focus_radius is LARGE, so we use all the points.
This is intended to be used if no uncertainties are available, and we need to
manually select the focus region.
- focus_radius: optional value; LARGE by default. Used to indicate that we're
interested only in a subset of pixels q0, a distance focus_radius from
focus_center. By default focus_radius is LARGE, so we use all the points. This
is intended to be used if no uncertainties are available, and we need to
manually select the focus region.
RETURNED VALUE
An array of shape (4,3), representing an Rt transformation from camera0 to
camera1. If atinfinity then we're computing a rotation-fit only, but we still
report a full Rt transformation with the t component set to 0
- ingest_packed_state(p_packed, **optimization_inputs)
- Read a given packed state into optimization_inputs
SYNOPSIS
# A simple gradient check
model = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = model.optimization_inputs()
p0,x0,J = mrcal.optimizer_callback(no_factorization = True,
**optimization_inputs)[:3]
dp = np.random.randn(len(p0)) * 1e-9
mrcal.ingest_packed_state(p0 + dp,
**optimization_inputs)
x1 = mrcal.optimizer_callback(no_factorization = True,
no_jacobian = True,
**optimization_inputs)[1]
dx_observed = x1 - x0
dx_predicted = nps.inner(J, dp_packed)
This is the converse of mrcal.optimizer_callback(). One thing
mrcal.optimizer_callback() does is to convert the expanded (intrinsics,
extrinsics, ...) arrays into a 1-dimensional scaled optimization vector
p_packed. mrcal.ingest_packed_state() allows updates to p_packed to be absorbed
back into the (intrinsics, extrinsics, ...) arrays for further evaluation with
mrcal.optimizer_callback() and others.
ARGUMENTS
- p_packed: a numpy array of shape (Nstate,) containing the input packed state
- **optimization_inputs: a dict() of arguments passable to mrcal.optimize() and
mrcal.optimizer_callback(). The arrays in this dict are updated
RETURNED VALUE
None
- invert_Rt(Rt, out=None)
- Invert an Rt transformation
SYNOPSIS
Rt01 = nps.glue(rotation_matrix,translation, axis=-2)
print(Rt01.shape)
===>
(4,3)
Rt10 = mrcal.invert_Rt(Rt01)
print(x1.shape)
===>
(3,)
x0 = mrcal.transform_point_Rt(Rt01, x1)
print( nps.norm2( x1 - \
mrcal.transform_point_Rt(Rt10, x0) ))
===>
0
Given an Rt transformation to convert a point representated in coordinate system
1 to coordinate system 0 (let's call it Rt01), returns a transformation that
does the reverse: converts a representation in coordinate system 0 to coordinate
system 1 (let's call it Rt10).
Thus if you have a point in coordinate system 1 (let's call it x1), we can
convert it to a representation in system 0, and then back. And we'll get the
same thing out:
x1 == mrcal.transform_point_Rt( mrcal.invert_Rt(Rt01),
mrcal.transform_point_Rt( Rt01, x1 ))
An Rt transformation represents a rotation and a translation. It is a (4,3)
array formed by nps.glue(R,t, axis=-2) where R is a (3,3) rotation matrix and t
is a (3,) translation vector.
Applied to a point x the transformed result is rotate(x)+t. Given a matrix R,
the rotation is defined by a matrix multiplication. x and t are stored as a row
vector (that's how numpy stores 1-dimensional arrays), but the multiplication
works as if x was a column vector (to match linear algebra conventions). See the
docs for mrcal._transform_point_Rt() for more detail.
This function supports broadcasting fully.
ARGUMENTS
- Rt: array of shape (4,3). This matrix defines the transformation. Rt[:3,:] is
a rotation matrix; Rt[3,:] is a translation. It is assumed that the rotation
matrix is a valid rotation (matmult(R,transpose(R)) = I, det(R) = 1), but that
is not checked
- out: optional argument specifying the destination. By default, a new numpy
array is created and returned. To write the results into an existing (and
possibly non-contiguous) array, specify it with the 'out' kwarg. If 'out' is
given, we return the same array passed in. This is the standard behavior
provided by numpysane_pywrap.
RETURNED VALUE
The inverse Rt transformation in an array of shape (4,3).
- invert_rt(rt, get_gradients=False, out=None)
- Invert an rt transformation
SYNOPSIS
r = rotation_axis * rotation_magnitude
rt01 = nps.glue(r,t, axis=-1)
print(rt01.shape)
===>
(6,)
rt10 = mrcal.invert_rt(rt01)
print(x1.shape)
===>
(3,)
x0 = mrcal.transform_point_rt(rt01, x1)
print( nps.norm2( x1 -
mrcal.transform_point_rt(rt10, x0) ))
===>
0
Given an rt transformation to convert a point representated in coordinate system
1 to coordinate system 0 (let's call it rt01), returns a transformation that
does the reverse: converts a representation in coordinate system 0 to coordinate
system 1 (let's call it rt10).
Thus if you have a point in coordinate system 1 (let's call it x1), we can
convert it to a representation in system 0, and then back. And we'll get the
same thing out:
x1 == mrcal.transform_point_rt( mrcal.invert_rt(rt01),
mrcal.transform_point_rt( rt01, x1 ))
An rt transformation represents a rotation and a translation. It is a (6,) array
formed by nps.glue(r,t, axis=-1) where r is a (3,) Rodrigues vector and t is a
(3,) translation vector.
Applied to a point x the transformed result is rotate(x)+t. x and t are stored
as a row vector (that's how numpy stores 1-dimensional arrays). See the docs for
mrcal._transform_point_rt() for more detail.
By default this function returns the rt transformation only. If we also want
gradients, pass get_gradients=True. Logic:
if not get_gradients: return rt
else: return (rt, drtout_drtin)
Note that the poseutils C API returns only
- dtout_drin
- dtout_dtin
because
- drout_drin is always -I
- drout_dtin is always 0
This Python function, however fills in those constants to return the full (and
more convenient) arrays.
This function supports broadcasting fully.
ARGUMENTS
- rt: array of shape (6,). This vector defines the input transformation. rt[:3]
is a rotation defined as a Rodrigues vector; rt[3:] is a translation.
- get_gradients: optional boolean. By default (get_gradients=False) we return an
array of rt translation. Otherwise we return a tuple of arrays of rt
translations and their gradients.
- out: optional argument specifying the destination. By default, new numpy
array(s) are created and returned. To write the results into existing (and
possibly non-contiguous) arrays, specify them with the 'out' kwarg. If
get_gradients: 'out' is the one numpy array we will write into. Else: 'out' is
a tuple of all the output numpy arrays. If 'out' is given, we return the same
arrays passed in. This is the standard behavior provided by numpysane_pywrap.
RETURNED VALUE
If not get_gradients: we return an array of rt transformation(s). Each
broadcasted slice has shape (6,)
If get_gradients: we return a tuple of arrays containing the rt transformation(s)
and the gradients (rt, drtout/drtin)
1. The rt transformation. Each broadcasted slice has shape (6,)
2. The gradient drtout/drtin. Each broadcasted slice has shape (6,6). The first
dimension selects elements of rtout, and the last dimension selects elements
of rtin
- is_within_valid_intrinsics_region(q, model)
- Which of the pixel coordinates fall within the valid-intrinsics region?
SYNOPSIS
mask = mrcal.is_within_valid_intrinsics_region(q, model)
q_trustworthy = q[mask]
mrcal camera models may have an estimate of the region of the imager where the
intrinsics are trustworthy (originally computed with a low-enough error and
uncertainty). When using a model, we may want to process points that fall
outside of this region differently from points that fall within this region.
This function returns a mask that indicates whether each point is within the
region or not.
If no valid-intrinsics region is defined in the model, returns None.
ARGUMENTS
- q: an array of shape (..., 2) of pixel coordinates
- model: the model we're interrogating
RETURNED VALUE
The mask that indicates whether each point is within the region
- knots_for_splined_models(...)
- Return a tuple of locations of x and y spline knots
SYNOPSIS
print(mrcal.knots_for_splined_models('LENSMODEL_SPLINED_STEREOGRAPHIC_order=2_Nx=4_Ny=3_fov_x_deg=200'))
( array([-3.57526078, -1.19175359, 1.19175359, 3.57526078]),
array([-2.38350719, 0. , 2.38350719]))
Splined models are defined by the locations of their control points. These are
arranged in a grid, the size and density of which is set by the model
configuration. This function returns a tuple:
- the locations of the knots along the x axis
- the locations of the knots along the y axis
The values in these arrays correspond to whatever is used to index the splined
surface. In the case of LENSMODEL_SPLINED_STEREOGRAPHIC, these are the
normalized stereographic projection coordinates. These can be unprojected to
observation vectors at the knots:
ux,uy = mrcal.knots_for_splined_models('LENSMODEL_SPLINED_STEREOGRAPHIC_order=2_Nx=4_Ny=3_fov_x_deg=200')
u = np.ascontiguousarray(nps.mv(nps.cat(*np.meshgrid(ux,uy)), 0, -1))
v = mrcal.unproject_stereographic(u, 1,1,0,0)
# v[index_y, index_x] is now an observation vector that will project to this
# knot
ARGUMENTS
- lensmodel: the "LENSMODEL_..." string we're querying. This function only makes
sense for "LENSMODEL_SPLINED_..." models
RETURNED VALUE
A tuple:
- An array of shape (Nx,) representing the knot locations along the x axis
- An array of shape (Ny,) representing the knot locations along the y axis
- lensmodel_metadata(...)
- Returns meta-information about a model
SYNOPSIS
import pprint
pprint.pprint(mrcal.lensmodel_metadata('LENSMODEL_SPLINED_STEREOGRAPHIC_order=3_Nx=16_Ny=14_fov_x_deg=200'))
{'Nx': 16,
'Ny': 14,
'can_project_behind_camera': 1,
'fov_x_deg': 200,
'has_core': 1,
'order': 3}
I support a number of lens models, which have different properties. Some models
have configuration embedded in the model string. This function returns a dict
with the model properties and all the configuration values. At the time of this
writing, the properties that ALL models have are
has_core: True if the first 4 values in the intrinsics vector are the "core":
fx,fy,cx,cy
can_project_behind_camera: True if this model is able to project vectors from
behind the camera. If it cannot, then unproject() will never report z<0
At the time of this writing the only lensmodel that has any configuration is
LENSMODEL_SPLINED_STEREOGRAPHIC_..., but more could be added later.
ARGUMENTS
- lensmodel: the "LENSMODEL_..." string we're querying
RETURNED VALUE
A dict containing all the properties and all the configuration values
- lensmodel_num_params(...)
- Get the number of lens parameters for a particular model type
SYNOPSIS
print(mrcal.lensmodel_num_params('LENSMODEL_OPENCV4'))
8
I support a number of lens models, which have different numbers of parameters.
Given a lens model, this returns how many parameters there are. Some models have
no configuration, and there's a static mapping between the lensmodel string and
the parameter count. Some other models DO have some configuration values inside
the model string (LENSMODEL_SPLINED_STEREOGRAPHIC_... for instance), and the
number of parameters is computed using the configuration values. The lens model
is given as a string such as
LENSMODEL_PINHOLE
LENSMODEL_OPENCV4
LENSMODEL_CAHVOR
LENSMODEL_SPLINED_STEREOGRAPHIC_order=3_Nx=16_Ny=12_fov_x_deg=100
The full list can be obtained with mrcal.supported_lensmodels()
ARGUMENTS
- lensmodel: the "LENSMODEL_..." string we're querying
RETURNED VALUE
An integer number of parameters needed to describe a lens of the given type
- mapping_file_framenocameraindex(*files_per_camera)
- Parse image filenames to get the frame numbers
SYNOPSIS
mapping_file_framenocameraindex = \
mapping_file_framenocameraindex( ('img5-cam2.jpg', 'img6-cam2.jpg'),
('img6-cam3.jpg', 'img7-cam3.jpg'),)
print(mapping_file_framenocameraindex)
===>
{ 'frame5-cam2.jpg': (5, 0),
'frame6-cam2.jpg': (6, 0),
'frame6-cam3.jpg': (6, 1),
'frame7-cam3.jpg': (7, 1) }
Prior to this call we already applied a glob to some images, so we already know
which images belong to which camera. This function further classifies the images
to find the frame number of each image. This is done by looking at the filenames
of images in each camera, removing common prefixes and suffixes, and using the
central varying filename component as the frame number. This varying component
should be numeric. If it isn't and we have multiple cameras, then we barf. If it
isn't, but we only have one camera, we fallback on sequential frame numbers.
If we have just one image for a camera, I can't tell what is constant in the
filenames, so I return framenumber=0.
ARGUMENTS
- *files_per_camera: one argument per camera. Each argument is a list of strings
of filenames of images observed by that camera
RETURNED VALUES
We return a dict from filenames to (framenumber, cameraindex) tuples. The
"cameraindex" is a sequential index counting up from 0. cameraindex==0
corresponds to files_per_camera[0] and so on.
The "framenumber" may not be sequential OR starting from 0: this comes directly
from the filename.
- measurement_index_boards(...)
- Return the measurement index of the start of a given board observation
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
x = mrcal.optimizer_callback(**optimization_inputs)[1]
Nmeas = mrcal.num_measurements_boards ( **optimization_inputs)
i_meas0 = mrcal.measurement_index_boards(0, **optimization_inputs)
x_boards_all = x[i_meas0:i_meas0+Nmeas]
The optimization algorithm tries to minimize the norm of a "measurements" vector
x. The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.measurement_index_...() functions report where particular items end up in
the vector of measurements.
THIS function reports the index in the measurement vector where a particular
board observation begins. When solving calibration problems, most if not all of
the measurements will come from these observations. These are stored
contiguously.
In order to determine the layout, we need quite a bit of context. If we have
the full set of inputs to the optimization function, we can pass in those (as
shown in the example above). Or we can pass the individual arguments that are
needed (see ARGUMENTS section for the full list). If the optimization inputs and
explicitly-given arguments conflict about the size of some array, the explicit
arguments take precedence. If any array size is not specified, it is assumed to
be 0. Thus most arguments are optional.
ARGUMENTS
- i_observation_board: an integer indicating which board observation we're
querying
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_frames
do_optimize_calobject_warp
do_apply_regularization
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
Nobservations_board
Nobservations_point
calibration_object_width_n
calibration_object_height_n
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting the variable index in the measurements vector where the
measurements for this particular board observation start
- measurement_index_points(...)
- Return the measurement index of the start of a given point observation
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
x = mrcal.optimizer_callback(**optimization_inputs)[1]
Nmeas = mrcal.num_measurements_points( **optimization_inputs)
i_meas0 = mrcal.measurement_index_points(0, **optimization_inputs)
x_points_all = x[i_meas0:i_meas0+Nmeas]
The optimization algorithm tries to minimize the norm of a "measurements" vector
x. The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.measurement_index_...() functions report where particular items end up in
the vector of measurements.
THIS function reports the index in the measurement vector where a particular
point observation begins. When solving structure-from-motion problems, most if
not all of the measurements will come from these observations. These are stored
contiguously.
In order to determine the layout, we need quite a bit of context. If we have the
full set of inputs to the optimization function, we can pass in those (as shown
in the example above). Or we can pass the individual arguments that are needed
(see ARGUMENTS section for the full list). If the optimization inputs and
explicitly-given arguments conflict about the size of some array, the explicit
arguments take precedence. If any array size is not specified, it is assumed to
be 0. Thus most arguments are optional.
ARGUMENTS
- i_observation_point: an integer indicating which point observation we're
querying
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_frames
do_optimize_calobject_warp
do_apply_regularization
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
Nobservations_board
Nobservations_point
calibration_object_width_n
calibration_object_height_n
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting the variable index in the measurements vector where the
measurements for this particular point observation start
- measurement_index_regularization(...)
- Return the index of the start of the regularization measurements
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
x = mrcal.optimizer_callback(**optimization_inputs)[1]
Nmeas = mrcal.num_measurements_regularization( **optimization_inputs)
i_meas0 = mrcal.measurement_index_regularization(**optimization_inputs)
x_regularization = x[i_meas0:i_meas0+Nmeas]
The optimization algorithm tries to minimize the norm of a "measurements" vector
x. The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.measurement_index_...() functions report where particular items end up in
the vector of measurements.
THIS function reports the index in the measurement vector where the
regularization terms begin. These don't model physical effects, but guide the
solver away from obviously-incorrect solutions, and resolve ambiguities. This
helps the solver converge to the right solution, quickly.
In order to determine the layout, we need quite a bit of context. If we have the
full set of inputs to the optimization function, we can pass in those (as shown
in the example above). Or we can pass the individual arguments that are needed
(see ARGUMENTS section for the full list). If the optimization inputs and
explicitly-given arguments conflict about the size of some array, the explicit
arguments take precedence. If any array size is not specified, it is assumed to
be 0. Thus most arguments are optional.
ARGUMENTS
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_frames
do_optimize_calobject_warp
do_apply_regularization
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
Nobservations_board
Nobservations_point
calibration_object_width_n
calibration_object_height_n
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting where in the measurement vector the regularization terms
start
- num_measurements(...)
- Return how many measurements we have in the full optimization problem
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
x,J = mrcal.optimizer_callback(**optimization_inputs)[1:3]
Nmeas = mrcal.num_measurements(**optimization_inputs)
print(x.shape[0] - Nmeas)
===>
0
print(J.shape[0] - Nmeas)
===>
0
The optimization algorithm tries to minimize the norm of a "measurements" vector
x. The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.num_measurements_...() functions report where particular items end up in
the vector of measurements.
THIS function reports the total number of measurements we have. This corresponds
to the number of elements in the vector x and to the number of rows in the
jacobian matrix J.
In order to determine the mapping, we need quite a bit of context. If we have
the full set of inputs to the optimization function, we can pass in those (as
shown in the example above). Or we can pass the individual arguments that are
needed (see ARGUMENTS section for the full list). If the optimization inputs and
explicitly-given arguments conflict about the size of some array, the explicit
arguments take precedence. If any array size is not specified, it is assumed to
be 0. Thus most arguments are optional.
ARGUMENTS
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_frames
do_optimize_calobject_warp
do_apply_regularization
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
Nobservations_board
Nobservations_point
calibration_object_width_n
calibration_object_height_n
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting the size of the measurement vector x
- num_measurements_boards(...)
- Return how many measurements we have from calibration object observations
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
x = mrcal.optimizer_callback(**optimization_inputs)[1]
Nmeas = mrcal.num_measurements_boards ( **optimization_inputs)
i_meas0 = mrcal.measurement_index_boards(0, **optimization_inputs)
x_boards_all = x[i_meas0:i_meas0+Nmeas]
The optimization algorithm tries to minimize the norm of a "measurements" vector
x. The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.num_measurements_...() functions report how many measurements are produced
by particular items.
THIS function reports how many measurements come from the observations of the
calibration object. When solving calibration problems, most if not all of the
measurements will come from these observations. These are stored contiguously.
In order to determine the layout, we need quite a bit of context. If we have
the full set of inputs to the optimization function, we can pass in those (as
shown in the example above). Or we can pass the individual arguments that are
needed (see ARGUMENTS section for the full list). If the optimization inputs and
explicitly-given arguments conflict about the size of some array, the explicit
arguments take precedence. If any array size is not specified, it is assumed to
be 0. Thus most arguments are optional.
ARGUMENTS
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_frames
do_optimize_calobject_warp
do_apply_regularization
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
Nobservations_board
Nobservations_point
calibration_object_width_n
calibration_object_height_n
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting how many elements of the measurement vector x come from
the calibration object observations
- num_measurements_points(...)
- Return how many measurements we have from point observations
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
x = mrcal.optimizer_callback(**optimization_inputs)[1]
Nmeas = mrcal.num_measurements_points( **optimization_inputs)
i_meas0 = mrcal.measurement_index_points(0, **optimization_inputs)
x_points_all = x[i_meas0:i_meas0+Nmeas]
The optimization algorithm tries to minimize the norm of a "measurements" vector
x. The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.num_measurements_...() functions report how many measurements are produced
by particular items.
THIS function reports how many measurements come from the observations of
discrete points. When solving structure-from-motion problems, most if not all of
the measurements will come from these observations. These are stored
contiguously.
In order to determine the layout, we need quite a bit of context. If we have the
full set of inputs to the optimization function, we can pass in those (as shown
in the example above). Or we can pass the individual arguments that are needed
(see ARGUMENTS section for the full list). If the optimization inputs and
explicitly-given arguments conflict about the size of some array, the explicit
arguments take precedence. If any array size is not specified, it is assumed to
be 0. Thus most arguments are optional.
ARGUMENTS
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_frames
do_optimize_calobject_warp
do_apply_regularization
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
Nobservations_board
Nobservations_point
calibration_object_width_n
calibration_object_height_n
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting how many elements of the measurement vector x come from
observations of discrete points
- num_measurements_regularization(...)
- Return how many measurements we have from regularization
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
x = mrcal.optimizer_callback(**optimization_inputs)[1]
Nmeas = mrcal.num_measurements_regularization( **optimization_inputs)
i_meas0 = mrcal.measurement_index_regularization(**optimization_inputs)
x_regularization = x[i_meas0:i_meas0+Nmeas]
The optimization algorithm tries to minimize the norm of a "measurements" vector
x. The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.num_measurements_...() functions report where particular items end up in
the vector of measurements.
THIS function reports how many measurements come from the regularization terms
of the optimization problem. These don't model physical effects, but guide the
solver away from obviously-incorrect solutions, and resolve ambiguities. This
helps the solver converge to the right solution, quickly.
In order to determine the layout, we need quite a bit of context. If we have the
full set of inputs to the optimization function, we can pass in those (as shown
in the example above). Or we can pass the individual arguments that are needed
(see ARGUMENTS section for the full list). If the optimization inputs and
explicitly-given arguments conflict about the size of some array, the explicit
arguments take precedence. If any array size is not specified, it is assumed to
be 0. Thus most arguments are optional.
ARGUMENTS
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_frames
do_optimize_calobject_warp
do_apply_regularization
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
Nobservations_board
Nobservations_point
calibration_object_width_n
calibration_object_height_n
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting how many elements of the measurement vector x come from
regularization terms
- num_states_calobject_warp(...)
- Get the number of parameters in the optimization vector for the board warp
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
p = mrcal.optimizer_callback(**optimization_inputs)[0]
mrcal.unpack_state(p, **optimization_inputs)
i_state0 = mrcal.state_index_calobject_warp(**optimization_inputs)
Nstates = mrcal.num_states_calobject_warp (**optimization_inputs)
calobject_warp = p[i_state0:i_state0+Nstates]
The optimization algorithm sees its world described in one, big vector of state.
The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.num_states_...() functions report how many variables in the optimization
vector are taken up by each particular kind of measurement.
THIS function reports how many variables are used to represent the
calibration-object warping. This is stored contiguously as in memory. These
warping parameters describe how the observed calibration object differs from the
expected calibration object. There will always be some difference due to
manufacturing tolerances and temperature and humidity effects.
In order to determine the variable mapping, we need quite a bit of context. If
we have the full set of inputs to the optimization function, we can pass in
those (as shown in the example above). Or we can pass the individual arguments
that are needed (see ARGUMENTS section for the full list). If the optimization
inputs and explicitly-given arguments conflict about the size of some array, the
explicit arguments take precedence. If any array size is not specified, it is
assumed to be 0. Thus most arguments are optional.
ARGUMENTS
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_calobject_warp
do_optimize_frames
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
Nobservations_board
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting the variable count of the calibration object warping
parameters
- num_states_extrinsics(...)
- Get the number of extrinsics parameters in the optimization vector
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
p = mrcal.optimizer_callback(**optimization_inputs)[0]
mrcal.unpack_state(p, **optimization_inputs)
i_state0 = mrcal.state_index_extrinsics(0, **optimization_inputs)
Nstates = mrcal.num_states_extrinsics ( **optimization_inputs)
extrinsics_rt_fromref_all = p[i_state0:i_state0+Nstates]
The optimization algorithm sees its world described in one, big vector of state.
The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.num_states_...() functions report how many variables in the optimization
vector are taken up by each particular kind of measurement.
THIS function reports how many variables are used to represent ALL the camera
extrinsics. The extrinsics are stored contiguously as an "rt transformation": a
3-element rotation represented as a Rodrigues vector followed by a 3-element
translation. These transform points represented in the reference coordinate
system to the coordinate system of the specific camera. Note that mrcal allows
the reference coordinate system to be tied to a particular camera. In this case
the extrinsics of that camera do not appear in the state vector at all, and
icam_extrinsics == -1 in the indices_frame_camintrinsics_camextrinsics
array.
In order to determine the variable mapping, we need quite a bit of context. If
we have the full set of inputs to the optimization function, we can pass in
those (as shown in the example above). Or we can pass the individual arguments
that are needed (see ARGUMENTS section for the full list). If the optimization
inputs and explicitly-given arguments conflict about the size of some array, the
explicit arguments take precedence. If any array size is not specified, it is
assumed to be 0. Thus most arguments are optional.
ARGUMENTS
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_calobject_warp
do_optimize_frames
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting the variable count of extrinsics in the state vector
- num_states_frames(...)
- Get the number of calibration object pose parameters in the optimization vector
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
p = mrcal.optimizer_callback(**optimization_inputs)[0]
mrcal.unpack_state(p, **optimization_inputs)
i_state0 = mrcal.state_index_frames(0, **optimization_inputs)
Nstates = mrcal.num_states_frames ( **optimization_inputs)
frames_rt_toref_all = p[i_state0:i_state0+Nstates]
The optimization algorithm sees its world described in one, big vector of state.
The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.num_states_...() functions report how many variables in the optimization
vector are taken up by each particular kind of measurement.
THIS function reports how many variables are used to represent ALL the frame
poses. Here a "frame" is a pose of the observed calibration object at some
instant in time. The frames are stored contiguously as an "rt transformation": a
3-element rotation represented as a Rodrigues vector followed by a 3-element
translation. These transform points represented in the internal calibration
object coordinate system to the reference coordinate system.
In order to determine the variable mapping, we need quite a bit of context. If
we have the full set of inputs to the optimization function, we can pass in
those (as shown in the example above). Or we can pass the individual arguments
that are needed (see ARGUMENTS section for the full list). If the optimization
inputs and explicitly-given arguments conflict about the size of some array, the
explicit arguments take precedence. If any array size is not specified, it is
assumed to be 0. Thus most arguments are optional.
ARGUMENTS
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_calobject_warp
do_optimize_frames
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting the variable count of frames in the state vector
- num_states_intrinsics(...)
- Get the number of intrinsics parameters in the optimization vector
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
p = mrcal.optimizer_callback(**optimization_inputs)[0]
mrcal.unpack_state(p, **optimization_inputs)
i_state0 = mrcal.state_index_intrinsics(0, **optimization_inputs)
Nstates = mrcal.num_states_intrinsics ( **optimization_inputs)
intrinsics_all = p[i_state0:i_state0+Nstates]
The optimization algorithm sees its world described in one, big vector of state.
The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.num_states_...() functions report how many variables in the optimization
vector are taken up by each particular kind of measurement.
THIS function reports how many variables are used to represent ALL the camera
intrinsics. The intrinsics are stored contiguously. They consist of a 4-element
"intrinsics core" (focallength-x, focallength-y, centerpixel-x, centerpixel-y)
followed by a lensmodel-specific vector of "distortions". The number of
intrinsics elements (including the core) for a particular lens model can be
queried with mrcal.lensmodel_num_params(lensmodel). Note that
do_optimize_intrinsics_core and do_optimize_intrinsics_distortions can be used
to lock down one or both of those quantities, which would omit them from the
optimization vector.
In order to determine the variable mapping, we need quite a bit of context. If
we have the full set of inputs to the optimization function, we can pass in
those (as shown in the example above). Or we can pass the individual arguments
that are needed (see ARGUMENTS section for the full list). If the optimization
inputs and explicitly-given arguments conflict about the size of some array, the
explicit arguments take precedence. If any array size is not specified, it is
assumed to be 0. Thus most arguments are optional.
ARGUMENTS
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_calobject_warp
do_optimize_frames
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting the variable count of intrinsics in the state vector
- num_states_points(...)
- Get the number of point-position parameters in the optimization vector
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
p = mrcal.optimizer_callback(**optimization_inputs)[0]
mrcal.unpack_state(p, **optimization_inputs)
i_state0 = mrcal.state_index_points(0, **optimization_inputs)
Nstates = mrcal.num_states_points ( **optimization_inputs)
points_all = p[i_state0:i_state0+Nstates]
The optimization algorithm sees its world described in one, big vector of state.
The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.num_states_...() functions report how many variables in the optimization
vector are taken up by each particular kind of measurement.
THIS function reports how many variables are used to represent ALL the points.
The points are stored contiguously as a 3-element coordinates in the reference
frame.
In order to determine the variable mapping, we need quite a bit of context. If
we have the full set of inputs to the optimization function, we can pass in
those (as shown in the example above). Or we can pass the individual arguments
that are needed (see ARGUMENTS section for the full list). If the optimization
inputs and explicitly-given arguments conflict about the size of some array, the
explicit arguments take precedence. If any array size is not specified, it is
assumed to be 0. Thus most arguments are optional.
ARGUMENTS
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_calobject_warp
do_optimize_frames
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting the variable count of points in the state vector
- optimize(...)
- Invoke the calibration routine
SYNOPSIS
stats = mrcal.optimize( intrinsics_data,
extrinsics_rt_fromref,
frames_rt_toref, points,
observations_board, indices_frame_camintrinsics_camextrinsics,
observations_point, indices_point_camintrinsics_camextrinsics,
lensmodel,
imagersizes = imagersizes,
observed_pixel_uncertainty = observed_pixel_uncertainty,
do_optimize_intrinsics_core = True,
do_optimize_intrinsics_distortions= True,
calibration_object_spacing = object_spacing,
point_min_range = 0.1,
point_max_range = 100.0,
do_apply_outlier_rejection = True,
do_apply_regularization = True,
verbose = False)
Please see the mrcal documentation at
http://mrcal.secretsauce.net/formulation.html for details.
This is a flexible implementation of a calibration system core that uses sparse
Jacobians, performs outlier rejection and reports some metrics back to the user.
Measurements from any number of cameras can beat used simultaneously, and this
routine is flexible-enough to solve structure-from-motion problems.
The input is a combination of observations of a calibration board and
observations of discrete points. The point observations MAY have a known
range.
The cameras and what they're observing is given in the arrays
- intrinsics_data
- extrinsics_rt_fromref
- frames_rt_toref
- points
- indices_frame_camintrinsics_camextrinsics
- indices_point_camintrinsics_camextrinsics
intrinsics_data contains the intrinsics for all the physical cameras present in
the problem. len(intrinsics_data) = Ncameras_intrinsics
extrinsics_rt_fromref contains all the camera poses present in the problem,
omitting any cameras that sit at the reference coordinate system.
len(extrinsics_rt_fromref) = Ncameras_extrinsics.
frames_rt_toref is all the poses of the calibration board in the problem, and
points is all the discrete points being observed in the problem.
indices_frame_camintrinsics_camextrinsics describes which board observations
were made by which camera, and where this camera was. Each board observation is
described by a tuple (iframe,icam_intrinsics,icam_extrinsics). The board at
frames_rt_toref[iframe] was observed by camera
intrinsics_data[icam_intrinsics], which was at
extrinsics_rt_fromref[icam_extrinsics]
indices_point_camintrinsics_camextrinsics is the same thing for discrete points.
If we're solving a vanilla calibration problem, we have stationary cameras
observing a moving target. By convention, camera 0 is at the reference
coordinate system. So
- Ncameras_intrinsics = Ncameras_extrinsics+1
- All entries in indices_frame_camintrinsics_camextrinsics have
icam_intrinsics = icam_extrinsics+1
- frames_rt_toref, points describes the motion of the moving target we're
observing
Conversely, in a structure-from-motion problem we have some small number of
moving cameras (often just 1) observing stationary target(s). We would have
- Ncameras_intrinsics is small: it's how many physical cameras we have
- Ncameras_extrinsics is large: it describes the motion of the cameras
- frames_rt_toref, points is small: it describes the non-moving world we're
observing
Any combination of these extreme cases is allowed.
REQUIRED ARGUMENTS
- intrinsics: array of dims (Ncameras_intrinsics, Nintrinsics). The intrinsics
of each physical camera. Each intrinsic vector is given as
(focal_x, focal_y, center_pixel_x, center_pixel_y, distortion0, distortion1,
...)
The focal lengths are given in pixels.
On input this is a seed. On output the optimal data is returned. THIS ARRAY IS
MODIFIED BY THIS CALL.
- extrinsics_rt_fromref: array of dims (Ncameras_extrinsics, 6). The pose of
each camera observation. Each pose is given as 6 values: a Rodrigues rotation
vector followed by a translation. This represents a transformation FROM the
reference coord system TO the coord system of each camera.
On input this is a seed. On output the optimal data is returned. THIS ARRAY IS
MODIFIED BY THIS CALL.
If we only have one camera, pass either None or np.zeros((0,6))
- frames_rt_toref: array of dims (Nframes, 6). The poses of the calibration
object over time. Each pose is given as 6 values: a rodrigues rotation vector
followed by a translation. This represents a transformation FROM the coord
system of the calibration object TO the reference coord system. THIS IS
DIFFERENT FROM THE CAMERA EXTRINSICS.
On input this is a seed. On output the optimal data is returned. THIS ARRAY IS
MODIFIED BY THIS CALL.
If we don't have any frames, pass either None or np.zeros((0,6))
- points: array of dims (Npoints, 3). The estimated positions of discrete points
we're observing. These positions are represented in the reference coord
system. The initial Npoints-Npoints_fixed points are optimized by this
routine. The final Npoints_fixed points are fixed. By default
Npoints_fixed==0, and we optimize all the points.
On input this is a seed. On output the optimal data is returned. THIS ARRAY IS
MODIFIED BY THIS CALL.
- observations_board: array of dims (Nobservations_board,
calibration_object_height_n,
calibration_object_width_n,
3).
Each slice is an (x,y,weight) tuple where (x,y) are the observed pixel
coordinates of the corners in the calibration object, and "weight" is the
relative weight of this point observation. Most of the weights are expected to
be 1.0, which implies that the noise on that observation has standard
deviation of observed_pixel_uncertainty (in addition to the overall assumption
of gaussian noise, independent on x,y). observed_pixel_uncertainty scales
inversely with the weight. weight<0 indicates that this is an outlier. This is
respected on input (even if !do_apply_outlier_rejection). New outliers are
marked with weight<0 on output. Subpixel interpolation is assumed, so these
contain 64-bit floating point values, like all the other data. The frame and
camera that produced these observations are given in the
indices_frame_camintrinsics_camextrinsics
THIS ARRAY IS MODIFIED BY THIS CALL (to mark outliers)
- indices_frame_camintrinsics_camextrinsics: array of dims (Nobservations_board,
3). For each observation these are an
(iframe,icam_intrinsics,icam_extrinsics) tuple. icam_extrinsics == -1
means this observation came from a camera in the reference coordinate system.
iframe indexes the "frames_rt_toref" array, icam_intrinsics indexes the
"intrinsics_data" array, icam_extrinsics indexes the "extrinsics_rt_fromref"
array
All of the indices are guaranteed to be monotonic. This array contains 32-bit
integers.
- observations_point: array of dims (Nobservations_point, 3). Each slice is an
(x,y,weight) tuple where (x,y) are the pixel coordinates of the observed
point, and "weight" is the relative weight of this point observation. Most of
the weights are expected to be 1.0, which implies that the noise on the
observation is gaussian, independent on x,y, and has standard deviation of
observed_pixel_uncertainty. observed_pixel_uncertainty scales inversely with
the weight. weight<0 indicates that this is an outlier. This is respected on
input (even if !do_apply_outlier_rejection). At this time, no new outliers are
detected for point observations. Subpixel interpolation is assumed, so these
contain 64-bit floating point values, like all the other data. The point index
and camera that produced these observations are given in the
indices_point_camera_points array.
- indices_point_camintrinsics_camextrinsics: array of dims (Nobservations_point,
3). For each observation these are an
(i_point,icam_intrinsics,icam_extrinsics) tuple. Analogous to
indices_frame_camintrinsics_camextrinsics, but for observations of discrete
points.
The indices can appear in any order. No monotonicity is required. This array
contains 32-bit integers.
- lensmodel: a string such as
LENSMODEL_PINHOLE
LENSMODEL_OPENCV4
LENSMODEL_CAHVOR
LENSMODEL_SPLINED_STEREOGRAPHIC_order=3_Nx=16_Ny=12_fov_x_deg=100
LENSMODEL_CAHVORE is explicitly NOT supported at this time
- imagersizes: integer array of dims (Ncameras_intrinsics,2)
OPTIONAL ARGUMENTS
- calobject_warp
A numpy array of shape (2,) describing the non-flatness of the calibration
board. If omitted or None, the board is assumed to be perfectly flat. And if
do_optimize_calobject_warp then we optimize these parameters to find the
best-fitting board shape.
- Npoints_fixed
Specifies how many points at the end of the points array are fixed, and remain
unaffected by the optimization. This is 0 by default, and we optimize all the
points.
- do_optimize_intrinsics_core
- do_optimize_intrinsics_distortions
- do_optimize_extrinsics
- do_optimize_frames
- do_optimize_calobject_warp
Indicate whether to optimize a specific set of variables. The intrinsics core
is fx,fy,cx,cy. These all default to True so if we specify none of these, we
will optimize ALL the variables.
- calibration_object_spacing: the width of each square in a calibration board.
Can be omitted if we have no board observations, just points. The calibration
object has shape (calibration_object_height_n,calibration_object_width_n),
given by the dimensions of "observations_board"
- verbose: if True, write out all sorts of diagnostic data to STDERR. Defaults
to False
- do_apply_outlier_rejection: if False, don't bother with detecting or rejecting
outliers. The outliers we get on input (observations_board[...,2] < 0) are
honered regardless. Defaults to True
- do_apply_regularization: if False, don't include regularization terms in the
solver. Defaults to True
- observed_pixel_uncertainty: Required if do_apply_outlier_rejection. The
standard deviation of x and y pixel coordinates of the input observations. The
distribution of the inputs is assumed to be gaussian, with the standard
deviation specified by this argument. Note: this is the x and y standard
deviation, treated independently. If each of these is s, then the LENGTH of
the deviation of each pixel is a Rayleigh distribution with expected value
s*sqrt(pi/2) ~ s*1.25. This is used to set the outlier rejection threshold
- point_min_range, point_max_range: Required ONLY if point observations are
given. These are lower, upper bounds for the distance of a point observation
to its observing camera. Each observation outside of this range is penalized.
This helps the solver by guiding it away from unreasonable solutions.
We return a dict with various metrics describing the computation we just
performed
- optimizer_callback(...)
- Call the optimization callback function
SYNOPSIS
model = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = model.optimization_inputs()
p_packed,x,J_packed,factorization = \
mrcal.optimizer_callback( **optimization_inputs )
Please see the mrcal documentation at
http://mrcal.secretsauce.net/formulation.html for details.
The main optimization routine in mrcal.optimize() searches for optimal
parameters by repeatedly calling a function to evaluate each hypothethical
parameter set. This evaluation function is available by itself here, separated
from the optimization loop. The arguments are largely the same as those to
mrcal.optimize(), but the inputs are all read-only. Some arguments that have
meaning in calls to optimize() have no meaning in calls to optimizer_callback().
These are accepted, and effectively ignored. Currently these are:
- do_apply_outlier_rejection
ARGUMENTS
This function accepts lots of arguments, but they're the same as the arguments
to mrcal.optimize() so please see that documentation for details. Arguments
accepted by optimizer_callback() on top of those in optimize():
- no_jacobian: optional boolean defaulting to False. If True, we do not compute
a jacobian, which would speed up this function. We then return None in its
place. if no_jacobian and not no_factorization then we still compute and
return a jacobian, since it's needed for the factorization
- no_factorization: optional boolean defaulting to False. If True, we do not
compute a cholesky factorization of JtJ, which would speed up this function.
We then return None in its place. if no_jacobian and not no_factorization then
we still compute and return a jacobian, since it's needed for the
factorization
RETURNED VALUES
The output is returned in a tuple:
- p_packed: a numpy array of shape (Nstate,). This is the packed (unitless)
state vector that represents the inputs, as seen by the optimizer. If the
optimization routine was running, it would use this as a starting point in the
search for different parameters, trying to find those that minimize norm2(x).
This packed state can be converted to the expanded representation like this:
p = mrcal.optimizer_callback(**optimization_inputs)[0
mrcal.unpack_state(p, **optimization_inputs)
- x: a numpy array of shape (Nmeasurements,). This is the error vector. If the
optimization routine was running, it would be testing different parameters,
trying to find those that minimize norm2(x)
- J: a sparse matrix of shape (Nmeasurements,Nstate). These are the gradients of
the measurements in respect to the packed parameters. This is a SPARSE array
of type scipy.sparse.csr_matrix. This object can be converted to a numpy array
like this:
p,x,J_sparse = mrcal.optimizer_callback(...)[:3]
J_numpy = J_sparse.toarray()
Note that the numpy array is dense, so it is very inefficient for sparse data,
and working with it could be very memory-intensive and slow.
This jacobian matrix comes directly from the optimization callback function,
which uses packed, unitless state. To convert a densified packed jacobian to
full units, one can do this:
J_sparse = mrcal.optimizer_callback(**optimization_inputs)[2]
J_numpy = J_sparse.toarray()
mrcal.pack_state(J_numpy, **optimization_inputs)
Note that we're calling pack_state() intead of unpack_state() because the
packed variables are in the denominator
- factorization: a Cholesky factorization of JtJ in a
mrcal.CHOLMOD_factorization object. The core of the optimization algorithm is
solving a linear system JtJ x = b. J is a large, sparse matrix, so we do this
with a Cholesky factorization of J using the CHOLMOD library. This
factorization is also useful in other contexts, such as uncertainty
quantification, so we make it available here. If the factorization could not
be computed (because JtJ isn't full-rank for instance), this is set to None
- pack_state(...)
- Scales a state vector to the packed, unitless form used by the optimizer
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
Jpacked = mrcal.optimizer_callback(**optimization_inputs)[2].toarray()
J = Jpacked.copy()
mrcal.pack_state(J, **optimization_inputs)
In order to make the optimization well-behaved, we scale all the variables in
the state and the gradients before passing them to the optimizer. The internal
optimization library thus works only with unitless (or "packed") data.
This function takes a full numpy array of shape (...., Nstate), and scales it to
produce packed data. This function applies the scaling directly to the input
array; the input is modified, and nothing is returned.
To unpack a state vector, you naturally call unpack_state(). To unpack a
jacobian matrix, you would call pack_state() because in a jacobian, the state is
in the denominator. This is shown in the example above.
Broadcasting is supported: any leading dimensions will be processed correctly,
as long as the given array has shape (..., Nstate).
In order to know what the scale factors should be, and how they should map to
each variable in the state vector, we need quite a bit of context. If we have
the full set of inputs to the optimization function, we can pass in those (as
shown in the example above). Or we can pass the individual arguments that are
needed (see ARGUMENTS section for the full list). If the optimization inputs and
explicitly-given arguments conflict about the size of some array, the explicit
arguments take precedence. If any array size is not specified, it is assumed to
be 0. Thus most arguments are optional.
ARGUMENTS
- p: a numpy array of shape (..., Nstate). This is the full state on input, and
the packed state on output. The input array is modified.
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_calobject_warp
do_optimize_frames
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
None. The scaling is applied to the input array
- pinhole_model_for_reprojection(model_from, fit=None, scale_focal=None, scale_image=None)
- Generate a pinhole model suitable for reprojecting an image
SYNOPSIS
model_orig = mrcal.cameramodel("xxx.cameramodel")
image_orig = cv2.imread("image.jpg")
model_pinhole = mrcal.pinhole_model_for_reprojection(model_orig,
fit = "corners")
mapxy = mrcal.image_transformation_map(model_orig, model_pinhole)
image_undistorted = mrcal.transform_image(image_orig, mapxy)
Many algorithms work with images assumed to have been captured with a pinhole
camera, even though real-world lenses never fit a pinhole model. mrcal provides
several functions to remap images captured with non-pinhole lenses into images
of the same scene as if they were observed by a pinhole lens. When doing this,
we're free to choose all of the parameters of this pinhole lens model. THIS
function produces the pinhole camera model based on some guidance in the
arguments, and this model can then be used to "undistort" images.
ARGUMENTS
- model_from: the mrcal.cameramodel object used to build the pinhole model. We
use the intrinsics as the baseline, and we copy the extrinsics to the
resulting pinhole model.
- fit: optional specification for focal-length scaling. By default we use the
focal length values from the input model. This is either a numpy array of
shape (...,2) containing pixel coordinates that the resulting pinhole model
must represent, or one of ("corners","centers-horizontal","centers-vertical").
See the docstring for scale_focal__best_pinhole_fit() for details. Exclusive
with 'scale_focal'
- scale_focal: optional specification for focal-length scaling. By default we
use the focal length values from the input model. If given, we scale the input
focal lengths by the given value. Exclusive with 'fit'
- scale_image: optional specification for the scaling of the image size. By
default the output model represents an image of the same resolution as the
input model. If we want something else, the scaling can be given here.
RETURNED VALUE
A mrcal.cameramodel object with lensmodel = LENSMODEL_PINHOLE corresponding to
the input model.
- plotoptions_measurement_boundaries(**optimization_inputs)
- Return the 'set' plot options for gnuplotlib to show the measurement boundaries
SYNOPSIS
import numpy as np
import gnuplotlib as gp
import mrcal
model = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = model.optimization_inputs()
x = mrcal.optimizer_callback(**optimization_inputs)[1]
gp.plot( np.abs(x),
_set = mrcal.plotoptions_measurement_boundaries(**optimization_inputs) )
# a plot pops up showing the magnitude of each measurement, with boundaries
# between the different measurements denoted
When plotting the measurement vector (or anything relating to it, such as
columns of the Jacobian), it is usually very useful to infer at a glance the
meaning of each part of the plot. This function returns a list of 'set'
directives passable to gnuplotlib that show the boundaries inside the
measurement vector.
ARGUMENTS
**optimization_inputs: a dict() of arguments passable to mrcal.optimize() and
mrcal.optimizer_callback(). These define the full optimization problem, and can
be obtained from the optimization_inputs() method of mrcal.cameramodel
RETURNED VALUE
A list of 'set' directives passable as plot options to gnuplotlib
- plotoptions_state_boundaries(**optimization_inputs)
- Return the 'set' plot options for gnuplotlib to show the state boundaries
SYNOPSIS
import numpy as np
import gnuplotlib as gp
import mrcal
model = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = model.optimization_inputs()
J = mrcal.optimizer_callback(**optimization_inputs)[2]
gp.plot( np.sum(np.abs(J.toarray()), axis=-2),
_set = mrcal.plotoptions_state_boundaries(**optimization_inputs) )
# a plot pops up showing the magnitude of the effects of each element of the
# packed state (as seen by the optimizer), with boundaries between the
# different state variables denoted
When plotting the state vector (or anything relating to it, such as rows of the
Jacobian), it is usually very useful to infer at a glance the meaning of each
part of the plot. This function returns a list of 'set' directives passable to
gnuplotlib that show the boundaries inside the state vector.
ARGUMENTS
**optimization_inputs: a dict() of arguments passable to mrcal.optimize() and
mrcal.optimizer_callback(). These define the full optimization problem, and can
be obtained from the optimization_inputs() method of mrcal.cameramodel
RETURNED VALUE
A list of 'set' directives passable as plot options to gnuplotlib
- polygon_difference(positive, negative)
- Return the difference of two closed polygons
SYNOPSIS
import numpy as np
import numpysane as nps
import gnuplotlib as gp
A = np.array(((-1,-1),( 1,-1),( 1, 1),(-1, 1),(-1,-1)))
B = np.array(((-.1,-1.1),( .1,-1.1),( .1, 1.1),(-.1, 1.1),(-.1,-1.1)))
diff = mrcal.polygon_difference(A, B)
gp.plot( (A, dict(legend = 'A', _with = 'lines')),
(B, dict(legend = 'B', _with = 'lines')),
*[ ( r, dict( _with = 'filledcurves closed fillcolor "red"',
legend = 'difference'))
for r in diff],
tuplesize = -2,
square = True,
wait = True)
Given two polygons specified as a point sequence in arrays of shape (N,2) this
function computes the topological difference: all the regions contained in the
positive polygon, but missing in the negative polygon. The result could be
empty, or it could contain any number of disconnected polygons, so a list of
polygons is returned. Each of the constituent resulting polygons is guaranteed
to not have holes. If any holes are found when computing the difference, we cut
apart the resulting shape until no holes remain.
ARGUMENTS
- positive: a polygon specified by a sequence of points in an array of shape
(N,2). The resulting difference describes regions contained inside the
positive polygon
- negative: a polygon specified by a sequence of points in an array of shape
(N,2). The resulting difference describes regions outside the negative polygon
RETURNED VALUE
A list of arrays of shape (N,2). Each array in the list describes a hole-free
polygon as a sequence of points. The difference is a union of all these
constituent polygons. This list could have 0 elements (empty difference) or N
element (difference consists of N separate polygons)
- project(v, lensmodel, intrinsics_data, get_gradients=False, out=None)
- Projects a set of 3D camera-frame points to the imager
SYNOPSIS
points = mrcal.project( # (...,3) array of 3d points we're projecting
v,
lensmodel, intrinsics_data)
# points is a (...,2) array of pixel coordinates
Given a shape-(...,3) array of points in the camera frame (x,y aligned with the
imager coords, z 'forward') and an intrinsic model, this function computes the
projection, optionally with gradients.
Projecting out-of-bounds points (beyond the field of view) returns undefined
values. Generally things remain continuous even as we move off the imager
domain. Pinhole-like projections will work normally if projecting a point behind
the camera. Splined projections clamp to the nearest spline segment: the
projection will fly off to infinity quickly since we're extrapolating a
polynomial, but the function will remain continuous.
Broadcasting is fully supported across v and intrinsics_data
ARGUMENTS
- points: array of dims (...,3); the points we're projecting
- lensmodel: a string such as
LENSMODEL_PINHOLE
LENSMODEL_OPENCV4
LENSMODEL_CAHVOR
LENSMODEL_SPLINED_STEREOGRAPHIC_order=3_Nx=16_Ny=12_fov_x_deg=100
- intrinsics: array of dims (Nintrinsics):
(focal_x, focal_y, center_pixel_x, center_pixel_y, distortion0, distortion1,
...)
The focal lengths are given in pixels.
- get_gradients: optional boolean that defaults to False. Whether we should
compute and report the gradients. This affects what we return
- out: optional argument specifying the destination. By default, new numpy
array(s) are created and returned. To write the results into existing arrays,
specify them with the 'out' kwarg. If get_gradients: 'out' is the one numpy
array we will write into. Else: 'out' is a tuple of all the output numpy
arrays. If 'out' is given, we return the same arrays passed in. This is the
standard behavior provided by numpysane_pywrap.
RETURNED VALUE
if not get_gradients:
we return an (...,2) array of projected pixel coordinates
if get_gradients: we return a tuple:
- (...,2) array of projected pixel coordinates
- (...,2,3) array of the gradients of the pixel coordinates in respect to
the input 3D point positions
- (...,2,Nintrinsics) array of the gradients of the pixel coordinates in
respect to the intrinsics
The unprojected observation vector of shape (..., 3).
- project_stereographic(...)
- Projects a set of 3D camera-frame points using a stereographic map
SYNOPSIS
q = mrcal.project_stereographic( # (N,3) array of 3d points we're projecting
points )
# q is now a (N,2) array of normalized stereographic pixel coordinates
This is a special case of mrcal.project(). Useful as part of data analysis, not
to represent any real-world lens.
Given a (N,3) array of points in the camera frame (x,y aligned with the imager
coords, z 'forward') and parameters of a perfect stereographic camera, this
function computes the projection, optionally with gradients. No actual lens ever
follows this model exactly, but this is useful as a baseline for other models.
The user can pass in focal length and center-pixel values. Or they can be
omitted to compute a "normalized" stereographic projection (fx = fy = 1, cx = cy
= 0).
The stereographic projection is able to represent points behind the camera, and
has only one singular observation direction: directly behind the camera, along
the optical axis.
This projection acts radially. If the observation vector v makes an angle theta
with the optical axis, then the projected point q is 2 tan(theta/2) f from the
image center.
ARGUMENTS
- points: array of dims (...,3); the points we're projecting. This supports
broadcasting fully, and any leading dimensions are allowed, including none
- fx, fy: optional focal-lengths, in pixels. Both default to 1, as in the
normalized stereographic projection
- cx, cy: optional projection center, in pixels. Both default to 0, as in the
normalized stereographic projection
- get_gradients: optional boolean, defaults to False. This affects what we
return (see below)
RETURNED VALUE
if not get_gradients: we return an (...,2) array of projected pixel coordinates
if get_gradients: we return a tuple:
- (...,2) array of projected pixel coordinates
- (...,2,3) array of the gradients of the pixel coordinates in respect to
the input 3D point positions
- projection_diff(models, gridn_width=60, gridn_height=None, distance=None, use_uncertainties=True, focus_center=None, focus_radius=-1.0, implied_Rt10=None)
- Compute the difference in projection between N models
SYNOPSIS
models = ( mrcal.cameramodel('cam0-dance0.cameramodel'),
mrcal.cameramodel('cam0-dance1.cameramodel') )
difference,_,q0,_ = mrcal.projection_diff(models)
print(q0.shape)
==> (40,60)
print(difference.shape)
==> (40,60)
# The differences are computed across a grid. 'q0' is the pixel centers of
# each grid cell. 'difference' is the projection variation between the two
# models at each cell
The operation of this tool is documented at
http://mrcal.secretsauce.net/differencing.html
It is often useful to compare the projection behavior of two camera models. For
instance, one may want to validate a calibration by comparing the results of two
different chessboard dances. Or one may want to evaluate the stability of the
intrinsics in response to mechanical or thermal stresses. This function makes
these comparisons, and returns the results. mrcal.show_projection_diff() ALSO
produces a visualization.
In the most common case we're given exactly 2 models to compare, and we compute
the differences in projection of each point. If we're given more than 2 models,
we instead compute the standard deviation of the differences between models 1..N
and model0.
We do this:
- grid the imager
- unproject each point in the grid from one camera to produce a world point
- apply a transformation we compute to match up the two camera geometries
- reproject the transformed points to the other camera
- look at the resulting pixel difference in the reprojection
When looking at multiple cameras, their lens intrinsics differ. Less obviously,
the position and orientation of the camera coordinate system in respect to the
physical camera housing differ also. These geometric uncertainties are baked
into the intrinsics. So when we project "the same world point" into both
cameras, we must apply a geometric transformation because we want to be
comparing projections of world points (relative to the camera housing), not
projections relative to the (floating) camera coordinate systems. This
transformation is unknown, but we can estimate it by fitting projections across
the imager: the "right" transformation would result in apparent low projection
differences in a wide area.
This transformation is computed by implied_Rt10__from_unprojections(), and some
details of its operation are significant:
- The imager area we use for the fit
- Which world points we're looking at
In most practical usages, we would not expect a good fit everywhere in the
imager: areas where no chessboards were observed will not fit well, for
instance. From the point of view of the fit we perform, those ill-fitting areas
should be treated as outliers, and they should NOT be a part of the solve. How
do we specify the well-fitting area? The best way is to use the model
uncertainties: these can be used to emphasize the confident regions of the
imager. This behavior is selected with use_uncertainties=True, which is the
default. If uncertainties aren't available, or if we want a faster solve, pass
use_uncertainties=False. The well-fitting region can then be passed using the
focus_center,focus_radius arguments to indicate the circle in the imager we care
about.
If use_uncertainties then the defaults for focus_center,focus_radius are set to
utilize all the data in the imager. If not use_uncertainties, then the defaults
are to use a more reasonable circle of radius min(width,height)/6 at the center
of the imager. Usually this is sufficiently correct, and we don't need to mess
with it. If we aren't guided to the correct focus region, the
implied-by-the-intrinsics solve will try to fit lots of outliers, which would
result in an incorrect transformation, which in turn would produce overly-high
reported diffs. A common case when this happens is if the chessboard
observations used in the calibration were concentrated to the side of the image
(off-center), no uncertainties were used, and the focus_center was not pointed
to that area.
If we KNOW that there is no geometric difference between our cameras, and we
thus should look at the intrinsics differences only, then we don't need to
estimate the transformation. Indicate this case by passing focus_radius=0.
Unlike the projection operation, the diff operation is NOT invariant under
geometric scaling: if we look at the projection difference for two points at
different locations along a single observation ray, there will be a variation in
the observed diff. This is due to the geometric difference in the two cameras.
If the models differed only in their intrinsics parameters, then this variation
would not appear. Thus we need to know how far from the camera to look, and this
is specified by the "distance" argument. By default (distance = None) we look
out to infinity. If we care about the projection difference at some other
distance, pass that here. Multiple distances can be passed in an iterable. We'll
then fit the implied-by-the-intrinsics transformation using all the distances,
and we'll display the best-fitting difference for each pixel. Generally the most
confident distance will be where the chessboards were observed at calibration
time.
ARGUMENTS
- models: iterable of mrcal.cameramodel objects we're comparing. Usually there
will be 2 of these, but more than 2 is possible. The intrinsics are used; the
extrinsics are NOT.
- gridn_width: optional value, defaulting to 60. How many points along the
horizontal gridding dimension
- gridn_height: how many points along the vertical gridding dimension. If None,
we compute an integer gridn_height to maintain a square-ish grid:
gridn_height/gridn_width ~ imager_height/imager_width
- distance: optional value, defaulting to None. The projection difference varies
depending on the range to the observed world points, with the queried range
set in this 'distance' argument. If None (the default) we look out to
infinity. We can compute the implied-by-the-intrinsics transformation off
multiple distances if they're given here as an iterable. This is especially
useful if we have uncertainties, since then we'll emphasize the best-fitting
distances.
- use_uncertainties: optional boolean, defaulting to True. If True we use the
whole imager to fit the implied-by-the-intrinsics transformation, using the
uncertainties to emphasize the confident regions. If False, it is important to
select the confident region using the focus_center and focus_radius arguments.
If use_uncertainties is True, but that data isn't available, we report a
warning, and try to proceed without.
- focus_center: optional array of shape (2,); the imager center by default. Used
to indicate that the implied-by-the-intrinsics transformation should use only
those pixels a distance focus_radius from focus_center. This is intended to be
used if no uncertainties are available, and we need to manually select the
focus region.
- focus_radius: optional value. If use_uncertainties then the default is LARGE,
to use the whole imager. Else the default is min(width,height)/6. Used to
indicate that the implied-by-the-intrinsics transformation should use only
those pixels a distance focus_radius from focus_center. This is intended to be
used if no uncertainties are available, and we need to manually select the
focus region. Pass focus_radius=0 to avoid computing the transformation, and
to use the identity. This would mean there're no geometric differences, and
we're comparing the intrinsics only
- implied_Rt10: optional Rt transformation (numpy array of shape (4,3)). If
given, I use the given value for the implied-by-the-intrinsics transformation
instead of fitting it. If omitted, I compute the transformation. Exclusive
with focus_center, focus_radius. Valid only if exactly two models are given.
RETURNED VALUE
A tuple
- difflen: a numpy array of shape (gridn_height,gridn_width) containing the
magnitude of differences at each cell, or the standard deviation of the
differences between models 1..N and model0 if len(models)>2. if
len(models)==2: this is nps.mag(diff)
- diff: a numpy array of shape (gridn_height,gridn_width,2) containing the
vector of differences at each cell. If len(models)>2 this isn't defined, so
None is returned
- q0: a numpy array of shape (gridn_height,gridn_width,2) containing the pixel
coordinates of each grid cell
- implied_Rt10: the geometric Rt transformation in an array of shape (...,4,3).
This is either whatever was passed into this function (if anything was), or
the identity if focus_radius==0 or the fitted results. if len(models)>2: this
is an array of shape (len(models)-1,4,3), with slice i representing the
transformation between camera 0 and camera i+1.
- projection_uncertainty(p_cam, model, atinfinity=False, what='covariance')
- Compute the projection uncertainty of a camera-referenced point
This is the interface to the uncertainty computations described in
http://mrcal.secretsauce.net/uncertainty.html
SYNOPSIS
model = mrcal.cameramodel("xxx.cameramodel")
q = np.array((123., 443.))
distance = 10.0
pcam = distance * mrcal.unproject(q, *model.intrinsics(), normalize=True)
print(mrcal.projection_uncertainty(pcam,
model = model,
what = 'worstdirection-stdev'))
===> 0.5
# So if we have observed a world point at pixel coordinates q, and we know
# it's 10m out, then we know that the standard deviation of the noise of the
# pixel obsevation is 0.5 pixels, in the worst direction
After a camera model is computed via a calibration process, the model is
ultimately used in projection/unprojection operations to map between world
coordinates and projected pixel coordinates. We never know the parameters of the
model perfectly, and it is VERY useful to know the resulting uncertainty of
projection. This can be used, among other things, to
- propagate the projection noise down to whatever is using the observed pixels
to do stuff
- evaluate the quality of calibrations, to know whether a given calibration
should be accepted, or rejected
- evaluate the stability of a computed model
I quantify uncertainty by propagating expected noise on observed chessboard
corners through the optimization problem we're solving during calibration time
to the solved parameters. And then propagating the noise on the parameters
through projection.
The below derivation is double-checked via simulated noise in
test-projection-uncertainty.py
The uncertainties can be visualized with the mrcal-show-projection-uncertainty
tool.
ARGUMENTS
This function accepts an array of camera-referenced points p_cam and some
representation of parameters and uncertainties (either a single
mrcal.cameramodel object or all of
(lensmodel,intrinsics_data,extrinsics_rt_fromref,frames_rt_toref,Var_ief)). And
a few meta-parameters that describe details of the behavior. This function
broadcasts on p_cam only. We accept
- p_cam: a numpy array of shape (..., 3). This is the set of camera-coordinate
points where we're querying uncertainty. if not atinfinity: then the full 3D
coordinates of p_cam are significant, even distance to the camera. if
atinfinity: the distance to the camera is ignored.
- model: a mrcal.cameramodel object containing the intrinsics, extrinsics, frame
poses and their covariance. If this isn't given, then each of these MUST be
given in a separate argument
- lensmodel: a string describing which lens model we're using. This is something
like 'LENSMODEL_OPENCV4'. This is required if and only if model is None
- intrinsics_data: a numpy array of shape (Nintrinsics,) where Nintrinsics is
the number of parameters in the intrinsics vector for this lens model,
returned by mrcal.lensmodel_num_params(lensmodel). This is required if and only if
model is None
- extrinsics_rt_fromref: a numpy array of shape (6,) or None. This is an rt
transformation from the reference coordinate system to the camera coordinate
system. If None: the camera is at the reference coordinate system. Note that
these are the extrinsics AT CALIBRATION TIME. If we moved the camera after
calibrating, then this is OK, but for the purposes of uncertainty
computations, we care about where the camera used to be. This is required if
and only if model is None
- frames_rt_toref: a numpy array of shape (Nframes,6). These are rt
transformations from the coordinate system of each calibration object coord
system to the reference coordinate system. This array represents ALL the
observed chessboards in a calibration optimization problem. This is required
if and only if model is None
- Var_ief: a square numpy array with the intrinsics, extrinsics, frame
covariance. It is the caller's responsibility to make sure that the dimensions
match the frame counts and whether extrinsics_rt_fromref is None or not. This
is required if and only if model is None
- atinfinity: optional boolean, defaults to False. If True, we want to know the
projection uncertainty, looking at a point infinitely-far away. We propagate
all the uncertainties, ignoring the translation components of the poses
- what: optional string, defaults to 'covariance'. This chooses what kind of
output we want. Known options are:
- 'covariance': return a full (2,2) covariance matrix Var(q) for
each p_cam
- 'worstdirection-stdev': return the worst-direction standard deviation for
each p_cam
- 'rms-stdev': return the RMS of the worst and best direction
standard deviations
RETURN VALUE
A numpy array of uncertainties. If p_cam has shape (..., 3) then:
if what == 'covariance': we return an array of shape (..., 2,2)
else: we return an array of shape (...)
- quat_from_R(R)
- Convert a rotation defined as a rotation matrix to a unit quaternion
SYNOPSIS
print(R.shape)
===>
(3,3)
quat = mrcal.quat_from_R(R)
print(quat.shape)
===>
(4,)
c = quat[0]
s = nps.mag(quat[1:])
rotation_magnitude = 2. * np.arctan2(s,c)
rotation_axis = quat[1:] / s
This is mostly for compatibility with some old stuff. mrcal doesn't use
quaternions anywhere. Test this thoroughly before using.
The implementation comes directly from the scipy project, the from_dcm()
function in
https://github.com/scipy/scipy/blob/master/scipy/spatial/transform/rotation.py
Commit: 1169d27ad47a29abafa8a3d2cb5b67ff0df80a8f
License:
Copyright (c) 2001-2002 Enthought, Inc. 2003-2019, SciPy Developers.
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided
with the distribution.
3. Neither the name of the copyright holder nor the names of its
contributors may be used to endorse or promote products derived
from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- r_from_R(R, get_gradients=False, out=None)
- Compute a Rodrigues vector from a rotation matrix
SYNOPSIS
r = mrcal.r_from_R(R)
rotation_magnitude = nps.mag(r)
rotation_axis = r / rotation_magnitude
Given a rotation specified as a (3,3) rotation matrix, converts it to a
Rodrigues vector, a unit rotation axis scaled by the rotation magnitude, in
radians.
By default this function returns the Rodrigues vector(s) only. If we also want
gradients, pass get_gradients=True. Logic:
if not get_gradients: return r
else: return (r, dr/dR)
This function supports broadcasting fully.
ARGUMENTS
- R: array of shape (3,3). This matrix defines the rotation. It is assumed that
this is a valid rotation (matmult(R,transpose(R)) = I, det(R) = 1), but that
is not checked
- get_gradients: optional boolean. By default (get_gradients=False) we return an
array of Rodrigues vectors. Otherwise we return a tuple of arrays of Rodrigues
vectors and their gradients.
- out: optional argument specifying the destination. By default, new numpy
array(s) are created and returned. To write the results into existing (and
possibly non-contiguous) arrays, specify them with the 'out' kwarg. If
get_gradients: 'out' is the one numpy array we will write into. Else: 'out' is
a tuple of all the output numpy arrays. If 'out' is given, we return the same
arrays passed in. This is the standard behavior provided by numpysane_pywrap.
RETURNED VALUE
If not get_gradients: we return an array of Rodrigues vector(s). Each
broadcasted slice has shape (3,)
If get_gradients: we return a tuple of arrays containing the Rodrigues vector(s)
and the gradients (r, dr/dR)
1. The Rodrigues vector. Each broadcasted slice has shape (3,)
2. The gradient dr/dR. Each broadcasted slice has shape (3,3,3). The first
dimension selects the element of r, and the last two dimensions select the
element of R
- reduce(...)
- reduce(function, sequence[, initial]) -> value
Apply a function of two arguments cumulatively to the items of a sequence,
from left to right, so as to reduce the sequence to a single value.
For example, reduce(lambda x, y: x+y, [1, 2, 3, 4, 5]) calculates
((((1+2)+3)+4)+5). If initial is present, it is placed before the items
of the sequence in the calculation, and serves as a default when the
sequence is empty.
- ref_calibration_object(W, H, object_spacing, calobject_warp=None)
- Return the geometry of the calibration object
SYNOPSIS
import gnuplotlib as gp
import numpysane as nps
obj = mrcal.ref_calibration_object( 10,6, 0.1 )
print(obj.shape)
===> (6, 10, 3)
gp.plot( nps.clump( obj[...,:2], n=2),
tuplesize = -2,
_with = 'points',
_xrange = (-0.1,1.0),
_yrange = (-0.1,0.6),
unset = 'grid',
square = True,
terminal = 'dumb 74,45')
0.6 +---------------------------------------------------------------+
| + + + + + |
| |
0.5 |-+ A A A A A A A A A A +-|
| |
| |
0.4 |-+ A A A A A A A A A A +-|
| |
| |
0.3 |-+ A A A A A A A A A A +-|
| |
| |
0.2 |-+ A A A A A A A A A A +-|
| |
| |
0.1 |-+ A A A A A A A A A A +-|
| |
| |
0 |-+ A A A A A A A A A A +-|
| |
| + + + + + |
-0.1 +---------------------------------------------------------------+
0 0.2 0.4 0.6 0.8 1
Returns the geometry of a calibration object in its own reference coordinate
system in a (H,W,3) array. Only a grid-of-points calibration object is
supported, possibly with some bowing (i.e. what the internal mrcal solver
supports). Each row of the output is an (x,y,z) point. The origin is at the
corner of the grid, so ref_calibration_object(...)[0,0,:] is
np.array((0,0,0)). The grid spans x and y, with z representing the depth: z=0
for a flat calibration object.
A simple parabolic board warping model is supported by passing a (2,) array in
calobject_warp. These 2 values describe additive flex along the x axis and along
the y axis, in that order. In each direction the flex is a parabola, with the
parameter k describing the max deflection at the center. If the edges were at
+-1 we'd have
z = k*(1 - x^2)
The edges we DO have are at (0,N-1), so the equivalent expression is
xr = x / (N-1)
z = k*( 1 - 4*xr^2 + 4*xr - 1 ) =
4*k*(xr - xr^2) =
ARGUMENTS
- W: how many points we have in the horizontal direction
- H: how many points we have in the vertical direction
- object_spacing: the distance between adjacent points in the calibration
object. A square object is assumed, so the vertical and horizontal distances
are assumed to be identical.
- calobject_warp: optional array of shape (2,) defaults to None. Describes the
warping of the calibration object. If None, the object is flat. If an array is
given, the values describe the maximum additive deflection along the x and y
axes
RETURNED VALUES
The calibration object geometry in a (H,W,3) array
- rotate_point_R(R, x, get_gradients=False, out=None)
- Rotate point(s) using a rotation matrix
SYNOPSIS
r = rotation_axis * rotation_magnitude
R = mrcal.R_from_r(r)
print(R.shape)
===>
(3,3)
print(x.shape)
===>
(10,3)
print( mrcal.rotate_point_R(R, x).shape )
===>
(10,3)
print( [arr.shape for arr in mrcal.rotate_point_R(R, x,
get_gradients = True)] )
===>
[(10,3), (10,3,3,3), (10,3,3)]
Rotate point(s) by a rotation matrix. This is a matrix multiplication. x is
stored as a row vector (that's how numpy stores 1-dimensional arrays), but the
multiplication works as if x was a column vector (to match linear algebra
conventions):
rotate_point_R(R,x) = transpose( matmult(R, transpose(x))) =
= matmult(x, transpose(R))
By default this function returns the rotated points only. If we also want
gradients, pass get_gradients=True. Logic:
if not get_gradients: return u=R(x)
else: return (u=R(x),du/dR,du/dx)
This function supports broadcasting fully, so we can rotate lots of points at
the same time and/or apply lots of different rotations at the same time
ARGUMENTS
- R: array of shape (3,3). This matrix defines the rotation. It is assumed that
this is a valid rotation (matmult(R,transpose(R)) = I, det(R) = 1), but that
is not checked
- x: array of shape (3,). The point being rotated
- get_gradients: optional boolean. By default (get_gradients=False) we return an
array of rotated points. Otherwise we return a tuple of arrays of rotated
points and their gradients.
- out: optional argument specifying the destination. By default, new numpy
array(s) are created and returned. To write the results into existing (and
possibly non-contiguous) arrays, specify them with the 'out' kwarg. If
get_gradients: 'out' is the one numpy array we will write into. Else: 'out' is
a tuple of all the output numpy arrays. If 'out' is given, we return the same
arrays passed in. This is the standard behavior provided by numpysane_pywrap.
RETURNED VALUE
If not get_gradients: we return an array of rotated point(s). Each broadcasted
slice has shape (3,)
If get_gradients: we return a tuple of arrays containing the rotated points and
the gradients (u=R(x),du/dR,du/dx):
1. The rotated point(s). Each broadcasted slice has shape (3,)
2. The gradient du/dR. Each broadcasted slice has shape (3,3,3). The first
dimension selects the element of u, and the last 2 dimensions select the
element of R
3. The gradient du/dx. Each broadcasted slice has shape (3,3). The first
dimension selects the element of u, and the last dimension selects the
element of x
- rotate_point_r(r, x, get_gradients=False, out=None, inverted=False)
- Rotate point(s) using a Rodrigues vector
SYNOPSIS
r = rotation_axis * rotation_magnitude
print(r.shape)
===>
(3,)
print(x.shape)
===>
(10,3)
print(mrcal.rotate_point_r(r, x).shape)
===>
(10,3)
print( [arr.shape for arr in mrcal.rotate_point_r(r, x,
get_gradients = True)] )
===>
[(10,3), (10,3,3), (10,3,3)]
Rotate point(s) by a rotation matrix. The Rodrigues vector is converted to a
rotation matrix internally, and then this function is a matrix multiplication. x
is stored as a row vector (that's how numpy stores 1-dimensional arrays), but
the multiplication works as if x was a column vector (to match linear algebra
conventions):
rotate_point_r(r,x) = transpose( matmult(R(r), transpose(x))) =
= matmult(x, transpose(R(r)))
By default this function returns the rotated points only. If we also want
gradients, pass get_gradients=True. Logic:
if not get_gradients: return u=r(x)
else: return (u=r(x),du/dr,du/dx)
This function supports broadcasting fully, so we can rotate lots of points at
the same time and/or apply lots of different rotations at the same time
ARGUMENTS
- r: array of shape (3,). The Rodrigues vector that defines the rotation. This is
a unit rotation axis scaled by the rotation magnitude, in radians
- x: array of shape (3,). The point being rotated
- get_gradients: optional boolean. By default (get_gradients=False) we return an
array of rotated points. Otherwise we return a tuple of arrays of rotated
points and their gradients.
- inverted: optional boolean, defaulting to False. If True, the opposite
rotation is computed. The gradient du/dr is returned in respect to the input r
- out: optional argument specifying the destination. By default, new numpy
array(s) are created and returned. To write the results into existing (and
possibly non-contiguous) arrays, specify them with the 'out' kwarg. If
get_gradients: 'out' is the one numpy array we will write into. Else: 'out' is
a tuple of all the output numpy arrays. If 'out' is given, we return the same
arrays passed in. This is the standard behavior provided by numpysane_pywrap.
RETURNED VALUE
If not get_gradients: we return an array of rotated point(s). Each broadcasted
slice has shape (3,)
If get_gradients: we return a tuple of arrays containing the rotated points and
the gradients (u=r(x),du/dr,du/dx):
A tuple (u=r(x),du/dr,du/dx):
1. The rotated point(s). Each broadcasted slice has shape (3,)
2. The gradient du/dr. Each broadcasted slice has shape (3,3). The first
dimension selects the element of u, and the last dimension selects the
element of r
3. The gradient du/dx. Each broadcasted slice has shape (3,3). The first
dimension selects the element of u, and the last dimension selects the
element of x
- rt_from_Rt(Rt, get_gradients=False, out=None)
- Compute an rt transformation from a Rt transformation
SYNOPSIS
Rt = nps.glue(rotation_matrix,translation, axis=-2)
print(Rt.shape)
===>
(4,3)
rt = mrcal.rt_from_Rt(Rt)
print(rt.shape)
===>
(6,)
translation = rt[3:]
rotation_magnitude = nps.mag(rt[:3])
rotation_axis = rt[:3] / rotation_magnitude
Converts an Rt transformation to an rt transformation. Both specify a rotation
and translation. An Rt transformation is a (4,3) array formed by nps.glue(R,t,
axis=-2) where R is a (3,3) rotation matrix and t is a (3,) translation vector.
An rt transformation is a (6,) array formed by nps.glue(r,t, axis=-1) where r is
a (3,) Rodrigues vector and t is a (3,) translation vector.
Applied to a point x the transformed result is rotate(x)+t. Given a matrix R,
the rotation is defined by a matrix multiplication. x and t are stored as a row
vector (that's how numpy stores 1-dimensional arrays), but the multiplication
works as if x was a column vector (to match linear algebra conventions). See the
docs for mrcal._transform_point_Rt() for more detail.
By default this function returns the rt transformations only. If we also want
gradients, pass get_gradients=True. Logic:
if not get_gradients: return rt
else: return (rt, dr/dR)
Note that the translation gradient isn't returned: it is always the identity
This function supports broadcasting fully.
ARGUMENTS
- Rt: array of shape (4,3). This matrix defines the transformation. Rt[:3,:] is
a rotation matrix; Rt[3,:] is a translation. It is assumed that the rotation
matrix is a valid rotation (matmult(R,transpose(R)) = I, det(R) = 1), but that
is not checked
- get_gradients: optional boolean. By default (get_gradients=False) we return an
array of rt transformations. Otherwise we return a tuple of arrays of rt
transformations and their gradients.
- out: optional argument specifying the destination. By default, new numpy
array(s) are created and returned. To write the results into existing (and
possibly non-contiguous) arrays, specify them with the 'out' kwarg. If
get_gradients: 'out' is the one numpy array we will write into. Else: 'out' is
a tuple of all the output numpy arrays. If 'out' is given, we return the same
arrays passed in. This is the standard behavior provided by numpysane_pywrap.
RETURNED VALUE
If not get_gradients: we return the rt transformation. Each broadcasted slice
has shape (6,). rt[:3] is a rotation defined as a Rodrigues vector; rt[3:] is a
translation.
If get_gradients: we return a tuple of arrays containing the rt transformation
and the gradient (rt, dr/dR):
1. The rt transformation. Each broadcasted slice has shape (6,)
2. The gradient dr/dR. Each broadcasted slice has shape (3,3,3). The first
dimension selects the element of r, and the last two dimension select the
element of R
- sample_imager(gridn_width, gridn_height, imager_width, imager_height)
- Returns regularly-sampled, gridded pixels coordinates across the imager
SYNOPSIS
q = sample_imager( 60, 40, *model.imagersize() )
print(q.shape)
===>
(40,60,2)
Note that the arguments are given in width,height order, as is customary when
generally talking about images and indexing. However, the output is in
height,width order, as is customary when talking about matrices and numpy
arrays.
If we ask for gridding dimensions (gridn_width, gridn_height), the output has
shape (gridn_height,gridn_width,2) where each row is an (x,y) pixel coordinate.
The top-left corner is at [0,0,:]:
sample_imager(...)[0,0] = [0,0]
The the bottom-right corner is at [-1,-1,:]:
sample_imager(...)[ -1, -1,:] =
sample_imager(...)[gridn_height-1,gridn_width-1,:] =
(imager_width-1,imager_height-1)
When making plots you probably want to call mrcal.imagergrid_using(). See the
that docstring for details.
ARGUMENTS
- gridn_width: how many points along the horizontal gridding dimension
- gridn_height: how many points along the vertical gridding dimension. If None,
we compute an integer gridn_height to maintain a square-ish grid:
gridn_height/gridn_width ~ imager_height/imager_width
- imager_width,imager_height: the width, height of the imager. With a
mrcal.cameramodel object this is *model.imagersize()
RETURNED VALUES
We return an array of shape (gridn_height,gridn_width,2). Each row is an (x,y)
pixel coordinate.
- sample_imager_unproject(gridn_width, gridn_height, imager_width, imager_height, lensmodel, intrinsics_data, normalize=False)
- Reports 3D observation vectors that regularly sample the imager
SYNOPSIS
import gnuplotlib as gp
import mrcal
...
Nwidth = 60
Nheight = 40
# shape (Nheight,Nwidth,3)
v,q = \
mrcal.sample_imager_unproject(Nw, Nh,
*model.imagersize(),
*model.intrinsics())
# shape (Nheight,Nwidth)
f = interesting_quantity(v)
gp.plot(f,
tuplesize = 3,
ascii = True,
using = mrcal.imagergrid_using(model.imagersize, Nw, Nh),
square = True,
_with = 'image')
This is a utility function used by functions that evalute some interesting
quantity for various locations across the imager. Grid dimensions and lens
parameters are passed in, and the grid points and corresponding unprojected
vectors are returned. The unprojected vectors are unique only up-to-length, and
the returned vectors aren't normalized by default. If we want them to be
normalized, pass normalize=True.
This function has two modes of operation:
- One camera. lensmodel is a string, and intrinsics_data is a 1-dimensions numpy
array. With a mrcal.cameramodel object together these are *model.intrinsics().
We return (v,q) where v is a shape (Nheight,Nwidth,3) array of observation
vectors, and q is a (Nheight,Nwidth,2) array of corresponding pixel
coordinates (the grid returned by sample_imager())
- Multiple cameras. lensmodel is a list or tuple of strings; intrinsics_data is
an iterable of 1-dimensional numpy arrays (a list/tuple or a 2D array). We
return the same q as before (only one camera is gridded), but the unprojected
array v has shape (Ncameras,Nheight,Nwidth,3) where Ncameras is the leading
dimension of lensmodel. The gridded imager appears in camera0: v[0,...] =
unproject(q)
ARGUMENTS
- gridn_width: how many points along the horizontal gridding dimension
- gridn_height: how many points along the vertical gridding dimension. If None,
we compute an integer gridn_height to maintain a square-ish grid:
gridn_height/gridn_width ~ imager_height/imager_width
- imager_width,imager_height: the width, height of the imager. With a
mrcal.cameramodel object this is *model.imagersize()
- lensmodel, intrinsics_data: the lens parameters. With a single camera,
lensmodel is a string, and intrinsics_data is a 1-dimensions numpy array; with
a mrcal.cameramodel object together these are *model.intrinsics(). With
multiple cameras, lensmodel is a list/tuple of strings. And intrinsics_data is
an iterable of 1-dimensional numpy arrays (a list/tuple or a 2D array).
- normalize: optional boolean defaults to False. If True: normalize the output
vectors
RETURNED VALUES
We return a tuple:
- v: the unprojected vectors. If we have a single camera this has shape
(Nheight,Nwidth,3). With multiple cameras this has shape
(Ncameras,Nheight,Nwidth,3). These are NOT normalized by default. To get
normalized vectors, pass normalize=True
- q: the imager-sampling grid. This has shape (Nheight,Nwidth,2) regardless of
how many cameras were given (we always sample just one camera). This is what
sample_imager() returns
- scale_focal__best_pinhole_fit(model, fit)
- Compute the optimal focal-length scale for reprojection to a pinhole lens
SYNOPSIS
model = mrcal.cameramodel('from.cameramodel')
lensmodel,intrinsics_data = model.intrinsics()
scale_focal = mrcal.scale_focal__best_pinhole_fit(model,
'centers-horizontal')
intrinsics_data[:2] *= scale_focal
model_pinhole = \
mrcal.cameramodel(intrinsics = ('LENSMODEL_PINHOLE',
intrinsics_data[:4]),
imagersize = model.imagersize(),
extrinsics_rt_fromref = model.extrinsics_rt_fromref() )
Many algorithms work with images assumed to have been captured with a pinhole
camera, even though real-world lenses never fit a pinhole model. mrcal provides
several functions to remap images captured with non-pinhole lenses into images
of the same scene as if they were observed by a pinhole lens. When doing this,
we're free to choose all of the parameters of this pinhole lens model, and this
function allows us to pick the best scaling on the focal-length parameter.
The focal length parameters serve as a "zoom" factor: changing these parameters
can increase the resolution of the center of the image, at the expense of
cutting off the edges. This function computes the best focal-length scaling, for
several possible meanings of "best".
I assume an output pinhole model that has pinhole parameters
(k*fx, k*fy, cx, cy)
where (fx, fy, cx, cy) are the parameters from the input model, and k is the
scaling we compute.
This function looks at some points on the edge of the input image. I choose k so
that all of these points end up inside the pinhole-reprojected image, leaving
the worst one at the edge. The set of points I look at are specified in the
"fit" argument.
ARGUMENTS
- model: a mrcal.cameramodel object for the input lens model
- fit: which pixel coordinates in the input image must project into the output
pinhole-projected image. The 'fit' argument must be one of
- a numpy array of shape (N,2) where each row is a pixel coordinate in the
input image
- "corners": each of the 4 corners of the input image must project into the
output image
- "centers-horizontal": the two points at the left and right edges of the
input image, half-way vertically, must both project into the output image
- "centers-vertical": the two points at the top and bottom edges of the input
image, half-way horizontally, must both project into the output image
RETURNED VALUE
A scalar scale_focal that can be passed to pinhole_model_for_reprojection()
- seed_stereographic(imagersizes, focal_estimate, indices_frame_camera, observations, object_spacing)
- Compute an optimization seed for a camera calibration
SYNOPSIS
print( imagersizes.shape )
===>
(4, 2)
print( indices_frame_camera.shape )
===>
(123, 2)
print( observations.shape )
===>
(123, 3)
intrinsics_data, \
extrinsics_rt_fromref, \
frames_rt_toref = \
mrcal.seed_stereographic(imagersizes = imagersizes,
focal_estimate = 1500,
indices_frame_camera = indices_frame_camera,
observations = observations,
object_spacing = object_spacing)
....
mrcal.optimize(intrinsics_data, extrinsics_rt_fromref, frames_rt_toref,
lensmodel = 'LENSMODEL_STEREOGRAPHIC',
...)
mrcal solves camera calibration problems by iteratively optimizing a nonlinear
least squares problem to bring the pixel observation predictions in line with
actual pixel observations. This requires an initial "seed", an initial estimate
of the solution. This function computes a usable seed, and its results can be
fed to mrcal.optimize(). The output of this function is just an initial estimate
that will be refined, so the results of this function do not need to be exact.
This function assumes we have stereographic lenses, and the returned intrinsics
apply to LENSMODEL_STEREOGRAPHIC. This is usually good-enough to serve as a seed
for both long lenses and wide lenses, every ultra-wide fisheyes. The returned
intrinsics can be expanded to whatever lens model we actually want to use prior
to invoking the optimizer.
By convention, we have a "reference" coordinate system that ties the poses of
all the frames (calibration objects) and the cameras together. And by
convention, this "reference" coordinate system is the coordinate system of
camera 0. Thus the array of camera poses extrinsics_rt_fromref holds Ncameras-1
transformations: the first camera has an identity transformation, by definition.
This function assumes we're observing a moving object from stationary cameras
(i.e. a vanilla camera calibration problem). The mrcal solver is more general,
and supports moving cameras, hence it uses a more general
indices_frame_camintrinsics_camextrinsics array instead of the
indices_frame_camera array used here.
See test/test-calibration-basic.py and mrcal-calibrate-cameras for usage
examples.
ARGUMENTS
- imagersizes: an iterable of (imager_width,imager_height) iterables. Defines
the imager dimensions for each camera we're calibrating. May be an array of
shape (Ncameras,2) or a tuple of tuples or a mix of the two
- focal_estimate: an initial estimate of the focal length of the cameras, in
pixels. For the purposes of the initial estimate we use the same focal length
value for both the x and y focal length of ALL the cameras
- indices_frame_camera: an array of shape (Nobservations,2) and dtype
numpy.int32. Each row (iframe,icam) represents an observation of a
calibration object by camera icam. iframe is not used by this function
- observations: an array of shape
(Nobservations,object_height_n,object_width_n,3). Each observation corresponds
to a row in indices_frame_camera, and contains a row of shape (3,) for each
point in the calibration object. Each row is (x,y,weight) where x,y are the
observed pixel coordinates. Any point where x<0 or y<0 or weight<0 is ignored.
This is the only use of the weight in this function.
- object_spacing: the distance between adjacent points in the calibration
object. A square object is assumed, so the vertical and horizontal distances
are assumed to be identical. Usually we need the object dimensions in the
object_height_n,object_width_n arguments, but here we get those from the shape
of the observations array
RETURNED VALUES
We return a tuple:
- intrinsics_data: an array of shape (Ncameras,4). Each slice contains the
stereographic intrinsics for the given camera. These intrinsics are
(focal_x,focal_y,centerpixel_x,centerpixel_y), and define
LENSMODEL_STEREOGRAPHIC model. mrcal refers to these 4 values as the
"intrinsics core". For models that have such a core (currently, ALL supported
models), the core is the first 4 parameters of the intrinsics vector. So to
calibrate some cameras, call seed_stereographic(), append to intrinsics_data
the proper number of parameters to match whatever lens model we're using, and
then invoke the optimizer.
- extrinsics_rt_fromref: an array of shape (Ncameras-1,6). Each slice is an rt
transformation TO the camera coordinate system FROM the reference coordinate
system. By convention camera 0 defines the reference coordinate system, so
that camera's extrinsics are the identity, by definition, and we don't store
that data in this array
- frames_rt_toref: an array of shape (Nframes,6). Each slice represents the pose
of the calibration object at one instant in time: an rt transformation TO the
reference coordinate system FROM the calibration object coordinate system.
- shellquote = quote(s)
- Return a shell-escaped version of the string *s*.
- show_distortion_off_pinhole(model, mode, scale=1.0, cbmax=25.0, gridn_width=60, gridn_height=None, show_fisheye_projections=False, extratitle=None, return_plot_args=False, **kwargs)
- Visualize a lens's deviation from a pinhole projection
SYNOPSIS
model = mrcal.cameramodel('xxx.cameramodel')
mrcal.show_distortion_off_pinhole( model, 'heatmap' )
... A plot pops up displaying how much this model deviates from a pinhole
... model across the imager
This function treats a pinhole projection as a baseline, and visualizes
deviations from this baseline. So wide lenses will have a lot of reported
"distortion".
This function has 3 modes of operation, specified as a string in the 'mode'
argument.
ARGUMENTS
- model: the mrcal.cameramodel object being evaluated
- mode: this function can produce several kinds of visualizations, with the
specific mode selected as a string in this argument. Known values:
- 'heatmap': the imager is gridded, as specified by the
gridn_width,gridn_height arguments. For each point in the grid, we evaluate
the difference in projection between the given model, and a pinhole model
with the same core intrinsics (focal lengths, center pixel coords). This
difference is color-coded and a heat map is displayed.
- 'vectorfield': this is the same as 'heatmap', except we display a vector for
each point, intead of a color-coded cell. If legibility requires the vectors
being larger or smaller, pass an appropriate value in the 'scale' argument
- 'radial': Looks at radial distortion only. Plots a curve showing the
magnitude of the radial distortion as a function of the distance to the
center
- scale: optional value, defaulting to 1.0. Used to scale the rendered vectors
if mode=='vectorfield'
- show_fisheye_projections: optional boolean defaulting to False. If
show_fisheye_projections: the radial plots include the behavior of common
fisheye projections, in addition to the behavior of THIS lens
- cbmax: optional value, defaulting to 25.0. Sets the maximum range of the color
map and of the contours if mode=='heatmap'
- gridn_width: how many points along the horizontal gridding dimension. Used if
mode=='vectorfield' or mode=='heatmap'
- gridn_height: how many points along the vertical gridding dimension. If None,
we compute an integer gridn_height to maintain a square-ish grid:
gridn_height/gridn_width ~ imager_height/imager_width. Used if
mode=='vectorfield' or mode=='heatmap'
- extratitle: optional string to include in the title of the resulting plot
- return_plot_args: boolean defaulting to False. if return_plot_args: we return
a (data_tuples, plot_options) tuple instead of making the plot. The plot can
then be made with gp.plot(*data_tuples, **plot_options). Useful if we want to
include this as a part of a more complex plot
- **kwargs: optional arguments passed verbatim as plot options to gnuplotlib.
Useful to make hardcopies, etc
RETURNED VALUE
if not return_plot_args (the usual path): we return the gnuplotlib plot object.
The plot disappears when this object is destroyed (by the garbage collection,
for instance), so save this returned plot object into a variable, even if you're
not going to be doing anything with this object.
if return_plot_args: we return a (data_tuples, plot_options) tuple instead of
making the plot. The plot can then be made with gp.plot(*data_tuples,
**plot_options). Useful if we want to include this as a part of a more complex
plot
- show_geometry(models_or_extrinsics_rt_fromref, cameranames=None, cameras_Rt_plot_ref=None, frames_rt_toref=None, points=None, show_calobjects=True, axis_scale=1.0, object_width_n=None, object_height_n=None, object_spacing=0, calobject_warp=None, point_labels=None, return_plot_args=False, **kwargs)
- Visualize the world resulting from a calibration run
SYNOPSIS
# Visualize the geometry from some models on disk
models = [mrcal.cameramodel(m) for m in model_filenames]
plot1 = mrcal.show_geometry(models)
# Solve a calibration problem. Visualize the resulting geometry AND the
# observed calibration objects and points
...
mrcal.optimize(intrinsics,
extrinsics_rt_fromref,
frames_rt_toref,
points,
...)
plot2 = \
mrcal.show_geometry(extrinsics_rt_fromref,
frames_rt_toref = frames_rt_toref,
points = points)
This function visualizes the world described by a set of camera models. It shows
the geometry of the cameras themselves (each one is represented by the axes of
its coordinate system). If available (via a frames_rt_toref argument or from
model.optimization_inputs() in the given models), the geometry of the
calibration objects used to compute these models is shown also. We use
frames_rt_toref if this is given. If not, we use the optimization_inputs() from
the FIRST model that provides them.
This function can also be used to visualize the output (or input) of
mrcal.optimize(); the relevant parameters are all identical to those
mrcal.optimize() takes.
This function is the core of the mrcal-show-geometry tool.
All arguments except models_or_extrinsics_rt_fromref are optional.
ARGUMENTS
- models_or_extrinsics_rt_fromref: an iterable of mrcal.cameramodel objects or
(6,) rt arrays. A array of shape (N,6) works to represent N cameras. If
mrcal.cameramodel objects are given here and frames_rt_toref is omitted, we
get the frames_rt_toref from the first model that provides
optimization_inputs().
- cameranames: optional array of strings of labels for the cameras. If omitted,
we use generic labels. If given, the array must have the same length as
models_or_extrinsics_rt_fromref
- cameras_Rt_plot_ref: optional transformation(s). If omitted, we plot
everything in the camera reference coordinate system. If given, we use a
"plot" coordinate system with the transformation TO plot coordinates FROM the
reference coordinates given in this argument. This argument can be given as an
iterable of Rt transformations to use a different one for each camera (None
means "identity"). Or a single Rt transformation can be given to use that one
for ALL the cameras
- frames_rt_toref: optional array of shape (N,6). If given, each row of shape
(6,) is an rt transformation representing the transformation TO the reference
coordinate system FROM the calibration object coordinate system. The
calibration object then MUST be defined by passing in valid object_width_n,
object_height_n, object_spacing parameters. If frames_rt_toref is omitted or
None, we look for this data in the given camera models. I look at the given
models in order, and grab the frames from the first model that has them. If
none of the models have this data and frames_rt_toref is omitted or NULL, then
I don't plot any frames at all
- object_width_n: the number of horizontal points in the calibration object
grid. Required only if frames_rt_toref is not None
- object_height_n: the number of vertical points in the calibration object grid.
Required only if frames_rt_toref is not None
- object_spacing: the distance between adjacent points in the calibration
object. A square object is assumed, so the vertical and horizontal distances
are assumed to be identical. Required only if frames_rt_toref is not None
- calobject_warp: optional (2,) array describing the calibration board warping.
None means "no warping": the object is flat. Used only if frames_rt_toref is
not None. See the docs for mrcal.ref_calibration_object() for a description.
- points: optional array of shape (N,3). If omitted, we don't plot the observed
points. If given, each row of shape (3,) is a point in the reference
coordinate system.
- point_labels: optional dict from a point index to a string describing it.
Points in this dict are plotted with this legend; all other points are plotted
under a generic "points" legend. As many or as few of the points may be
labelled in this way. If omitted, none of the points will be labelled
specially. This is used only if points is not None
- show_calobjects: optional boolean defaults to True. if show_calobjects: we
render the observed calibration objects (if they are available in
frames_rt_toref of model.optimization_inputs())
- axis_scale: optional scale factor for the size of the axes used to represent
the cameras. Can be omitted to use some reasonable default size, but for very
large or very small problems, this may be required to make the plot look right
- return_plot_args: boolean defaulting to False. if return_plot_args: we return
a (data_tuples, plot_options) tuple instead of making the plot. The plot can
then be made with gp.plot(*data_tuples, **plot_options). Useful if we want to
include this as a part of a more complex plot
- **kwargs: optional arguments passed verbatim as plot options to gnuplotlib.
Useful to make hardcopies, set the plot title, etc.
RETURNED VALUES
if not return_plot_args (the usual path): we return the gnuplotlib plot object.
The plot disappears when this object is destroyed (by the garbage collection,
for instance), so save this returned plot object into a variable, even if you're
not going to be doing anything with this object.
if return_plot_args: we return a (data_tuples, plot_options) tuple instead of
making the plot. The plot can then be made with gp.plot(*data_tuples,
**plot_options). Useful if we want to include this as a part of a more complex
plot
- show_projection_diff(models, gridn_width=60, gridn_height=None, observations=False, valid_intrinsics_region=False, distance=None, use_uncertainties=True, focus_center=None, focus_radius=-1.0, implied_Rt10=None, vectorfield=False, vectorscale=1.0, directions=False, extratitle=None, cbmax=4, return_plot_args=False, **kwargs)
- Visualize the difference in projection between N models
SYNOPSIS
models = ( mrcal.cameramodel('cam0-dance0.cameramodel'),
mrcal.cameramodel('cam0-dance1.cameramodel') )
mrcal.show_projection_diff(models)
# A plot pops up displaying the projection difference between the two models
The operation of this tool is documented at
http://mrcal.secretsauce.net/differencing.html
It is often useful to compare the projection behavior of two camera models. For
instance, one may want to evaluate the quality of a calibration by comparing the
results of two different chessboard dances. Or one may want to evaluate the
stability of the intrinsics in response to mechanical or thermal stresses. This
function makes these comparisons, and produces a visualization of the results.
mrcal.projection_diff() computes the differences, and returns the results
WITHOUT making plots.
In the most common case we're given exactly 2 models to compare. We then display
the projection difference as either a vector field or a heat map. If we're given
more than 2 models, then a vector field isn't possible and we instead display as
a heatmap the standard deviation of the differences between models 1..N and
model0.
The details of how the comparison is computed, and the meaning of the arguments
controlling this, are in the docstring of mrcal.projection_diff().
ARGUMENTS
- models: iterable of mrcal.cameramodel objects we're comparing. Usually there
will be 2 of these, but more than 2 is possible. The intrinsics are used; the
extrinsics are NOT.
- gridn_width: optional value, defaulting to 60. How many points along the
horizontal gridding dimension
- gridn_height: how many points along the vertical gridding dimension. If None,
we compute an integer gridn_height to maintain a square-ish grid:
gridn_height/gridn_width ~ imager_height/imager_width
- observations: optional boolean, defaulting to False. If True, we overlay
calibration-time observations on top of the difference plot. We should then
see that more data produces more consistent results.
- valid_intrinsics_region: optional boolean, defaulting to False. If True, we
overlay the valid-intrinsics regions onto the plot. If the valid-intrinsics
regions aren't available, we will silently omit them
- distance: optional value, defaulting to None. The projection difference varies
depending on the range to the observed world points, with the queried range
set in this 'distance' argument. If None (the default) we look out to
infinity. We can compute the implied-by-the-intrinsics transformation off
multiple distances if they're given here as an iterable. This is especially
useful if we have uncertainties, since then we'll emphasize the best-fitting
distances.
- use_uncertainties: optional boolean, defaulting to True. If True we use the
whole imager to fit the implied-by-the-intrinsics transformation, using the
uncertainties to emphasize the confident regions. If False, it is important to
select the confident region using the focus_center and focus_radius arguments.
If use_uncertainties is True, but that data isn't available, we report a
warning, and try to proceed without.
- focus_center: optional array of shape (2,); the imager center by default. Used
to indicate that the implied-by-the-intrinsics transformation should use only
those pixels a distance focus_radius from focus_center. This is intended to be
used if no uncertainties are available, and we need to manually select the
focus region.
- focus_radius: optional value. If use_uncertainties then the default is LARGE,
to use the whole imager. Else the default is min(width,height)/6. Used to
indicate that the implied-by-the-intrinsics transformation should use only
those pixels a distance focus_radius from focus_center. This is intended to be
used if no uncertainties are available, and we need to manually select the
focus region. Pass focus_radius=0 to avoid computing the transformation, and
to use the identity. This would mean there're no geometric differences, and
we're comparing the intrinsics only
- implied_Rt10: optional Rt transformation (numpy array of shape (4,3)). If
given, I use the given value for the implied-by-the-intrinsics transformation
instead of fitting it. If omitted, I compute the transformation. Exclusive
with focus_center, focus_radius. Valid only if exactly two models are given.
- vectorfield: optional boolean, defaulting to False. By default we produce a
heat map of the projection differences. If vectorfield: we produce a vector
field instead. This is more busy, and is often less clear at first glance, but
unlike a heat map, this shows the directions of the differences in addition to
the magnitude. This is only valid if we're given exactly two models to compare
- vectorscale: optional value, defaulting to 1.0. Applicable only if
vectorfield. The magnitude of the errors displayed in the vector field is
often very small, and impossible to make out when looking at the whole imager.
This argument can be used to scale all the displayed vectors to improve
legibility.
- directions: optional boolean, defaulting to False. By default the plot is
color-coded by the magnitude of the difference vectors. If directions: we
color-code by the direction instead. This is especially useful if we're
plotting a vector field. This is only valid if we're given exactly two models
to compare
- extratitle: optional string to include in the title of the resulting plot
- cbmax: optional value, defaulting to 4.0. Sets the maximum range of the color
map
- return_plot_args: boolean defaulting to False. if return_plot_args: we return
a (data_tuples, plot_options) tuple instead of making the plot. The plot can
then be made with gp.plot(*data_tuples, **plot_options). Useful if we want to
include this as a part of a more complex plot
- **kwargs: optional arguments passed verbatim as plot options to gnuplotlib.
Useful to make hardcopies, etc
RETURNED VALUE
A tuple:
- if not return_plot_args (the usual path): the gnuplotlib plot object. The plot
disappears when this object is destroyed (by the garbage collection, for
instance), so save this returned plot object into a variable, even if you're
not going to be doing anything with this object.
if return_plot_args: a (data_tuples, plot_options) tuple. The plot can then be
made with gp.plot(*data_tuples, **plot_options). Useful if we want to include
this as a part of a more complex plot
- implied_Rt10: the geometric Rt transformation in an array of shape (...,4,3).
This is either whatever was passed into this function (if anything was), or
the identity if focus_radius==0 or the fitted results. if len(models)>1: this
is an array of shape (len(models)-1,4,3), with slice i representing the
transformation between camera 0 and camera i+1.
- show_projection_uncertainty(model, gridn_width=60, gridn_height=None, observations=False, valid_intrinsics_region=False, distance=None, isotropic=False, extratitle=None, cbmax=3, contour_increment=None, contour_labels_styles='boxed', contour_labels_font=None, return_plot_args=False, **kwargs)
- Visualize the uncertainty in camera projection
SYNOPSIS
model = mrcal.cameramodel('xxx.cameramodel')
mrcal.show_projection_uncertainty(model)
... A plot pops up displaying the expected projection uncertainty across the
... imager
This function uses the expected noise of the calibration-time observations to
estimate the uncertainty of projection of the final model. At calibration time we estimate
- The intrinsics (lens paramaters) of a number of cameras
- The extrinsics (geometry) of a number of cameras in respect to some reference
coordinate system
- The poses of observed chessboards, also in respect to some reference
coordinate system
All the coordinate systems move around, and all 3 of these sets of data have
some uncertainty. This tool takes into account all the uncertainties to report
an estimated uncertainty metric. See
http://mrcal.secretsauce.net/uncertainty.html for a detailed description of
the computation.
This function grids the imager, and reports an uncertainty for each point on the
grid. The resulting plot contains a heatmap of the uncertainties for each cell
in the grid, and corresponding contours.
Since the projection uncertainty is based on calibration-time observation
uncertainty, it is sometimes useful to see where the calibration-time
observations were. Pass observations=True to do that.
Since the projection uncertainty is based partly on the uncertainty of the
camera pose, points at different distance from the camera will have different
reported uncertainties EVEN IF THEY PROJECT TO THE SAME PIXEL. The queried
distance is passed in the distance argument. If distance is None (the default)
then we look out to infinity.
To see a 3D view showing the calibration-time observations AND uncertainties for
a set of distances at the same time, call show_projection_uncertainty_xydist()
instead.
For each cell we compute the covariance matrix of the projected (x,y) coords,
and by default we report the worst-direction standard deviation. If isotropic:
we report the RMS standard deviation instead.
ARGUMENTS
- model: the mrcal.cameramodel object being evaluated
- gridn_width: optional value, defaulting to 60. How many points along the
horizontal gridding dimension
- gridn_height: how many points along the vertical gridding dimension. If None,
we compute an integer gridn_height to maintain a square-ish grid:
gridn_height/gridn_width ~ imager_height/imager_width
- observations: optional boolean, defaulting to False. If True, we overlay
calibration-time observations on top of the uncertainty plot. We should then
see that more data produces more confident results.
- valid_intrinsics_region: optional boolean, defaulting to False. If True, we
overlay the valid-intrinsics region onto the plot. If the valid-intrinsics
region isn't available, we will silently omit it
- distance: optional value, defaulting to None. The projection uncertainty
varies depending on the range to the observed point, with the queried range
set in this 'distance' argument. If None (the default) we look out to
infinity.
- isotropic: optional boolean, defaulting to False. We compute the full 2x2
covariance matrix of the projection. The 1-sigma contour implied by this
matrix is an ellipse, and we use the worst-case direction by default. If we
want the RMS size of the ellipse instead of the worst-direction size, pass
isotropic=True.
- extratitle: optional string to include in the title of the resulting plot
- cbmax: optional value, defaulting to 3.0. Sets the maximum range of the color
map
- contour_increment: optional value, defaulting to None. If given, this will be
used as the distance between adjacent contours. If omitted of None, a
reasonable value will be estimated
- contour_labels_styles: optional string, defaulting to 'boxed'. The style of
the contour labels. This will be passed to gnuplot as f"with labels
{contour_labels_styles} nosurface". Can be used to box/unbox the label, set
the color, etc. To change the font use contour_labels_font.
- contour_labels_font: optional string, defaulting to None, If given, this is
the font string for the contour labels. Will be passed to gnuplot as f'set
cntrlabel font "{contour_labels_font}"'
- return_plot_args: boolean defaulting to False. if return_plot_args: we return
a (data_tuples, plot_options) tuple instead of making the plot. The plot can
then be made with gp.plot(*data_tuples, **plot_options). Useful if we want to
include this as a part of a more complex plot
- **kwargs: optional arguments passed verbatim as plot options to gnuplotlib.
Useful to make hardcopies, etc
RETURNED VALUE
if not return_plot_args (the usual path): we return the gnuplotlib plot object.
The plot disappears when this object is destroyed (by the garbage collection,
for instance), so save this returned plot object into a variable, even if you're
not going to be doing anything with this object.
if return_plot_args: we return a (data_tuples, plot_options) tuple instead of
making the plot. The plot can then be made with gp.plot(*data_tuples,
**plot_options). Useful if we want to include this as a part of a more complex
plot
- show_projection_uncertainty_vs_distance(model, where='centroid', isotropic=False, extratitle=None, return_plot_args=False, **kwargs)
- Visualize the uncertainty in camera projection along one observation ray
SYNOPSIS
model = mrcal.cameramodel('xxx.cameramodel')
mrcal.show_projection_uncertainty_vs_distance(model)
... A plot pops up displaying the expected projection uncertainty along an
... observation ray at different distances from the camera
This function is similar to show_projection_uncertainty(). That function
displays the uncertainty at different locations along the imager, for one
observation distance. Conversely, THIS function displays it in one location on
the imager, but at different distances.
This function uses the expected noise of the calibration-time observations to
estimate the uncertainty of projection of the final model. At calibration time
we estimate
- The intrinsics (lens paramaters) of a number of cameras
- The extrinsics (geometry) of a number of cameras in respect to some reference
coordinate system
- The poses of observed chessboards, also in respect to some reference
coordinate system
All the coordinate systems move around, and all 3 of these sets of data have
some uncertainty. This tool takes into account all the uncertainties to report
an estimated uncertainty metric. See
http://mrcal.secretsauce.net/uncertainty.html for a detailed description of
the computation.
The curve produced by this function has a characteristic shape:
- At low ranges, the camera translation dominates, and the uncertainty increases
to infinity, as the distance to the camera goes to 0
- As we move away from the camera, the uncertainty drops to a minimum, at around
the distance where the chessboards were observed
- Past the minimum, the uncertainty climbs to asymptotically approach the
uncertainty at infinity
ARGUMENTS
- model: the mrcal.cameramodel object being evaluated
- where: optional value, defaulting to "centroid". Indicates the point on the
imager we're examining. May be one of
- "center": the center of the imager
- "centroid": the midpoint of all the chessboard corners observed at
calibration time
- A numpy array (x,y) indicating the pixel
- isotropic: optional boolean, defaulting to False. We compute the full 2x2
covariance matrix of the projection. The 1-sigma contour implied by this
matrix is an ellipse, and we use the worst-case direction by default. If we
want the RMS size of the ellipse instead of the worst-direction size, pass
isotropic=True.
- extratitle: optional string to include in the title of the resulting plot
- return_plot_args: boolean defaulting to False. if return_plot_args: we return
a (data_tuples, plot_options) tuple instead of making the plot. The plot can
then be made with gp.plot(*data_tuples, **plot_options). Useful if we want to
include this as a part of a more complex plot
- **kwargs: optional arguments passed verbatim as plot options to gnuplotlib.
Useful to make hardcopies, etc
RETURNED VALUE
if not return_plot_args (the usual path): we return the gnuplotlib plot object.
The plot disappears when this object is destroyed (by the garbage collection,
for instance), so save this returned plot object into a variable, even if you're
not going to be doing anything with this object.
if return_plot_args: we return a (data_tuples, plot_options) tuple instead of
making the plot. The plot can then be made with gp.plot(*data_tuples,
**plot_options). Useful if we want to include this as a part of a more complex
plot
- show_projection_uncertainty_xydist(model, gridn_width=15, gridn_height=None, extratitle=None, cbmax=3, return_plot_args=False, **kwargs)
- Visualize in 3D the uncertainty in camera projection
SYNOPSIS
model = mrcal.cameramodel('xxx.cameramodel')
mrcal.show_projection_uncertainty_xydist(model)
... A plot pops up displaying the calibration-time observations and the
... expected projection uncertainty for various ranges and for various
... locations on the imager
This function is similar to show_projection_uncertainty(), but it visualizes the
uncertainty at multiple ranges at the same time.
This function uses the expected noise of the calibration-time observations to
estimate the uncertainty of projection of the final model. At calibration time
we estimate
- The intrinsics (lens paramaters) of a number of cameras
- The extrinsics (geometry) of a number of cameras in respect to some reference
coordinate system
- The poses of observed chessboards, also in respect to some reference
coordinate system
All the coordinate systems move around, and all 3 of these sets of data have
some uncertainty. This tool takes into account all the uncertainties to report
an estimated uncertainty metric. See
http://mrcal.secretsauce.net/uncertainty.html for a detailed description of
the computation.
This function grids the imager and a set of observation distances, and reports
an uncertainty for each point on the grid. It also plots the calibration-time
observations. The expectation is that the uncertainty will be low in the areas
where we had observations at calibration time.
ARGUMENTS
- model: the mrcal.cameramodel object being evaluated
- gridn_width: optional value, defaulting to 15. How many points along the
horizontal gridding dimension
- gridn_height: how many points along the vertical gridding dimension. If None,
we compute an integer gridn_height to maintain a square-ish grid:
gridn_height/gridn_width ~ imager_height/imager_width
- extratitle: optional string to include in the title of the resulting plot
- return_plot_args: boolean defaulting to False. if return_plot_args: we return
a (data_tuples, plot_options) tuple instead of making the plot. The plot can
then be made with gp.plot(*data_tuples, **plot_options). Useful if we want to
include this as a part of a more complex plot
- **kwargs: optional arguments passed verbatim as plot options to gnuplotlib.
Useful to make hardcopies, etc
RETURNED VALUE
if not return_plot_args (the usual path): we return the gnuplotlib plot object.
The plot disappears when this object is destroyed (by the garbage collection,
for instance), so save this returned plot object into a variable, even if you're
not going to be doing anything with this object.
if return_plot_args: we return a (data_tuples, plot_options) tuple instead of
making the plot. The plot can then be made with gp.plot(*data_tuples,
**plot_options). Useful if we want to include this as a part of a more complex
plot
- show_splined_model_surface(model, xy, imager_domain=False, valid_intrinsics_region=True, observations=False, extratitle=None, return_plot_args=False, **kwargs)
- Visualize the surface represented by a splined model
SYNOPSIS
model = mrcal.cameramodel(model_filename)
mrcal.show_splined_model_surface( model, 'x' )
... A plot pops up displaying the spline knots, the spline surface (for the
... "x" coordinate), the spline-in-bounds regions and the valid-intrinsics
... region
Splined models are parametried by flexible surfaces that define the projection,
and visualizing these surfaces is useful for understanding projection behavior
of a given lens. Details of these models are described in the documentation:
http://mrcal.secretsauce.net/lensmodels.html#splined-stereographic-lens-model
The surfaces are defined by control points we call "knots". These knots are
arranged in a fixed grid (defined by the model configuration) with the value at
each knot set in the intrinsics vector. The configuration selects the control
point density and the expected field of view of the lens. This field of view
should roughly match the actual lens+camera we're using, and this alignment can
be visualized with this function. This function displays the spline-in-bounds
region together with the usable projection region (either the valid-intrinsics
region or the imager bounds). Ideally, the spline-in-bounds region is slightly
bigger than the usable projection region.
If the fov_x_deg configuration value is too big, many of the knots will lie well
outside the visible area, and will not be used. This is wasteful. If fov_x_deg
is too small, then some parts of the imager will lie outside of the
spline-in-bounds region, resulting in less-flexible projection behavior at the
edges of the imager. This function detects this scenario, displays the offending
regions, and throws a warning.
The usable projection region visualized by this function is controlled by the
valid_intrinsics_region argument. If True (the default), we display the
valid-intrinsics region. This is recommended, but keep in mind that this region
is smaller than the full imager, so a fov_x_deg that aligns well for one
calibration may be too small in a subsequent calibration of the same lens. If
the subsequent calibration has better coverage, and thus a bigger
valid-intrinsics region. If not valid_intrinsics_region: we use the imager
bounds instead. The issue here is that the projection near the edges of the
imager is usually poorly-defined because usually there isn't a lot of
calibration data there. This makes the projection behavior at the imager edges
unknowable. Consequently, plotting the projection at the imager edges is usually
too alarming or not alarming enough. Passing valid_intrinsics_region=False is
thus recommended only if we have very good calibration coverage at the edge of
the imager.
Another method of visualizing the usable projection region is to look at the
individual chessboard corners by passing observations=True. The valid-intrinsics
region largely corresponds to the area where the observations were available.
This function can produce a plot in the imager domain or in the spline index
domain. Both are useful, and this is controlled by the imager_domain argument
(the default is False). The spline is defined in the stereographic projection
domain, so in the imager domain the knot grid and the domain boundary become
skewed. Note: if calibration data coverage is missing at the edges (a very
common case), the edges of the spline-in-bounds region will have poorly-defined
projection, and will look very strange if imager_domain. Thus the default of
imager_domain==False is recommended.
ARGUMENTS
- model: the mrcal.cameramodel object being evaluated
- xy: 'x' or 'y': selects the surface we're looking at. We have a separate
surface for the x and y coordinates, with the two sharing the knot positions
- imager_domain: optional boolean defaults to False. If False: we plot
everything against normalized stereographic coordinates; in this
representation the knots form a regular grid, and the surface domain is a
rectangle, but the imager boundary is curved. If True: we plot everything
against the rendered pixel coordinates; the imager boundary is a rectangle,
while the knots and domain become curved
- valid_intrinsics_region: optional boolean defaults to True. If True: we
communicate the usable projection region to the user by displaying the
valid-intrinsics region. This isn't available in all models. To fall back on
the boundary of the full imager, pass False here. In the usual case of
incomplete calibration-time coverage at the edges, this results in a very
unrealistic representation of reality. Passing True here is strongly
recommended
- observations: optional boolean defaults to False. If True: we plot the
calibration-time point observations on top of the surface and the knots. These
make it more clear if the unprojectable regions in the model really are a
problem
- extratitle: optional string to include in the title of the resulting plot
- return_plot_args: boolean defaulting to False. if return_plot_args: we return
a (data_tuples, plot_options) tuple instead of making the plot. The plot can
then be made with gp.plot(*data_tuples, **plot_options). Useful if we want to
include this as a part of a more complex plot
- **kwargs: optional arguments passed verbatim as plot options to gnuplotlib.
Useful to make hardcopies, etc
RETURNED VALUES
if not return_plot_args (the usual path): we return the gnuplotlib plot object.
The plot disappears when this object is destroyed (by the garbage collection,
for instance), so save this returned plot object into a variable, even if you're
not going to be doing anything with this object.
if return_plot_args: we return a (data_tuples, plot_options) tuple instead of
making the plot. The plot can then be made with gp.plot(*data_tuples,
**plot_options). Useful if we want to include this as a part of a more complex
plot
- show_valid_intrinsics_region(models, cameranames=None, image=None, points=None, return_plot_args=False, **kwargs)
- Visualize a model's valid-intrinsics region
SYNOPSIS
filenames = ('cam0-dance0.cameramodel',
'cam0-dance1.cameramodel')
models = [ mrcal.cameramodel(f) for f in filenames ]
mrcal.show_valid_intrinsics_region( models,
cameranames = filenames,
image = 'image.jpg' )
This function displays the valid-intrinsics region in the given camera models.
Multiple models can be passed-in to see their valid-intrinsics regions together.
This is useful to evaluate different calibrations of the same lens. A captured
image can be passed-in to see the regions overlaid on an actual image produced
by the camera.
All given models MUST have a valid-intrinsics region defined. A model may have
an empty region. This cannot be plotted (there's no contour to plot), but the
plot legend will still contain an entry for this model, with a note indicating
its emptiness
This tool produces a gnuplotlib plot. To annotate an image array, call
annotate_image__valid_intrinsics_region() instead
ARGUMENTS
- models: an iterable of mrcal.cameramodel objects we're visualizing. If we're
looking at just a single model, it can be passed directly in this argument,
instead of wrapping it into a list.
- cameranames: optional an iterable of labels, one for each model. These will
appear as the legend in the plot. If omitted, we will simply enumerate the
models.
- image: optional image to annotate. May be given as an image filename or an
array of image data. If omitted, we plot the valid-intrinsics region only.
- points: optional array of shape (N,2) of pixel coordinates to plot. If given,
we show these arbitrary points in our plot. Useful to visualize the feature
points used in a vision algorithm to see how reliable they are expected to be
- return_plot_args: boolean defaulting to False. if return_plot_args: we return
a (data_tuples, plot_options) tuple instead of making the plot. The plot can
then be made with gp.plot(*data_tuples, **plot_options). Useful if we want to
include this as a part of a more complex plot
- **kwargs: optional arguments passed verbatim as plot options to gnuplotlib.
Useful to make hardcopies, etc
RETURNED VALUE
A tuple:
- if not return_plot_args (the usual path): the gnuplotlib plot object. The plot
disappears when this object is destroyed (by the garbage collection, for
instance), so save this returned plot object into a variable, even if you're
not going to be doing anything with this object.
if return_plot_args: a (data_tuples, plot_options) tuple. The plot can then be
made with gp.plot(*data_tuples, **plot_options). Useful if we want to include
this as a part of a more complex plot
- state_index_calobject_warp(...)
- Return the index in the optimization vector of the calibration object warp
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
p = mrcal.optimizer_callback(**optimization_inputs)[0]
mrcal.unpack_state(p, **optimization_inputs)
i_state = mrcal.state_index_calobject_warp(**optimization_inputs)
calobject_warp = p[i_state:i_state+2]
The optimization algorithm sees its world described in one, big vector of state.
The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.state_index_...() functions report where particular items end up in the
state vector.
THIS function reports the beginning of the calibration-object warping parameters
in the state vector. This is stored contiguously as a 2-element vector. These
warping parameters describe how the observed calibration object differs from the
expected calibration object. There will always be some difference due to
manufacturing tolerances and temperature and humidity effects.
In order to determine the variable mapping, we need quite a bit of context. If
we have the full set of inputs to the optimization function, we can pass in
those (as shown in the example above). Or we can pass the individual arguments
that are needed (see ARGUMENTS section for the full list). If the optimization
inputs and explicitly-given arguments conflict about the size of some array, the
explicit arguments take precedence. If any array size is not specified, it is
assumed to be 0. Thus most arguments are optional.
ARGUMENTS
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_calobject_warp
do_optimize_frames
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
Nobservations_board
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting the location in the state vector where the contiguous
block of variables for the calibration object warping begins
- state_index_extrinsics(...)
- Return the index in the optimization vector of the extrinsics of camera i
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
p = mrcal.optimizer_callback(**optimization_inputs)[0]
mrcal.unpack_state(p, **optimization_inputs)
icam_extrinsics = 1
i_state = mrcal.state_index_extrinsics(icam_extrinsics,
**optimization_inputs)
extrinsics_rt_fromref = p[i_state:i_state+6]
The optimization algorithm sees its world described in one, big vector of state.
The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.state_index_...() functions report where particular items end up in the
state vector.
THIS function reports the beginning of the i-th camera extrinsics in the state
vector. The extrinsics are stored contiguously as an "rt transformation": a
3-element rotation represented as a Rodrigues vector followed by a 3-element
translation. These transform points represented in the reference coordinate
system to the coordinate system of the specific camera. Note that mrcal allows
the reference coordinate system to be tied to a particular camera. In this case
the extrinsics of that camera do not appear in the state vector at all, and
icam_extrinsics == -1 in the indices_frame_camintrinsics_camextrinsics
array.
In order to determine the variable mapping, we need quite a bit of context. If
we have the full set of inputs to the optimization function, we can pass in
those (as shown in the example above). Or we can pass the individual arguments
that are needed (see ARGUMENTS section for the full list). If the optimization
inputs and explicitly-given arguments conflict about the size of some array, the
explicit arguments take precedence. If any array size is not specified, it is
assumed to be 0. Thus most arguments are optional.
ARGUMENTS
- icam_extrinsics: an integer indicating which camera we're asking about
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_calobject_warp
do_optimize_frames
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
Nobservations_board
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting the location in the state vector where the contiguous
block of extrinsics for camera icam_extrinsics begins
- state_index_frames(...)
- Return the index in the optimization vector of the pose of frame i
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
p = mrcal.optimizer_callback(**optimization_inputs)[0]
mrcal.unpack_state(p, **optimization_inputs)
iframe = 1
i_state = mrcal.state_index_frames(iframe,
**optimization_inputs)
frames_rt_toref = p[i_state:i_state+6]
The optimization algorithm sees its world described in one, big vector of state.
The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.state_index_...() functions report where particular items end up in the
state vector.
THIS function reports the beginning of the i-th frame pose in the state vector.
Here a "frame" is a pose of the observed calibration object at some instant in
time. The frames are stored contiguously as an "rt transformation": a 3-element
rotation represented as a Rodrigues vector followed by a 3-element translation.
These transform points represented in the internal calibration object coordinate
system to the reference coordinate system.
In order to determine the variable mapping, we need quite a bit of context. If
we have the full set of inputs to the optimization function, we can pass in
those (as shown in the example above). Or we can pass the individual arguments
that are needed (see ARGUMENTS section for the full list). If the optimization
inputs and explicitly-given arguments conflict about the size of some array, the
explicit arguments take precedence. If any array size is not specified, it is
assumed to be 0. Thus most arguments are optional.
ARGUMENTS
- iframe: an integer indicating which frame we're asking about
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_calobject_warp
do_optimize_frames
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
Nobservations_board
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting the location in the state vector where the contiguous
block of variables for frame iframe begins
- state_index_intrinsics(...)
- Return the index in the optimization vector of the intrinsics of camera i
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
p = mrcal.optimizer_callback(**optimization_inputs)[0]
mrcal.unpack_state(p, **optimization_inputs)
icam_intrinsics = 1
i_state = mrcal.state_index_intrinsics(icam_intrinsics,
**optimization_inputs)
Nintrinsics = mrcal.lensmodel_num_params(optimization_inputs['lensmodel'])
intrinsics_data = p[i_state:i_state+Nintrinsics]
The optimization algorithm sees its world described in one, big vector of state.
The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.state_index_...() functions report where particular items end up in the
state vector.
THIS function reports the beginning of the i-th camera intrinsics in the state
vector. The intrinsics are stored contiguously. They consist of a 4-element
"intrinsics core" (focallength-x, focallength-y, centerpixel-x, centerpixel-y)
followed by a lensmodel-specific vector of "distortions". The number of
intrinsics elements (including the core) for a particular lens model can be
queried with mrcal.lensmodel_num_params(lensmodel). Note that
do_optimize_intrinsics_core and do_optimize_intrinsics_distortions can be used
to lock down one or both of those quantities, which would omit them from the
optimization vector.
In order to determine the variable mapping, we need quite a bit of context. If
we have the full set of inputs to the optimization function, we can pass in
those (as shown in the example above). Or we can pass the individual arguments
that are needed (see ARGUMENTS section for the full list). If the optimization
inputs and explicitly-given arguments conflict about the size of some array, the
explicit arguments take precedence. If any array size is not specified, it is
assumed to be 0. Thus most arguments are optional.
ARGUMENTS
- icam_intrinsics: an integer indicating which camera we're asking about
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_calobject_warp
do_optimize_frames
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
Nobservations_board
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting the location in the state vector where the contiguous
block of intrinsics for camera icam_intrinsics begins
- state_index_points(...)
- Return the index in the optimization vector of the position of point i
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
p = mrcal.optimizer_callback(**optimization_inputs)[0]
mrcal.unpack_state(p, **optimization_inputs)
i_point = 1
i_state = mrcal.state_index_points(i_point,
**optimization_inputs)
point = p[i_state:i_state+3]
The optimization algorithm sees its world described in one, big vector of state.
The optimizer doesn't know or care about the meaning of each element of this
vector, but for later analysis, it is useful to know what's what. The
mrcal.state_index_...() functions report where particular items end up in the
state vector.
THIS function reports the beginning of the i-th point in the state vector. The
points are stored contiguously as a 3-element coordinates in the reference
frame.
In order to determine the variable mapping, we need quite a bit of context. If
we have the full set of inputs to the optimization function, we can pass in
those (as shown in the example above). Or we can pass the individual arguments
that are needed (see ARGUMENTS section for the full list). If the optimization
inputs and explicitly-given arguments conflict about the size of some array, the
explicit arguments take precedence. If any array size is not specified, it is
assumed to be 0. Thus most arguments are optional.
ARGUMENTS
- i_point: an integer indicating which point we're asking about
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_calobject_warp
do_optimize_frames
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
Nobservations_board
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
The integer reporting the location in the state vector where the contiguous
block of variables for point i_point begins
- stereo_range(disparity_pixels, baseline, pixels_per_deg_az, az=None, az_row=None, **kwargs)
- Compute ranges from observed disparities
SYNOPSIS
...
rectification_maps,cookie = \
mrcal.stereo_rectify_prepare(models, ...)
...
disparity16 = stereo.compute(*images_rectified) # in pixels*16
r = mrcal.stereo_range( disparity16.astype(np.float32) / 16.,
**cookie )
See the docstring stereo_rectify_prepare() for a complete example, and for a
description of the rectified space and the meaning of "azimuth" and "elevations"
in this context.
The rectification maps from stereo_rectify_prepare() can be used to produce
rectified images, which can then be processed by a stereo-matching routine to
get a disparity image. The next step is converting the disparities to ranges,
and is implemented by this function. If instead of ranges we want 3D points,
call stereo_unproject() instead.
In the most common usage we produce a disparity IMAGE, and then convert it to a
range IMAGE. In this common case we can simply call
r = mrcal.stereo_range(disparity, **cookie)
Where the cookie was returned to us by mrcal.stereo_rectify_prepare(). It is a
dict that contains all the information needed to interpret the disparities.
If we aren't processing the full disparity IMAGE, we can still use this
function. Pass in the disparities of any shape. And the corresponding azimuths
in the "az" argument. The "az" and "disparity_pixels" arrays describe
corresponding points. They may have identical dimensions. If there's any
redundancy (if every row of the disparity array has identical azimuths for
instance), then being broadcasting-compatible is sufficient. A very common case
of this is disparity images. In this case the disparity array has shape
(Nel,Naz), with each row containing an identical sequences of azimuths. We can
then pass an azimuth array of shape (Naz,).
The azimuths may also be passed in the "az_row" argument. This is usually not
done by the user directly, but via the **cookie from stereo_rectify_prepare().
At least one of "az", "az_row" must be given. If both are given, we use "az".
ARGUMENTS
- disparity_pixels: the numpy array of disparities being processed. This
array contains disparity in PIXELS. If the stereo-matching algorithm you're
using reports a scaled value, you need to convert it to pixels, possibly
converting to floating-point in the process. Any array shape is supported. In
the common case of a disparity IMAGE, this is an array of shape (Nel, Naz)
- baseline: the distance between the two stereo cameras. This is required, and
usually it comes from **cookie, with the cookie returned by
stereo_rectify_prepare()
- pixels_per_deg_az: the angular resolution of the disparity image. This is
required, and usually it comes from **cookie, with the cookie returned by
stereo_rectify_prepare(). Don't pass this yourself. This argument may go away
in the future
- az: optional array of azimuths corresponding to the given disparities. If
processing normal disparity images, this can be omitted, and we'll use az_row
from **cookie, with the cookie returned by stereo_rectify_prepare(). If
processing anything other than a full disparity image, pass the azimuths here.
The az array must be broadcasting-compatible with disparity_pixels. The
dimensions of the two arrays may be identical, but brodcasting can be used to
exploit any redundancy. For instance, disparity images have shape (Nel, Naz)
with each row containing identical azimuths. We can thus work with az of shape
(Naz,). At least one of az and az_row must be non-None. If both are given, we
use az.
- az_row: optional array of azimuths in the disparity_pixels array. Works
exactly like the "az" array. At least one MUST be given. If both are given, we
use "az". Usually az_row is not given by the user directly, but via a **cookie
from stereo_rectify_prepare()
RETURNED VALUES
- An array of ranges of the same dimensionality as the input disparity_pixels
array. Contains floating-point data. Invalid or missing ranges are represented
as 0.
- stereo_rectify_prepare(models, az_fov_deg, el_fov_deg, az0_deg=None, el0_deg=0, pixels_per_deg_az=None, pixels_per_deg_el=None)
- Precompute everything needed for stereo rectification and matching
SYNOPSIS
import sys
import mrcal
import cv2
import numpy as np
# Read commandline arguments: model0 model1 image0 image1
models = [ mrcal.cameramodel(sys.argv[1]),
mrcal.cameramodel(sys.argv[2]), ]
images = [ cv2.imread(sys.argv[i]) \
for i in (3,4) ]
# Prepare the stereo system
rectification_maps,cookie = \
mrcal.stereo_rectify_prepare(models,
az_fov_deg = 120,
el_fov_deg = 100)
# Visualize the geometry of the two cameras and of the rotated stereo
# coordinate system
Rt_cam0_ref = models[0].extrinsics_Rt_fromref()
Rt_cam0_stereo = cookie['Rt_cam0_stereo']
Rt_stereo_ref = mrcal.compose_Rt( mrcal.invert_Rt(Rt_cam0_stereo),
Rt_cam0_ref )
rt_stereo_ref = mrcal.rt_from_Rt(Rt_stereo_ref)
mrcal.show_geometry( models + [ rt_stereo_ref ],
( "camera0", "camera1", "stereo" ),
show_calobjects = False,
wait = True )
# Rectify the images
images_rectified = \
[ mrcal.transform_image(images[i], rectification_maps[i]) \
for i in range(2) ]
cv2.imwrite('/tmp/rectified0.jpg', images_rectified[0])
cv2.imwrite('/tmp/rectified1.jpg', images_rectified[1])
# Find stereo correspondences using OpenCV
block_size = 3
max_disp = 160 # in pixels
stereo = \
cv2.StereoSGBM_create(minDisparity = 0,
numDisparities = max_disp,
blockSize = block_size,
P1 = 8 *3*block_size*block_size,
P2 = 32*3*block_size*block_size,
uniquenessRatio = 5,
disp12MaxDiff = 1,
speckleWindowSize = 50,
speckleRange = 1)
disparity16 = stereo.compute(*images_rectified) # in pixels*16
cv2.imwrite('/tmp/disparity.png',
mrcal.apply_color_map(disparity16,
0, max_disp*16.))
# Convert the disparities to range to camera0
r = mrcal.stereo_range( disparity16.astype(np.float32) / 16.,
**cookie )
cv2.imwrite('/tmp/range.png', mrcal.apply_color_map(r, 5, 1000))
This function does the initial computation required to perform stereo matching,
and to get ranges from a stereo pair. It computes
- the pose of the rectified stereo coordinate system
- the azimuth/elevation grid used in the rectified images
- the rectification maps used to transform images into the rectified space
Using the results of one call to this function we can compute the stereo
disparities of many pairs of synchronized images.
This function is generic: the two cameras may have any lens models, any
resolution and any geometry. They don't even have to match. As long as there's
some non-zero baseline and some overlapping views, we can set up stereo matching
using this function. The input images are tranformed into a "rectified" space.
Geometrically, the rectified coordinate system sits at the origin of camera0,
with a rotation. The axes of the rectified coordinate system:
- x: from the origin of camera0 to the origin of camera1 (the baseline direction)
- y: completes the system from x,z
- z: the "forward" direction of the two cameras, with the component parallel to
the baseline subtracted off
In a nominal geometry (the two cameras are square with each other, camera1
strictly to the right of camera0), the rectified coordinate system exactly
matches the coordinate system of camera0. The above formulation supports any
geometry, however, including vertical and/or forward/backward shifts. Vertical
stereo is supported.
Rectified images represent 3D planes intersecting the origins of the two
cameras. The tilt of each plane is the "elevation". While the left/right
direction inside each plane is the "azimuth". We generate rectified images where
each pixel coordinate represents (x = azimuth, y = elevation). Thus each row
scans the azimuths in a particular elevation, and thus each row in the two
rectified images represents the same plane in 3D, and matching features in each
row can produce a stereo disparity and a range.
In the rectified system, elevation is a rotation along the x axis, while azimuth
is a rotation normal to the resulting tilted plane.
We produce rectified images whose pixel coordinates are linear with azimuths and
elevations. This means that the azimuth angular resolution is constant
everywhere, even at the edges of a wide-angle image.
We return a set of transformation maps and a cookie. The maps can be used to
generate rectified images. These rectified images can be processed by any
stereo-matching routine to generate a disparity image. To interpret the
disparity image, call stereo_unproject() or stereo_range() with the cookie
returned here.
The cookie is a Python dict that describes the rectified space. It is guaranteed
to have the following keys:
- Rt_cam0_stereo: an Rt transformation to map a representation of points in the
rectified coordinate system to a representation in the camera0 coordinate system
- baseline: the distance between the two cameras
- az_row: an array of shape (Naz,) describing the azimuths in each row of the
disparity image
- el_col: an array of shape (Nel,1) describing the elevations in each column of
the disparity image
ARGUMENTS
- models: an iterable of two mrcal.cameramodel objects representing the cameras
in the stereo system. Any sane combination of lens models and resolutions and
geometries is valid
- az_fov_deg: required value for the azimuth (along-the-baseline) field-of-view
of the desired rectified view, in pixels
- el_fov_deg: required value for the elevation (across-the-baseline)
field-of-view of the desired rectified view, in pixels
- az0_deg: optional value for the azimuth center of the rectified images. This
is especially significant in a camera system with some forward/backward shift.
That causes the baseline to no longer be perpendicular with the view axis of
the cameras, and thus azimuth = 0 is no longer at the center of the input
images. If omitted, we compute the center azimuth that aligns with the center
of the cameras' view
- el0_deg: optional value for the elevation center of the rectified system.
Defaults to 0.
- pixels_per_deg_az: optional value for the azimuth resolution of the rectified
image. If omitted (or None), we use the resolution of the input image at
(azimuth, elevation) = 0. If a resolution of <0 is requested, we use this as a
scale factor on the resolution of the input image. For instance, to downsample
by a factor of 2, pass pixels_per_deg_az = -0.5
- pixels_per_deg_el: same as pixels_per_deg_az but in the elevation direction
RETURNED VALUES
We return a tuple
- transformation maps: a tuple of length-2 containing transformation maps for
each camera. Each map can be used to mrcal.transform_image() images to the
rectified space
- cookie: a dict describing the rectified space. Intended as input to
stereo_unproject() and stereo_range(). See the description above for more
detail
- stereo_unproject(az=None, el=None, disparity_pixels=None, baseline=None, pixels_per_deg_az=None, get_gradients=False, az_row=None, el_col=None, **kwargs)
- Unprojection in the rectified stereo system
SYNOPSIS
...
rectification_maps,cookie = \
mrcal.stereo_rectify_prepare(models, ...)
...
disparity16 = stereo.compute(*images_rectified) # in pixels*16
disparity_pixels = disparity16.astype(np.float32) / 16.
p_stereo = \
mrcal.stereo_unproject( disparity_pixels = disparity_pixels,
**cookie )
p_cam0 = mrcal.transform_point_Rt( cookie['Rt_cam0_stereo'],
p_stereo )
See the docstring stereo_rectify_prepare() for a complete example, and for a
description of the rectified space and the meaning of "azimuth" and "elevations"
in this context.
This function supports several kinds of rectified-space unprojections:
- normalized (range-less) unprojections
This is the simplest mode. Two arguments are accepted: "az" and "el", and an
array of unit vectors is returned. The default value of disparity_pixels = None
selects this mode. "az" and "el" may have identical dimensions, or may be
broadcasting compatible. A very common case is evaluating a rectified image
where each row represents an identical sequence azimuths and every column
represents an identical sequence of elevations. We can use broadcasting to pass
in "az" of shape (Naz,) and "el" of shape (Nel,1) to get an output array of
vectors of shape (Nel,Naz,3)
- normalized (range-less) unprojections with gradients
This is identical as the previous mode, but we return the vectors AND their
gradients in respect to az,el. Pass in az, el as before, but ALSO pass in
get_gradients = True. Same dimensionality logic as before applies. We return (v,
dv_dazel) where v.shape is (..., 3) and dv_dazel.shape is (..., 3,2)
- ranged unprojections without gradients
Here we use a disparity image to compute ranges, and use them to scale the
unprojected direction vectors. This is very similar to stereo_range(), but we
report a 3D point, not just ranges.
The rectification maps from stereo_rectify_prepare() can be used to produce
rectified images, which can then be processed by a stereo-matching routine to
get a disparity image. The next step is converting the disparities to 3D points,
and is implemented by this function. If instead of 3D points we just want ranges,
call stereo_range() instead.
A common usage is to produce a disparity IMAGE, and then convert it to a point
cloud. In this common case we can call
r = mrcal.stereo_unproject(disparity_pixels = disparity_pixels,
**cookie)
Where the cookie was returned to us by mrcal.stereo_rectify_prepare(). It is a
dict that contains all the information needed to interpret the disparities.
If we aren't processing the full disparity IMAGE, we can still use this
function. Pass in the disparities of any shape. And the corresponding azimuths
in the "az" argument and elevations in "el". These 3 arrays describe
corresponding points. They may have identical dimensions. If there's any
redundancy (if every row of the disparity array has identical azimuths for
instance), then being broadcasting-compatible is sufficient. A common case of
this is disparity images. In this case the disparity array has shape (Nel,Naz),
with each row containing an identical sequences of azimuths and each column
containing identical sequences of elevations. We can then pass an azimuth array
of shape (Naz,) and en elevation array of shape (Nel,1)
The azimuths, elevations may also be passed in the "az_row", "el_col" arguments.
This is usually not done by the user directly, but via the **cookie from
stereo_rectify_prepare(). At least one of "az","az_row" and "el","el_col" must
be given. If both are given, we use "az" and "el".
ARGUMENTS
- az: optional array of azimuths corresponding to the given disparities. If
processing normal disparity images, this can be omitted, and we'll use az_row
from **cookie, with the cookie returned by stereo_rectify_prepare(). If
processing anything other than a full disparity image, pass the azimuths here.
The az array must be broadcasting-compatible with disparity_pixels. The
dimensions of the two arrays may be identical, but brodcasting can be used to
exploit any redundancy. For instance, disparity images have shape (Nel, Naz)
with each row containing identical azimuths. We can thus work with az of shape
(Naz,). At least one of az and az_row must be non-None. If both are given, we
use az.
- el: optional array of elevations corresponding to the given disparities.
Works identically to the "az" argument, but represents elevations.
- disparity_pixels: optional array of disparities, defaulting to None. If None,
we report unit vectors. Otherwise we use the disparities to compute ranges to
scale the vectors. This array contains disparity in PIXELS. If the
stereo-matching algorithm you're using reports a scaled value, you need to
convert it to pixels, possibly converting to floating-point in the process.
Any array shape is supported. In the common case of a disparity IMAGE, this is
an array of shape (Nel, Naz)
- baseline: optional value, representing the distance between the two stereo
cameras. This is required only if disparity_pixels is not None. Usually it
comes from **cookie, with the cookie returned by stereo_rectify_prepare()
- pixels_per_deg_az: optional value, representing the angular resolution of the
disparity image. This is required only if disparity_pixels is not None.
Usually it comes from **cookie, with the cookie returned by
stereo_rectify_prepare(). Don't pass this yourself. This argument may go away
in the future
- get_gradients: optional boolean, defaulting to False. If True, we return the
unprojected vectors AND their gradients in respect to (az,el). Gradients are
only available when performing rangeless unprojections (disparity_pixels is
None).
- az_row: optional array of azimuths in the disparity_pixels array. Works
exactly like the "az" array. At least one MUST be given. If both are given, we
use "az". Usually az_row is not given by the user directly, but via a **cookie
from stereo_rectify_prepare()
- el_col: optional array of elevations in the disparity_pixels array. Works just
like az_row, but for elevations
RETURNED VALUES
if get_gradients is False:
- An array of unprojected points of shape (....,3). Invalid or missing ranges
are represented as vectors of length 0.
else:
A tuple:
- The array of unprojected points as before.
- The array of gradients dv_dazel of shape (...,3,2)
- supported_lensmodels(...)
- Returns a tuple of strings for the various lens models we support
SYNOPSIS
print(mrcal.supported_lensmodels())
('LENSMODEL_PINHOLE',
'LENSMODEL_STEREOGRAPHIC',
'LENSMODEL_SPLINED_STEREOGRAPHIC_...',
'LENSMODEL_OPENCV4',
'LENSMODEL_OPENCV5',
'LENSMODEL_OPENCV8',
'LENSMODEL_OPENCV12',
'LENSMODEL_CAHVOR',
'LENSMODEL_CAHVORE')
mrcal knows about some set of lens models, which can be queried here. The above
list is correct as of this writing, but more models could be added with time.
The returned lens models are all supported, with possible gaps. The only missing
functionality at this time is that CAHVORE is gradients aren't implemented, so
we can't compute an optimal CAHVORE model.
Models ending in '...' have configuration parameters given in the model string,
replacing the '...'.
RETURNED VALUE
A tuple of strings listing out all the currently-supported lens models
- synthesize_board_observations(models, object_width_n, object_height_n, object_spacing, calobject_warp, rt_ref_boardcenter, rt_ref_boardcenter__noiseradius, Nframes, which='all_cameras_must_see_full_board')
- Produce synthetic chessboard observations
SYNOPSIS
models = [mrcal.cameramodel("0.cameramodel"),
mrcal.cameramodel("1.cameramodel"),]
# shapes (Nframes, Ncameras, object_height_n, object_width_n, 2) and
# (Nframes, 4, 3)
q,Rt_cam0_boardref = \
mrcal.synthesize_board_observations( \
models,
# board geometry
10,12,0.1,None,
# mean board pose and the radius of the added uniform noise
rt_ref_boardcenter,
rt_ref_boardcenter__noiseradius,
# How many frames we want
100,
which = 'some_cameras_must_see_half_board')
# q now contains the synthetic pixel observations, but some of them will be
# out of view. I construct an (x,y,weight) observations array, as expected
# by the optimizer, and I set the weight for the out-of-view points to -1 to
# tell the optimizer to ignore those points
# Set the weights to 1 initially
# shape (Nframes, Ncameras, object_height_n, object_width_n, 3)
observations = nps.glue(q,
np.ones( q.shape[:-1] + (1,) ),
axis = -1)
# shape (Ncameras, 1, 1, 2)
imagersizes = nps.mv( nps.cat(*[ m.imagersize() for m in models ]),
-2, -4 )
observations[ np.any( q < 0, axis=-1 ), 2 ] = -1.
observations[ np.any( q-imagersizes >= 0, axis=-1 ), 2 ] = -1.
Given a description of a calibration object and of the cameras observing it,
produces perfect pixel observations of the objects by those cameras. We return a
dense observation array: every corner observation from every chessboard pose
will be reported for every camera. Some of these observations MAY be
out-of-view, depending on the value of the 'which' argument; see description
below. The example above demonstrates how to mark such out-of-bounds
observations as outliers to tell the optimization to ignore these.
The "models" provides the intrinsics and extrinsics.
The calibration objects are nominally have pose rt_ref_boardcenter in the
reference coordinate system, with each pose perturbed uniformly with radius
rt_ref_boardcenter__noiseradius. This is slightly nonstandard since here I'm
placing the board origin at its center instead of the corner (as
mrcal.ref_calibration_object() does). But this is more appropriate to the usage
of this function. The returned Rt_cam0_boardref transformation DOES use the
normal corner-referenced board geometry
Returns the point observations and the chessboard poses that produced these
observations.
ARGUMENTS
- models: an array of mrcal.cameramodel objects, one for each camera we're
simulating. This is the intrinsics and the extrinsics. Ncameras = len(models)
- object_width_n: the number of horizontal points in the calibration object grid
- object_height_n: the number of vertical points in the calibration object grid
- object_spacing: the distance between adjacent points in the calibration
object. A square object is assumed, so the vertical and horizontal distances
are assumed to be identical.
- calobject_warp: a description of the calibration board warping. None means "no
warping": the object is flat. Otherwise this is an array of shape (2,). See
the docs for ref_calibration_object() for the meaning of the values in this
array.
- rt_ref_boardcenter: the nominal pose of the calibration object, in the
reference coordinate system. This is an rt transformation from a
center-referenced calibration object to the reference coordinate system
- rt_ref_boardcenter__noiseradius: the deviation-from-nominal for the chessboard
pose for each frame. I add uniform noise to rt_ref_boardcenter, with each
element sampled independently, with the radius given here.
- Nframes: how many frames of observations to return
- which: a string, defaulting to 'all_cameras_must_see_full_board'. Controls the
requirements on the visibility of the returned points. Valid values:
- 'all_cameras_must_see_full_board': We return only those chessboard poses
that produce observations that are FULLY visible by ALL the cameras.
- 'some_cameras_must_see_full_board': We return only those chessboard poses
that produce observations that are FULLY visible by AT LEAST ONE camera.
- 'all_cameras_must_see_half_board': We return only those chessboard poses
that produce observations that are AT LEAST HALF visible by ALL the cameras.
- 'some_cameras_must_see_half_board': We return only those chessboard poses
that produce observations that are AT LEAST HALF visible by AT LEAST ONE
camera.
RETURNED VALUES
We return a tuple:
- q: an array of shape (Nframes, Ncameras, object_height, object_width, 2)
containing the pixel coordinates of the generated observations
- Rt_cam0_boardref: an array of shape (Nframes, 4,3) containing the poses of the
chessboards. This transforms the object returned by ref_calibration_object()
to the pose that was projected
- transform_image(image, mapxy, out=None, borderMode=5, interpolation=1)
- Transforms a given image using a given map
SYNOPSIS
model_orig = mrcal.cameramodel("xxx.cameramodel")
image_orig = cv2.imread("image.jpg")
model_pinhole = mrcal.pinhole_model_for_reprojection(model_orig,
fit = "corners")
mapxy = mrcal.image_transformation_map(model_orig, model_pinhole)
image_undistorted = mrcal.transform_image(image_orig, mapxy)
# image_undistorted is now a pinhole-reprojected version of image_orig
Given an array of pixel mappings this function can be used to transform one
image to another. If we want to convert a scene image observed by one camera
model to the image of the same scene using a different model, we can produce a
suitable transformation map with mrcal.image_transformation_map(). An example of
this common usage appears above in the synopsis.
At this time this function is a thin wrapper around cv2.remap()
ARGUMENTS
- image: a numpy array containing an image we're transforming. May be grayscale:
shape (Nheight_input, Nwidth_input) or RGB: shape (Nheight_input,
Nwidth_input, 3)
- mapxy: a numpy array of shape (Nheight,Nwidth,2) where Nheight and Nwidth
represent the dimensions of the target image. This array is expected to have
dtype=np.float32, since the internals of this function are provided by
cv2.remap()
- out: optional numpy array of shape (Nheight,Nwidth) or (Nheight,Nwidth,3) to
receive the result. If omitted, a new array is allocated and returned.
- borderMode: optional constant defining out-of-bounds behavior. Defaults to
cv2.BORDER_TRANSPARENT, and is passed directly to cv2.remap(). Please see the
docs for that function for details. This option may disappear if mrcal stops
relying on opencv
- interpolation: optional constant defining pixel interpolation behavior.
Defaults to cv2.INTER_LINEAR, and is passed directly to cv2.remap(). Please
see the docs for that function for details. This option may disappear if mrcal
stops relying on opencv
RETURNED VALUE
A numpy array of shape (Nheight, Nwidth) if grayscale or (Nheight, Nwidth, 3) if
RGB. Contains the transformed image.
- transform_point_Rt(Rt, x, get_gradients=False, out=None)
- Transform point(s) using an Rt transformation
SYNOPSIS
Rt = nps.glue(rotation_matrix,translation, axis=-2)
print(Rt.shape)
===>
(4,3)
print(x.shape)
===>
(10,3)
print( mrcal.transform_point_Rt(Rt, x).shape )
===>
(10,3)
print( [arr.shape
for arr in mrcal.transform_point_Rt(Rt, x,
get_gradients = True)] )
===>
[(10,3), (10,3,4,3), (10,3,3)]
Transform point(s) by an Rt transformation: a (4,3) array formed by
nps.glue(R,t, axis=-2) where R is a (3,3) rotation matrix and t is a (3,)
translation vector. This transformation is defined by a matrix multiplication
and an addition. x and t are stored as a row vector (that's how numpy stores
1-dimensional arrays), but the multiplication works as if x was a column vector
(to match linear algebra conventions):
transform_point_Rt(Rt, x) = transpose( matmult(Rt[:3,:], transpose(x)) +
transpose(Rt[3,:]) ) =
= matmult(x, transpose(Rt[:3,:])) +
Rt[3,:]
By default this function returns the transformed points only. If we also want
gradients, pass get_gradients=True. Logic:
if not get_gradients: return u=Rt(x)
else: return (u=Rt(x),du/dRt,du/dx)
This function supports broadcasting fully, so we can transform lots of points at
the same time and/or apply lots of different transformations at the same time
ARGUMENTS
- Rt: array of shape (4,3). This matrix defines the transformation. Rt[:3,:] is
a rotation matrix; Rt[3,:] is a translation. It is assumed that the rotation
matrix is a valid rotation (matmult(R,transpose(R)) = I, det(R) = 1), but that
is not checked
- x: array of shape (3,). The point being transformed
- get_gradients: optional boolean. By default (get_gradients=False) we return an
array of transformed points. Otherwise we return a tuple of arrays of
transformed points and their gradients.
- out: optional argument specifying the destination. By default, new numpy
array(s) are created and returned. To write the results into existing (and
possibly non-contiguous) arrays, specify them with the 'out' kwarg. If
get_gradients: 'out' is the one numpy array we will write into. Else: 'out' is
a tuple of all the output numpy arrays. If 'out' is given, we return the same
arrays passed in. This is the standard behavior provided by numpysane_pywrap.
RETURNED VALUE
If not get_gradients: we return an array of transformed point(s). Each
broadcasted slice has shape (3,)
If get_gradients: we return a tuple of arrays of transformed points and the
gradients (u=Rt(x),du/dRt,du/dx):
1. The transformed point(s). Each broadcasted slice has shape (3,)
2. The gradient du/dRt. Each broadcasted slice has shape (3,4,3). The first
dimension selects the element of u, and the last 2 dimensions select the
element of Rt
3. The gradient du/dx. Each broadcasted slice has shape (3,3). The first
dimension selects the element of u, and the last dimension selects the
element of x
- transform_point_rt(rt, x, get_gradients=False, out=None, inverted=False)
- Transform point(s) using an rt transformation
SYNOPSIS
r = rotation_axis * rotation_magnitude
rt = nps.glue(r,t, axis=-1)
print(rt.shape)
===>
(6,)
print(x.shape)
===>
(10,3)
print( mrcal.transform_point_rt(rt, x).shape )
===>
(10,3)
print( [arr.shape
for arr in mrcal.transform_point_rt(rt, x,
get_gradients = True)] )
===>
[(10,3), (10,3,6), (10,3,3)]
Transform point(s) by an rt transformation: a (6,) array formed by
nps.glue(r,t, axis=-1) where r is a (3,) Rodrigues vector and t is a (3,)
translation vector. This transformation is defined by a matrix multiplication
and an addition. x and t are stored as a row vector (that's how numpy stores
1-dimensional arrays), but the multiplication works as if x was a column vector
(to match linear algebra conventions):
transform_point_rt(rt, x) = transpose( matmult(R_from_r(rt[:3]), transpose(x)) +
transpose(rt[3,:]) ) =
= matmult(x, transpose(R_from_r(rt[:3]))) +
rt[3:]
By default this function returns the transformed points only. If we also want
gradients, pass get_gradients=True. Logic:
if not get_gradients: return u=rt(x)
else: return (u=rt(x),du/drt,du/dx)
This function supports broadcasting fully, so we can transform lots of points at
the same time and/or apply lots of different transformations at the same time
ARGUMENTS
- rt: array of shape (6,). This vector defines the transformation. rt[:3] is a
rotation defined as a Rodrigues vector; rt[3:] is a translation.
- x: array of shape (3,). The point being transformed
- get_gradients: optional boolean. By default (get_gradients=False) we return an
array of transformed points. Otherwise we return a tuple of arrays of
transformed points and their gradients.
- inverted: optional boolean, defaulting to False. If True, the opposite
transformation is computed. The gradient du/drt is returned in respect to the
input rt
- out: optional argument specifying the destination. By default, new numpy
array(s) are created and returned. To write the results into existing (and
possibly non-contiguous) arrays, specify them with the 'out' kwarg. If
get_gradients: 'out' is the one numpy array we will write into. Else: 'out' is
a tuple of all the output numpy arrays. If 'out' is given, we return the same
arrays passed in. This is the standard behavior provided by numpysane_pywrap.
RETURNED VALUE
If not get_gradients: we return an array of transformed point(s). Each
broadcasted slice has shape (3,)
If get_gradients: we return a tuple of arrays of transformed points and the
gradients (u=rt(x),du/drt,du/dx):
1. The transformed point(s). Each broadcasted slice has shape (3,)
2. The gradient du/drt. Each broadcasted slice has shape (3,6). The first
dimension selects the element of u, and the last dimension selects the
element of rt
3. The gradient du/dx. Each broadcasted slice has shape (3,3). The first
dimension selects the element of u, and the last dimension selects the
element of x
- unpack_state(...)
- Scales a state vector from the packed, unitless form used by the optimizer
SYNOPSIS
m = mrcal.cameramodel('xxx.cameramodel')
optimization_inputs = m.optimization_inputs()
p_packed = mrcal.optimizer_callback(**optimization_inputs)[0]
p = p_packed.copy()
mrcal.unpack_state(p, **optimization_inputs)
In order to make the optimization well-behaved, we scale all the variables in
the state and the gradients before passing them to the optimizer. The internal
optimization library thus works only with unitless (or "packed") data.
This function takes a packed numpy array of shape (...., Nstate), and scales it
to produce full data with real units. This function applies the scaling directly
to the input array; the input is modified, and nothing is returned.
To unpack a state vector, you naturally call unpack_state(). To unpack a
jacobian matrix, you would call pack_state() because in a jacobian, the state is
in the denominator.
Broadcasting is supported: any leading dimensions will be processed correctly,
as long as the given array has shape (..., Nstate).
In order to know what the scale factors should be, and how they should map to
each variable in the state vector, we need quite a bit of context. If we have
the full set of inputs to the optimization function, we can pass in those (as
shown in the example above). Or we can pass the individual arguments that are
needed (see ARGUMENTS section for the full list). If the optimization inputs and
explicitly-given arguments conflict about the size of some array, the explicit
arguments take precedence. If any array size is not specified, it is assumed to
be 0. Thus most arguments are optional.
ARGUMENTS
- p: a numpy array of shape (..., Nstate). This is the packed state on input,
and the full state on output. The input array is modified.
- **kwargs: if the optimization inputs are available, they can be passed-in as
kwargs. These inputs contain everything this function needs to operate. If we
don't have these, then the rest of the variables will need to be given
- lensmodel: string specifying the lensmodel we're using (this is always
'LENSMODEL_...'). The full list of valid models is returned by
mrcal.supported_lensmodels(). This is required if we're not passing in the
optimization inputs
- do_optimize_intrinsics_core
do_optimize_intrinsics_distortions
do_optimize_extrinsics
do_optimize_calobject_warp
do_optimize_frames
optional booleans; default to True. These specify what we're optimizing. See
the documentation for mrcal.optimize() for details
- Ncameras_intrinsics
Ncameras_extrinsics
Nframes
Npoints
Npoints_fixed
optional integers; default to 0. These specify the sizes of various arrays in
the optimization. See the documentation for mrcal.optimize() for details
RETURNED VALUE
None. The scaling is applied to the input array
- unproject(q, lensmodel, intrinsics_data, normalize=False, out=None)
- Unprojects pixel coordinates to observation vectors
SYNOPSIS
v = mrcal.unproject( # (...,2) array of pixel observations
q,
lensmodel, intrinsics_data )
Maps a set of 2D imager points q to a 3d vector in camera coordinates that
produced these pixel observations. The 3d vector is unique only up-to-length,
and the returned vectors aren't normalized by default. If we want them to be
normalized, pass normalize=True.
This is the "reverse" direction, so an iterative nonlinear optimization is
performed internally to compute this result. This is much slower than
mrcal_project. For OpenCV distortions specifically, OpenCV has
cvUndistortPoints() (and cv2.undistortPoints()), but these are inaccurate:
https://github.com/opencv/opencv/issues/8811
Broadcasting is fully supported across q and intrinsics_data.
Models that have no gradients available cannot use mrcal_unproject() in C, but
CAN still use this mrcal.unproject() Python routine: a slower routine is
employed that uses numerical differences instead of analytical gradients.
ARGUMENTS
- q: array of dims (...,2); the pixel coordinates we're unprojecting
- lensmodel: a string such as
LENSMODEL_PINHOLE
LENSMODEL_OPENCV4
LENSMODEL_CAHVOR
LENSMODEL_SPLINED_STEREOGRAPHIC_order=3_Nx=16_Ny=12_fov_x_deg=100
- intrinsics_data: array of dims (Nintrinsics):
(focal_x, focal_y, center_pixel_x, center_pixel_y, distortion0, distortion1,
...)
The focal lengths are given in pixels.
- normalize: optional boolean defaults to False. If True: normalize the output
vectors
- out: optional argument specifying the destination. By default, new numpy
array(s) are created and returned. To write the results into existing arrays,
specify them with the 'out' kwarg. If get_gradients: 'out' is the one numpy
array we will write into. Else: 'out' is a tuple of all the output numpy
arrays. If 'out' is given, we return the same arrays passed in. This is the
standard behavior provided by numpysane_pywrap.
RETURNED VALUE
The unprojected observation vector of shape (..., 3). These are NOT normalized
by default. To get normalized vectors, pass normalize=True
- unproject_stereographic(...)
- Unprojects a set of 2D pixel coordinates using a stereographic map
SYNOPSIS
v = mrcal.unproject_stereographic( # (N,2) array of 2d imager points
points,
fx, fy, cx, cy )
# v is now a (N,3) array of observation directions. v are not normalized
This is a special case of mrcal.unproject(). Useful as part of data analysis,
not to represent any real-world lens.
Given a (N,2) array of pixel coordinates and parameters of a perfect
stereographic camera, this function computes the inverse projection, optionally
with gradients. No actual lens ever follows this model exactly, but this is
useful as a baseline for other models.
The user can pass in focal length and center-pixel values. Or they can be
omitted to compute a "normalized" stereographic projection (fx = fy = 1, cx = cy
= 0).
The stereographic projection is able to represent points behind the camera, and
has only one singular observation direction: directly behind the camera, along
the optical axis.
This projection acts radially. If the observation vector v makes an angle theta
with the optical axis, then the projected point q is 2 tan(theta/2) f from the
image center.
ARGUMENTS
- points: array of dims (...,2); the pixel coordinates we're projecting. This
supports broadcasting fully, and any leading dimensions are allowed, including
none
- fx, fy: optional focal-lengths, in pixels. Both default to 1, as in the
normalized stereographic projection
- cx, cy: optional projection center, in pixels. Both default to 0, as in the
normalized stereographic projection
- get_gradients: optional boolean, defaults to False. This affects what we
return (see below)
RETURNED VALUE
if not get_gradients: we return an (...,3) array of unprojected observation
vectors
if get_gradients: we return a tuple:
- (...,3) array of unprojected observation vectors
- (...,3,2) array of the gradients of the observation vectors in respect to
the input 2D pixel coordinates
- worst_direction_stdev(cov)
- Compute the worst-direction standard deviation from a 2x2 covariance matrix
SYNOPSIS
# A covariance matrix
print(cov)
===>
[[ 1. -0.4]
[-0.4 0.5]]
# Sample 1000 0-mean points using this covariance
x = np.random.multivariate_normal(mean = np.array((0,0)),
cov = cov,
size = (1000,))
# Compute the worst-direction standard deviation of the sampled data
print(np.sqrt(np.max(np.linalg.eig(np.mean(nps.outer(x,x),axis=0))[0])))
===>
1.1102510878087053
# The predicted worst-direction standard deviation
print(mrcal.worst_direction_stdev(cov))
===> 1.105304960905736
The covariance of a (2,) random variable can be described by a (2,2)
positive-definite symmetric matrix. The 1-sigma contour of this random variable
is described by an ellipse with its axes aligned with the eigenvectors of the
covariance, and the semi-major and semi-minor axis lengths specified as the sqrt
of the corresponding eigenvalues. This function returns the worst-case standard
deviation of the given covariance: the sqrt of the larger of the two
eigenvalues.
This function supports broadcasting fully.
DERIVATION
Let cov = (a b). If l is an eigenvalue of the covariance then
(b c)
(a-l)*(c-l) - b^2 = 0 --> l^2 - (a+c) l + ac-b^2 = 0
--> l = (a+c +- sqrt( a^2 + 2ac + c^2 - 4ac + 4b^2)) / 2 =
= (a+c +- sqrt( a^2 - 2ac + c^2 + 4b^2)) / 2 =
= (a+c)/2 +- sqrt( (a-c)^2/4 + b^2)
So the worst-direction standard deviation is
sqrt((a+c)/2 + sqrt( (a-c)^2/4 + b^2))
ARGUMENTS
- cov: the covariance matrices given as a (..., 2,2) array. Valid covariances
are positive-semi-definite (symmetric with eigenvalues >= 0), but this is not
checked
RETURNED VALUES
The worst-direction standard deviation. This is a scalar or an array, if we're
broadcasting
|