MVG Chapter 18: N-View Computational Methods, Check that for deeper analysis.

If Kalman Filters are the "quick and dirty" recursive solution, Bundle Adjustment (BA) is the "slow and perfect" batch solution.
It is the de facto standard for offline SLAM and 3D reconstruction.

Definition: Simultaneous refinement of the 3D coordinates of scene geometry ($X_j$) and the parameters of the relative motion ($R_i,t_i$) and optical characteristics ($K$) to minimize the reprojection error.

1. The Visual Objective Function

Unlike ICP-SLAM (which minimizes 3D-3D distance), Visual BA minimizes 2D-3D Reprojection Error.

• ${\mathbf{z}}_{ij}$: Observed pixel coordinates $(u,v)$ of point $j$ in frame $i$.
• $\pi (\cdot )$: The Camera Projection function (World $\to$ Camera $\to$ Image).
• $T_i$: Camera Pose ($R,t$).
• $X_j$: 3D Landmark position.

2. The Projection Model $\pi$

This is where it differs from ICP. We have to project 3D points to the image plane.

1. Transform: $P^{\prime }=R{X}_w+t=[X^{\prime },Y^{\prime },Z^{\prime }{]}^T$
2. Normalize: $p_n=[X^{\prime }/Z^{\prime },Y^{\prime }/Z^{\prime }{]}^T$ (Perspective Division)
3. Pixelate: $\left[\begin{array}{c}u\\ v\end{array}\right]=\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\end{array}\right]\left[\begin{array}{c}{X}^{\prime }/Z^{\prime }\\ Y^{\prime }/Z^{\prime }\\ 1\end{array}\right]$

3. The Jacobian (The Chain Rule)

The residual is $\mathbf{r}=\mathbf{z}-\pi (T,X)$. We need $\frac{\partial \mathbf{r}}{\partial \delta \xi }$.
Using the Chain Rule:

Part A: The Projection Derivative ($\partial \mathbf{r}/\partial P^{\prime }$)**
How pixel $u,v$ changes when the camera-frame point $X^{\prime },Y^{\prime },Z^{\prime }$ moves.

Note the $1/Z^2$ term. This is why points close to the camera cause massive gradients (instability).

Part B: The Manifold Derivative ($\partial P^{\prime }/\partial \delta \xi$)
Same as ICP-SLAM.

Multiply A and B to get the $2\times 6$ Jacobian for one observation.

4. The Schur Complement (The Computational Trick)

We have a system $H\delta =b$.
$H$ is size $(6m+3n{)}^2$. For 1000 points and 100 cameras, $H$ is $3600\times 3600$. Inverting this is $O(N^3)$. Too slow.

The Structure:

• $B$: Camera-Camera block (Sparse).
• $C$: Point-Point block (Diagonal! Because points don't see each other).
• $E$: Camera-Point interaction.

Since $C$ is diagonal, it is trivial to invert. We can "marginalize out" the points to solve for cameras first:

1. Solve for Cameras (Reduced Camera System):

The matrix $S=B-E{C}^{-1}{E}^T$ is the Schur Complement. It is much smaller ($6m\times 6m$).

2. Back-Substitute for Points:

This reduces complexity from cubic in points to linear in points. This is why we can run BA on thousands of landmarks.