Mobile Robotics

Overview

Mobile Robotics covers the geometry and estimation a robot needs to build a map and localise itself in it. The course runs from rigid body transforms through LIDAR mapping, point cloud registration, pose graph optimization, camera modelling, two view geometry, bundle adjustment, and stereo. Assignments and the project are code, not derivation write-ups: the derivations happen in class so that the code you write afterwards is something you can debug.

Prerequisite: linear algebra and probability, plus enough Python to handle numpy and a point cloud library. Some math revision is scheduled mid-course (least squares, SVD, Levenberg-Marquardt).

Instructor

Teaching assistants

Tarun Ramakrishnan, Krish Pandya, Akash Kumbar.

Logistics

Lectures: Mondays and Thursdays, 2:05 PM to 2:30 PM.

Slides, class notes and some resources will be on Moodle.

Schedule

Topics per lecture are tentative. Rows might shift when a derivation takes longer than the slot allows.

Grading

Project weight splits as follows.

Important dates

Project

Teams of two to three. The project runs the whole semester and is submitted in three parts on Moodle: a timeline document, a mid submission, and a final submission with a demo.

The timeline document fixes what you are building, the dataset or robot you are building it on, and who does what by when. Pick a problem where you can show a metric, not a video alone: trajectory error against ground truth, map consistency after loop closure, reprojection error before and after bundle adjustment, disparity error on a stereo benchmark.

Bring your own problem if you have one. A topic you found yourself, in a lab, in a paper you liked, or on a robot you already work with, is preferred over anything on the list below. The list exists so that nobody is stuck without a starting point.

Aim the final submission at a venue. A semester is enough to get a workshop paper or a short paper out of a good project, and past projects have grown into ICRA, IROS, RA-L and CVPR or ICCV workshop submissions. If that is the target, say so in the timeline document and pick the evaluation protocol the venue expects (a public benchmark, a named metric, ablations against a baseline) from day one rather than adding it in November. The TAs will read drafts.

Starting points:

• LIDAR odometry and mapping on a public sequence, with loop closure and pose graph optimization.
• Occupancy mapping with dynamic obstacle handling, evaluated against a labelled sequence.
• LIDAR to camera calibration, including an error analysis of how the estimate degrades with fewer correspondences.
• Monocular visual odometry with scale recovery from a known constraint.
• Structure from motion on a small image set, ending in bundle adjustment.
• Dense stereo depth, comparing block matching against a learned baseline.
• Moving object detection from epipolar geometry in a driving sequence.

Anything else is fine if it uses the geometry taught here. Clear it with a TA before the timeline document is due.

Policies

Late submissions: each student has five late days for the semester, usable on assignments in whole-day units. Once they run out, late work loses 20% per day and is not accepted after five days. Project deadlines do not take late days.

Collaboration: discuss ideas with anyone. Write your own code and your own text. Copying code from a classmate, a previous year’s submission, or an unattributed online source counts as plagiarism. Using a library is fine and using a model to write code is fine, provided you cite it in the README and you can explain every line in a viva.

Grade queries: raise them within one week of the grade being published, to the TA who graded the component.

References

Books

• J. J. Craig, Introduction to Robotics: Mechanics and Control. Frames, rotations and homogeneous transforms, ch. 2.
• R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Camera models, epipolar geometry, triangulation, bundle adjustment. The sample chapters on the book page cover most of what the two view unit needs.
• S. Thrun, W. Burgard and D. Fox, Probabilistic Robotics. Inverse sensor models, occupancy grid mapping, SLAM.
• R. Szeliski, Computer Vision: Algorithms and Applications. Image formation, stereo, structure from motion. Free PDF on the book page.
• X. Gao, T. Zhang, Y. Liu and Q. Yan, 14 Lectures on Visual SLAM (slambook-en). Written for people implementing SLAM: transforms, camera models, feature matching, backend optimization with Ceres and g2o, loop closure. Code is in the repository.

Lectures

• Cyrill Stachniss, lecture channel and mobile robotics online training. Occupancy mapping, ICP, pose graph optimization, calibration and bundle adjustment, in notation close to what is used in class.
• Cyrill Stachniss, Photogrammetry I and II, University of Bonn. Camera modelling, DLT, calibration, bundle adjustment.
• Kris Kitani and others, 16-385 Computer Vision, CMU. Slide decks for two view geometry, rectification and stereo matching.
• NPTEL slide decks on camera modelling and ICP, on Moodle.

Papers

• J. Sola, Quaternion kinematics for the error-state Kalman filter, 2017. The reference for quaternion conventions, composition and the exponential map.
• K. Madsen, H. B. Nielsen and O. Tingleff, Methods for Non-Linear Least Squares Problems, DTU lecture notes, 2004. Gauss-Newton and Levenberg-Marquardt, with the damping parameter update rules.
• S. Rusinkiewicz and M. Levoy, Efficient Variants of the ICP Algorithm, 3DIM 2001. Sampling, matching, weighting and rejection strategies, compared on convergence speed.
• G. Grisetti, R. Kummerle, C. Stachniss and W. Burgard, A Tutorial on Graph-Based SLAM, IEEE ITS Magazine, 2010. Pose graph construction and least squares solution.
• Z. Zhang, A Flexible New Technique for Camera Calibration, TPAMI 2000. The planar target method behind the OpenCV calibration pipeline.
• R. Hartley, In Defence of the Eight-Point Algorithm, TPAMI 1997. Why normalization decides whether the eight point algorithm works.
• B. Triggs, P. McLauchlan, R. Hartley and A. Fitzgibbon, Bundle Adjustment: A Modern Synthesis, 2000. Cost functions, sparsity, gauge freedom, numerical conditioning.

Learned reconstruction

• P. Weinzaepfel et al., CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completion, NeurIPS 2022. Where the line starts: mask one of two views of a scene and reconstruct it from the other, which forces the network to learn correspondence and relative pose. CroCo v2 (ICCV 2023) and the code are in the same repository, and the DUSt3R encoder is initialized from this pre-training.
• S. Wang et al., DUSt3R: Geometric 3D Vision Made Easy, CVPR 2024. Pointmap regression from an image pair, without known intrinsics.
• V. Leroy, Y. Cabon and J. Revaud, Grounding Image Matching in 3D with MASt3R, ECCV 2024. Dense matching on top of DUSt3R, with code.
• J. Wang et al., VGGT: Visual Geometry Grounded Transformer, CVPR 2025. Cameras, depth and point tracks from one forward pass, with code.
• J. Wang et al., VGGT-Omega, CVPR 2026. Scaled up training for static and dynamic scenes, project page.
• Efficiently Reconstructing Dynamic Scenes One D4RT at a Time, CVPR 2026. Video encoded once into a latent scene, then queried for depth, tracks and camera parameters, project page.

Software

• Ceres Solver for nonlinear least squares, pose graphs and bundle adjustment.
• g2o as an alternative graph optimization backend.
• Open3D for point cloud IO, registration and visualization.
• OpenCV for calibration, feature matching and stereo.
• OctoMap for octree occupancy maps.

Datasets

• KITTI for LIDAR, stereo and odometry ground truth. The odometry benchmark is the usual target for project evaluation.