Key takeaways
- Hand-eye calibration finds the 3D transform between a camera frame and a robot end-effector or base frame.
- Eye-in-hand and eye-to-hand arrangements pursue the same goal but solve for different frame relationships.
- Paired robot and target poses produce relative motions whose common unknown transform satisfies AX=XB.
- Rotations about different axes are essential; a larger, well-spread pose set is usually better than the bare minimum.
- A credible result needs held-out poses, transform-closure checks and physical task tests, not only one reprojection number.
A camera measurement is not yet a robot command
Hand-eye calibration estimates the fixed position and orientation between a camera coordinate frame and a robot coordinate frame. A camera reports an object relative to its lens. A robot controller plans motion relative to its base or tool. Without a transform connecting those frames, an object measured ten centimeters to the camera’s right does not directly tell the robot which joints should move or where its gripper should go.
The hand is the gripper or tool frame, while the eye is the camera frame. The result is commonly represented by a 3D rotation matrix and translation vector. Once estimated, it converts points and poses from camera coordinates into end-effector or base coordinates. It remains constant only when the camera is rigidly mounted and the robot’s kinematic description matches the real mechanism.
Eye-in-hand and eye-to-hand solve different frame links
In an eye-in-hand arrangement, the camera is attached to the wrist and moves with the arm. Each robot pose gives the camera a different view of a stationary calibration target. The central unknown is the transform between camera and end effector. This arrangement can observe close work from convenient angles, but cable forces, wrist vibration and payload changes can disturb the supposedly rigid mount.
In an eye-to-hand arrangement, the camera is fixed near the cell while the robot or a target held by the robot moves. The transform of interest is usually between the camera and robot base. It can cover a large work area, but the arm or workpiece may occlude the target. Mixing the two arrangements or inverting a transform incorrectly can produce plausible numbers that send the physical command in the wrong direction.
Pose pairs turn the problem into AX=XB
Each sample pairs the end-effector pose measured in the robot base frame with the target pose estimated in the camera frame. Comparing two samples produces a relative robot motion A and a relative apparent target motion B. If X is the fixed camera-to-hand transform, ideal measurements obey AX equals XB. The stationary target pose disappears from this relative-motion equation, which is why several viewpoints can reveal the unknown link.
The solver estimates one rotation and translation that make the equation hold across all pose pairs. OpenCV’s calibrateHandEye implementation exposes Tsai, Park, Horaud, Andreff and Daniilidis methods. Some estimate rotation before translation, while others solve both simultaneously. Choosing a famous solver does not compensate for a poorly excited dataset or inaccurate pose measurements.
Motion diversity matters more than merely counting poses
OpenCV documents a minimum of two motions with nonparallel rotation axes, which requires at least three different poses, and strongly recommends using many more. The MoveIt tutorial begins solving after five samples and says its example commonly plateaus after roughly twelve to fifteen samples. Those counts are practical guidance from that workflow, not a universal accuracy guarantee for every lens, arm and working volume.
The arm should not repeat nearly identical motions around one axis. Tilt the wrist in different directions, vary distance and place the target across useful parts of the image. Small motions or almost parallel rotation axes make the equation weakly constrained, so several transforms can fit nearly equally well. Extremely oblique or blurred views create a different problem by making target detection noisy.
The inputs must be calibrated before solving
Camera intrinsics need to be calibrated first. Incorrect focal length, principal point or distortion changes the target pose estimated from each image. The MoveIt workflow likewise assumes good intrinsic parameters and an accurate camera frame. The target must be flat, its dimensions and marker spacing must be known, and it must remain stationary in the selected base frame throughout data collection.
Robot-side measurements also matter. Encoder bias, imperfect link dimensions and an incorrect tool center point contaminate every end-effector pose. If camera and robot timestamps are misaligned, a moving capture pairs observations from different moments. A careful workflow stops the arm, waits for vibration to settle, records frame names and timestamps, and separately measures synchronization error when samples are collected continuously.
Validate on new poses and the real task
After solving, do not validate only on the samples used for estimation. Predict the target pose at held-out robot configurations and compare it with the new observation. Compute whether alternative transform chains from robot base to target close at the same position, and inspect rotational and translational residuals separately. A small pixel reprojection error can coexist with a larger depth-direction error.
The final test should use the intended physical task. For example, detect the center of a cube and command the gripper to approach cubes placed at multiple positions. Record position and orientation error near the camera, farther away and toward the image edges. Report the worst error and the distribution across poses as well as the mean, because an average can hide a dangerous local failure.
Recheck the transform whenever the mounting changes
A hand-eye transform is not permanent. Retightening a bracket, bumping the camera, changing the tool or refocusing a lens can change the relationship. Cable load, temperature and structural compliance may cause the camera to move slightly with posture even when nothing looks loose. In that case, one rigid transform cannot perfectly explain the whole workspace.
A practical cell can periodically observe a reference object and track error over time. When drift crosses the task tolerance, diagnose camera intrinsics, robot kinematics, timing and hand-eye calibration separately. Hand-eye calibration is the coordinate bridge that lets vision guide motion, but it works reliably only when the measurements on both sides of that bridge are equally trustworthy.
Sources reviewed
- OpenCV 4.13.0: Camera Calibration and 3D Reconstruction — calibrateHandEye
- MoveIt Tutorials: Hand-Eye Calibration
Sources reviewed on September 23, 2026.
Related reading
Sean Woo — I have spent more than 15 years shaping robotics technology and business direction. Drawing on public technical documents, research papers and company announcements, I explain changes in robotics and AI in accessible terms. The interpretations in this publication do not represent the official position of any company or organization.