← All selected projects← Back to homepage
Prototype2026

ArUco + MediaPipe Hand Tracking

A computer-vision prototype combining calibrated ArUco pose estimation, MediaPipe hand landmarks and a Kalman-based fusion path with real-time visualisation and recordable experiment outputs.

Role

  • Developer
  • Computer vision prototyping

Stack

  • Python
  • OpenCV
  • MediaPipe
  • NumPy
  • Pandas
  • Kalman filtering

Problem

Marker-based pose estimation and markerless hand tracking offer different strengths; the prototype explores how they can be observed together and compared through one calibrated workflow.

Constraints

  • Accurate ArUco 3D pose estimation depends on camera calibration and known marker dimensions.
  • ArUco observations are sensitive to occlusion, while MediaPipe wrist landmarks are image-space estimates.
  • The same application must support live cameras, recorded video and headless data capture.

My contribution

  • Structured acquisition, detection, fusion and utility modules around a single command-line entry point.
  • Integrated camera calibration, per-frame CSV logging, raw and processed video recording and optional ground-truth comparison.
  • Added selectable ArUco, MediaPipe and fusion modes so the signals can be inspected independently or together.

Architecture / methodology

  • Estimate marker pose with OpenCV ArUco and detect hand landmarks with MediaPipe Hands.
  • Associate configured marker IDs with left and right hands and combine observations through the fusion module.
  • Capture timestamps, processing stages, detections and fused positions for later analysis.

Evaluation

  • Compare ArUco, MediaPipe and fusion modes using the same video or camera source.
  • Supply optional ground-truth positions in CSV form for frame-aligned comparison.
  • Inspect stage timings and exported frame data rather than claiming an unsupported accuracy result.

Results

  • The repository documents a runnable prototype and repeatable calibration/data-capture workflow.
  • No accuracy or runtime benchmark is claimed because the public evidence does not include a completed evaluation report.

Evidence and links