Skip to content

Add FLAME 3D morphable model fitting from MediaPipe landmarks #418

@dreamlessx

Description

@dreamlessx

What

Fit FLAME 3D morphable model parameters from MediaPipe's 478 face landmarks, producing a textured 3D mesh from a single 2D photo.

Why

This is a key step toward the 3D roadmap. FLAME fitting from 2D landmarks enables:

  • 3D-space surgical deformations (anatomically grounded, not pixel-level)
  • Multi-view rendering of predicted results
  • Interactive 3D previews for patient consultations

Technical context

  • MediaPipe outputs 478 3D landmarks (normalized coords + depth estimate)
  • FLAME has ~5K vertices; need a landmark-to-vertex correspondence mapping
  • Existing tools like DECA or EMOCA solve similar problems
  • The challenge is making it work from MediaPipe landmarks specifically (not DLIB 68 or FAN 98)

Deliverables

  1. landmarkdiff/flame_fitting.py — FLAME parameter estimation from MediaPipe landmarks
  2. Landmark-to-FLAME vertex correspondence mapping
  3. Basic textured mesh output (OBJ or PLY)
  4. Tests with synthetic landmarks

References

Difficulty: 🔴 Advanced — requires 3D vision experience

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is neededresearchResearch exploration and experimentation

    Projects

    Status

    Todo

    Status

    To triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions