BoxFlip
Reorient a box without grasping it while preserving surface contact as the box rotates. The correction direction is the demonstrated surface normal, signed into the contact surface.
Geometric Interaction Primitive Diffusion
for Contact-Rich Manipulation
Abstract
Contact-rich manipulation requires policies that encode task geometry while adapting rapidly to uncertain contact dynamics. We present Geometric Interaction Primitive Diffusion (GIPD), a structured policy-controller interface that represents Cartesian position, orientation, symmetric positive-definite stiffness, and interaction direction in a phase-aligned primitive. A diffusion model predicts dense trajectories, while a geometric ProDMP decoder produces smooth, boundary-consistent motion. At inference, the policy uses vision and proprioception without direct wrench input. During execution, measured force is projected along the predicted interaction direction and drives bounded correction either to restore surface contact or advance through resistance, without modifying the nominal primitive in the orthogonal subspace or prescribing a desired force. To determine when this correction should engage and when execution should stop, we introduce a failure-driven curriculum initialized from task-level interval annotations. Subsequent rollouts are executed autonomously, while recorded task events and frozen predecessor models generate dense labels as the reliable task prefix is extended from Failed through Nice Try to Successful outcomes. This process refines kinesthetic demonstrations that fail under direct replay without repeated frame-by-frame annotation. Experiments on box flipping, MCB switch toggling, and LEGO insertion show improved overall task success over vision-only and compliance-learning baselines.
Paper: PDF
Paper Methodology: Geometric ProDMP
A Geometric Interaction Primitive synchronizes four quantities with one primitive phase and one set of boundary conditions:
gi0 = (pi0, qi, Ki, ni) ∈ ℝ3 × SO(3) × S++ × S2
| Channel | Geometry | What it encodes | Training loss |
|---|---|---|---|
| Position p(t) | ℝ3 | The nominal Cartesian path. | Euclidean MSE |
| Orientation q(t) | SO(3) | A smooth unit-quaternion trajectory. | Geodesic rotation error |
| Stiffness K(t) | S++ | A valid symmetric positive-definite stiffness trajectory. | Affine-invariant SPD distance |
| Interaction direction n(t) | S2 | The task direction in which force feedback may adapt motion. | Validity/confidence-weighted angular error |
1. Smooth, boundary-consistent decoding
Every channel uses the same boundary-conditioned ProDMP. Non-Euclidean components are first mapped to local coordinates, evolved through the primitive, and then returned to their native manifolds.
Generic ProDMP
Position evolution
Orientation evolution
Stiffness evolution
Interaction-direction evolution
Geometric ProDMP rollouts
These real-robot rollouts visualize the decoded trajectory together with its SPD stiffness ellipsoids and task-relevant direction axes. Teal marks the interval in which execution-time admittance is active.
2. Learning when interaction is needed
Direct replay can fail because a demonstration does not say when compliance should engage. Failed rollouts reveal the first persistent stall, slip, or loss of contact; Nice Try rollouts extend the reliable task prefix; successful rollouts identify completion. This progression supervises separate visual predictors for when to request correction and when to stop—without frame-by-frame annotation.
3. Bounded, task-conditioned directional admittance
Directional admittance is a separate execution-time module. The decoded GIP supplies the nominal position and a signed interaction direction, while calibrated force is projected onto that direction to obtain a nonnegative reaction signal. The controller represents the correction as one scalar displacement along the direction, so the nominal ProDMP and predicted stiffness continue to govern the entire orthogonal subspace.
fc,i = max(0, −f̃iTni) ci = ρini pd,i = pd,i0 + ci
The selected direction determines the task mode. Along the contact axis, weak reaction triggers bounded inward motion to restore engagement; along the motion axis, measured resistance modulates progress toward visual completion. Excessive compression commands a bounded retreat. A filtered visual-admittance signal, completion latch, phase eligibility, direction validity, and safety/freshness checks gate the correction, while acceleration and displacement limits introduce and remove it smoothly. No desired-force setpoint is tracked, and wrench remains outside neural-policy inference.
Experiments
We evaluate fixed policies on a Franka FR3 across three contact-rich tasks. All methods use the same demonstrations, RGB observations, robot interface, and Cartesian impedance controller. Challenging conditions change the physical interaction without retraining.
MCB switch toggling
Engage a small lever and traverse its resistive transition without losing tip contact. The correction follows the demonstrated motion line toward the toggled state.
LEGO insertion
Align two blocks, establish contact, and maintain axial progress until fully seated. The correction follows the demonstrated insertion line toward completion.
Nominal-condition success
Physical task completion over 20 real-robot trials per task.
| Method | BoxFlip | MCB | LEGO |
|---|---|---|---|
| Diffusion Policy | 5/20 | 0/20 | 3/20 |
| Adaptive Compliance Policy | 15/20 | 4/20 | 5/20 |
| GIPD (ours) | 20/20 | 19/20 | 16/20 |
Challenging-condition success
Ten trials per task without retraining: a nearly fivefold box-mass increase, a slippery MCB tool tip, and a diagonal LEGO placement offset.
| Method | BoxFlip | MCB | LEGO |
|---|---|---|---|
| Diffusion Policy | 0/10 | 0/10 | 0/10 |
| Adaptive Compliance Policy | 7/10 | 0/10 | 0/10 |
| GIPD (ours) | 10/10 | 7/10 | 5/10 |