VisualForce: Robot Learning
with Visual Predicted Force

Haonan Chen* Feiyang Wu* Yuxiang Ma Mustafa Mete Pengfei Ye Junxuan Shen Cheng Zhu Aurora Ruggeri Kelvin Cheung Jiayuan Mao Edward Adelson Jiajun Wu Robert D. Howe Yilun Du

Harvard University Massachusetts Institute of Technology (MIT) University of Pennsylvania Stanford University

* Equal contribution

Read the paper Code Watch overview · 2:30
Overview · 2:30Includes narration · Turn on sound

VisualForce learns force-aware manipulation from gripper deformation, using predicted force to choose actions without force or tactile sensors at deployment.

Abstract

Force-aware manipulation typically relies on specialized force or tactile sensors. We show that force-aware manipulation can instead be achieved through visual force prediction from the deformation of a compliant Fin Ray gripper. Our approach trains two models. First, we train a visual force estimator on calibration data and use it to annotate task demonstrations with force estimates. Second, we train an action–force proposal policy on these force-augmented demonstrations to jointly generate candidate robot actions and their associated forces. At test time, we sample candidate actions and the forces they are expected to produce, then execute the action whose predicted force is closest to a target from the demonstrations. We evaluate our approach on berry picking, empty-can grasping, in-hand reorientation, and plug insertion. Our results show that visual force prediction can guide inference-time action selection for contact-rich manipulation without requiring force or tactile sensors at deployment.

See force through deformation

Interactive illustration

Increase the gripping force and watch the compliant fingers bend around an object. The dashed outline shows their unloaded shape.

Fin Ray gripper Compliant fingers under load
Compliant fingers under loadTwo tapered green outline fingers, each with four diagonal ribs, bend around an object as the illustrative force increases. Dashed outlines show the unloaded fingers.
Unloaded outlineLoaded shape
0 N · unloaded10 N · more deformation

Watch the fingers’ outline and internal ribs change as the load increases.

Illustrative loads, not calibrated measurements. Explore recorded force estimates ↓

Experiments

Both methods use the same trained policy. The baseline executes one sampled action; VisualForce samples 32 action–force pairs and chooses the action whose predicted force is closest to a target from demonstrations.

Success rates: 15 trials per task and method. Full results in the paper ↗. Videos are sped up; clip lengths do not indicate task completion times.

Plug insertion

Maintain enough grip to insert the plug without slipping.

Baseline26.7% success

The plug slips within the gripper.

↓
VisualForce73.3% success

The robot maintains its grip during insertion.

↓

Berry picking

Hold the berry securely without crushing it.

Baseline0.0% success

Excessive grip crushes the berry.

↓
VisualForce73.3% success

The berry stays intact during transfer.

↓

Empty-can grasping

Lift a thin-walled can without denting it.

Baseline46.7% success

Excessive force dents the can.

↓
VisualForce86.7% success

The can is transferred without damage.

↓

In-hand reorientation

Allow the plate to pivot while maintaining contact with the table.

Baseline6.7% success

The plate does not reorient upright.

↓
VisualForce46.7% success

The plate pivots upright.

↓

Recorded force estimates

Play the wrist-camera video to watch force change as the gripper deforms, or drag the time slider to inspect a moment.

Wrist-camera recording
Measured forceVisually predicted forceMotor-current estimate
Recorded force over timeMeasured force, visually predicted force, and motor-current force estimates from the paper. A cursor follows the wrist-camera video and marks the selected sample on each force trace.
0 s—

One recorded sequence from Figure 4(b). Download the data.

Loading the recorded force samples…

Citation

@misc{chen2026robotlearningvisualpredicted,
      title={Robot Learning with Visual Predicted Force},
      author={Haonan Chen and Feiyang Wu and Yuxiang Ma and Mustafa Mete and Pengfei Ye and Junxuan Shen and Cheng Zhu and Aurora Ruggeri and Kelvin Cheung and Jiayuan Mao and Edward Adelson and Jiajun Wu and Robert D. Howe and Yilun Du},
      year={2026},
      eprint={2610.04741},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2610.04741},
}

Available on arXiv:2610.04741.