Robot Foundation Models and Cross-Embodiment Learning

Why Does This Exist?

In natural language processing and computer vision, foundation models thrive because billions of text documents and images share universal representations: standard Unicode text and RGB pixel grids.

In robotics, historically every laboratory trained isolated models on their own specific hardware. A policy trained for a 7-DOF Franka Panda arm could not be loaded onto a 6-DOF UR5 arm or a low-cost 4-DOF WidowX arm. If a robot broke or an institution upgraded hardware, all past teleoperation data became obsolete:

1. Morphological incompatibility: Robots differ fundamentally in kinematic structure, joint limits, link lengths, and degrees of freedom. A joint-space command $\dot{q} \in \mathbb{R}^7$ is meaningless to a 4-axis or dual-arm robot.
2. Sensor and viewpoint divergence: One robot setup uses an over-the-shoulder wide-angle camera; another uses a narrow wrist-mounted eye-in-hand sensor; a third uses stereo depth cameras.
3. Data scarcity: Collecting 10,000 real-world demonstrations on a single robot arm can take months of human labor. No single lab can collect enough data to achieve internet-scale generalization.

Cross-embodiment robot foundation models (such as Octo and models trained on the Open X-Embodiment dataset) solve this by aggregating heterogeneous demonstration datasets across dozens of institutions. By projecting diverse robot hardware into a shared, morphology-agnostic representation and conditioning the neural network on embodiment tokens, a single foundation model learns universal manipulation physics—like grasping, pushing, and inserting—that transfer to novel physical bodies with minimal adaptation.

Think of It Like This

How It Actually Works

Training a cross-embodiment foundation model requires solving two core technical challenges: defining a universal action interface and rebalancing wildly mismatched datasets.

1. Universal Cartesian Action Space

Instead of predicting raw joint angles $\mathbf{q} \in \mathbb{R}^n$ (which are strictly tied to a robot's specific kinematic joint count), cross-embodiment models standardize on end-effector Cartesian delta actions:

• $\Delta x, \Delta y, \Delta z$: 3D translation of the tool center point (TCP) relative to the robot base or camera frame.
• $\Delta \theta_x, \Delta \theta_y, \Delta \theta_z$: 3D rotation delta (often expressed as Euler angle differences, axis-angle, or 6D rotation representations).
• $g \in [0, 1]$: Continuous normalized gripper command ($0 = \text{open}$, $1 = \text{closed}$).

Because every manipulator interacts with the physical world through an end-effector tool, Cartesian delta actions allow a 4-DOF arm, a 7-DOF arm, and a dual-arm system to share the exact same output layer. A low-level Inverse Kinematics (IK) or operational space controller on each specific robot then maps these Cartesian deltas into its own specific joint motor currents.

To inform the shared backbone of kinematic differences (such as maximum reach or stiffness), the network is conditioned on discrete embodiment identifier tokens (e.g. [EMB_FRANKA], [EMB_WIDOWX], [EMB_ALOHA]) appended to the prompt sequence.

2. Temperature-Scaled Dataset Rebalancing

Cross-embodiment datasets (like Open X-Embodiment) combine massive datasets from industrial labs with tiny datasets from specialized academic setups.

If training batches are sampled linearly proportional to dataset size ($N_i$), massive datasets completely dominate training gradients, and the model fails to learn the tasks in small datasets. Conversely, if datasets are sampled uniformly, the model rapidly overfits to small datasets and memorizes noisy demonstrations.

To achieve an optimal balance, dataset mixing probabilities are calculated using temperature scaling:

Let $N_i$ be the number of demonstrations in dataset $i$, and let $\tau \in [0, 1]$ be the dataset mixing temperature. The sampling weight $w_i$ is:

The normalized sampling probability $P(i)$ for dataset $i$ is:

• $\tau = 1.0$: Standard linear frequency sampling (proportional to dataset size).
• $\tau = 0.0$: Uniform sampling (each dataset has equal weight regardless of size).
• $\tau = 0.5$: Square-root rebalancing (the standard practice in modern robot foundation models), which boosts rare datasets while respecting the statistical power of large datasets.

3. Worked Example: Linear vs. Temperature Rebalancing

Consider training a cross-embodiment foundation model on two distinct datasets:

• Dataset A (WidowX Tabletop Pick-and-Place): $N_A = 10{,}000$ demonstrations.
• Dataset B (Franka High-Precision Peg-in-Hole): $N_B = 100$ demonstrations.

Case 1: Linear sampling ($\tau = 1.0$)

Total trajectories: $10{,}000 + 100 = 10{,}100$.

Under linear sampling, only 1 out of every 101 training batches contains a precision peg-in-hole demonstration. The foundation model fails to learn fine assembly skills.

Case 2: Square-root temperature scaling ($\tau = 0.5$)

Calculate the scaled weights:

Total weight sum: $100 + 10 = 110$.

Calculate the new sampling probabilities:

Applying square-root temperature scaling increases the sampling frequency of Dataset B from $0.99\%$ to $9.09\%$—a $9.18\times$ boost. The model sees precision peg-in-hole demonstrations roughly 1 out of every 11 batches, allowing it to acquire high-precision insertion skills without starving WidowX general pick-and-place coverage.

Code

The following Python script computes temperature-scaled dataset mixing weights and normalizes Cartesian action deltas across different robot embodiments:

1from dataclasses import dataclass2from typing import Dict, List, Tuple3import numpy as np4
5
6@dataclass7class EmbodimentDataset:8    name: str9    num_trajectories: int10    action_mean: np.ndarray  # [dx, dy, dz] in meters11    action_std: np.ndarray12
13
14def compute_mixing_probabilities(15    datasets: List[EmbodimentDataset], temperature: float = 0.516) -> Dict[str, float]:17    """Calculates temperature-scaled sampling probabilities for heterogeneous datasets."""18    weights = [d.num_trajectories ** temperature for d in datasets]19    total_weight = sum(weights)20    return {d.name: w / total_weight for d, w in zip(datasets, weights)}21
22
23def normalize_action(24    raw_action: np.ndarray, dataset: EmbodimentDataset, eps: float = 1e-625) -> np.ndarray:26    """Normalizes raw physical action to zero mean and unit variance per embodiment."""27    return (raw_action - dataset.action_mean) / (dataset.action_std + eps)28
29
30def denormalize_action(31    normalized_action: np.ndarray, dataset: EmbodimentDataset32) -> np.ndarray:33    """Reconstructs physical metric action from model output."""34    return (normalized_action * dataset.action_std) + dataset.action_mean35
36
37# Define two datasets matching worked example38widowx = EmbodimentDataset(39    name="WidowX_PickPlace",40    num_trajectories=10_000,41    action_mean=np.array([0.000, 0.000, -0.002]),42    action_std=np.array([0.025, 0.025, 0.015]),43)44
45franka = EmbodimentDataset(46    name="Franka_PegHole",47    num_trajectories=100,48    action_mean=np.array([0.000, 0.000, -0.0005]),49    action_std=np.array([0.005, 0.005, 0.002]),50)51
52# Verify temperature scaling53datasets = [widowx, franka]54linear_probs = compute_mixing_probabilities(datasets, temperature=1.0)55balanced_probs = compute_mixing_probabilities(datasets, temperature=0.5)56
57print(f"Linear: Franka = {linear_probs['Franka_PegHole'] * 100:.2f}%")58print(f"Sqrt:   Franka = {balanced_probs['Franka_PegHole'] * 100:.2f}%")59
60assert abs(balanced_probs["Franka_PegHole"] - 0.0909) < 0.00161
62# Verify action normalization63raw_franka_step = np.array([0.005, 0.000, -0.0025])64norm_act = normalize_action(raw_franka_step, franka)65recovered = denormalize_action(norm_act, franka)66
67assert np.allclose(raw_franka_step, recovered)68print(f"Normalized Franka action: {norm_act}")

Watch Out For

The Quick Version

• Robot foundation models pool demonstration datasets across diverse robot arms, mobile bases, and camera setups to learn universal physical manipulation priors.
• To handle varying degrees of freedom, models standardize on 7-DoF Cartesian end-effector delta actions ($\Delta x, \Delta y, \Delta z, \Delta \text{rot}, \text{grip}$) mapped by low-level IK.
• Discrete embodiment tokens ([EMB_ID]) inform the transformer of specific hardware kinematic limits and control conventions.
• Heterogeneous dataset scales require temperature-scaled rebalancing ($w_i = N_i^\tau$). Square-root scaling ($\tau = 0.5$) prevents large datasets from drowning out small, specialized demonstration sets.
• Cross-embodiment pre-training drastically reduces fine-tuning data requirements: a few dozen target demonstrations can adapt a massive pre-trained model to a new robot.

What to Read Next

• Vision-Language-Action Models
• Learning Robot Manipulation from Demonstrations
• Deploying Robot Policies Across the Reality Gap
• Embodied AI