M2Diffuser is a diffusion-based, scene-conditioned generative model that directly generates coordinated and efficient whole-body motion trajectories for mobile manipulation based on robot-centric 3D scans.

• Website, code: https://m2diffuser.github.io/, https://github.com/m2diffuser/M2Diffuser/tree/main
• Model learns from trajectory-level distributions from mobile manipulation trajectories from an expert planner
• Optimization module can flexibly accommodate physical constraints and task objectives by modeling them as cost & energy functions during inference

I read NeuralMP right before reading M2Diffuser (this paper), so I'll start by drawing connections between these two setups before diving into the details of the M2Diffuser paper.

NeuralMP vs. M2Diffuser

M2Diffuser and NeuralMP approach the same approximate problem of training a neural network to imitate a slow expert planner - except M2Diffuser looks at whole-body mobile manipulation (base + arm), while NeuralMP looks at a fixed-base manipulator.

Feature	NeuralMP	M2Diffuser
Problem scope	Fixed-base manipulator (7 DoF)	Whole-body mobile manipulation (base + arm)
Architecture	LSTM sequence model + GMM	Scene-conditioned diffusion model
Expert data	AIT* (sampling-based)	VKC (optimization-based)
Optimization (prevent collisions, ...)	Post-generation Monte Carlo sorting	In-process gradient guidance during denoising
Task target	Explicit target joint config $g$

GMMs vs. diffusion

• NeuralMP formulates motion planning as a sequential prediction problem
  • it rolls out an LSTM step-by-step, predicting the params of a GMM to capture the multi-modal nature of the paths
  • 🔑 $q_0$ → NeuralMP($q_0, PCD, g$) outputs $\Delta q_0$ → $q_1 = q_0 + \Delta q_0$ → NeuralMP($q_1, PCD$) outputs $\Delta q_1$ → ... → $q_T$, and the full trajectory is $\langle q_0, q_1, \ldots, q_T \rangle$
• M2Diffuser treats the entire trajectory as a single high-dim. image of data
  • it learns the trajectory-level distribution using a diffusion model → instead of predicting the next step $\Delta q_t$, it starts with a trajectory of pure Gaussian noise ($\tau_T$) and iteratively denoises it over $T$ steps into a smooth, coordinated trajectory
  • 🔑 M2Diffuser's base network does not take $g$ (an explicit goal pose) as an input - it takes the target objective as a separate energy gradient $\nabla e (\tau)$
    1. Initialize: Start with a full trajectory of pure Gaussian noise across the entire horizon $H$: $$\tau_T = \langle \mathcal{N}(0, I)_0, \mathcal{N}(0, I)_1, \ldots, \mathcal{N}(0, I)_H\rangle$$
    2. The denoising loop: For every diffusion step, the network evaluates the entire noisy trajectory and the point cloud ($PCD$), predicting the noise to subtract, while the energy functions physically push the trajectory toward the goal and away from obstacles:
       1. Predict noise: $\epsilon = \text{M2Diffuser}(\tau_t, PCD, t)$
       2. Calculate base mean: $\mu_t = \text{RemoveNoise}(\tau_t, \epsilon)$
       3. Calculate guidance: Compute the gradient of the task objective ($e$) and physical collisions ($c$) wrt. the trajectory - $$g_\text{guid} = \nabla_\tau (-e_\text{grasp}(\tau) - \lambda \cdot c_\text{collision}(\tau)$$
       4. Guided sample: Shift the trajectory using that gradient to get the cleaner path for the next step: $\tau_{t-1} \sim \mathcal{N}(\mu_t + \Sigma_t g_\text{guide}, \Sigma_t)$
       5. Anchor the start: Force the first frame of the trajectory to be the current robot state: $\tau_{t-1} [0] = q_0$.
    3. ^ After $T$ iterations of that loop, the noise resolves completely - $$\tau_0 = \langle q_0, q_1, \ldots, q_H \rangle$$

Optimization philosophy: sorting vs. gradient guidance

Target conditioning: explicit joints vs. energy functions

Footnotes

Q: If standard $L_2$ (MSE) loss fails on multi-modal data by averaging everything to the center, and diffusion models use MSE to train, why don't diffusion models fail?