Luwu Dynamics Quadruped — Reinforcement Learning Training & Deployment Tutorial

Audience: Mini2S / Mini2SW / Lite3 quadruped robots, trained in MuJoCo simulation with RSL-RL (PPO) and deployed to real hardware. Code base: luwu_mjlab (extension of mjlab v1.2.0).


Table of Contents


Part 0 Fundamentals

0.1 Reinforcement Learning & PPO

Reinforcement Learning (RL) is a computational paradigm for sequential decision-making. The learner is embodied as an Agent that learns through continuous interaction with an Environment: at each step it senses the State, picks an Action according to its Policy, the environment transitions to a new state and emits a Reward as feedback. The objective is not to maximize single-step reward but to optimize the policy and value function so that the long-term cumulative Return over the whole trajectory is maximized.

Five core elements

Element Meaning In this project
State Information about the agent relative to the environment, used to decide future actions/observations/rewards Base pose, joint angles, velocities, IMU (see Part 2 observations)
Action Operation executable in the current situation 12/15 joint target-angle offsets (see Part 2 actions)
Reward Quantitative feedback on action quality Weighted mix of velocity tracking, posture, gait (see Part 3)
Return Discounted sum of future rewards Discount factor γ=0.99
Policy State→action mapping; the "solution" to be learned Neural-network Actor (see Part 4)

Discount factor γ: without discount, returns diverge in infinite-horizon tasks; economically, immediate reward is worth more than uncertain future reward. γ→0 is myopic, γ→1 is far-sighted. Here gamma=0.99.

PPO (Proximal Policy Optimization) is a model-free, on-policy, Actor-Critic policy-gradient algorithm. Its core idea is to build a trust region for policy updates, keeping the new policy π_θ close to the old policy π_θ_old so that overly large update steps cannot collapse performance. Compared with TRPO's second-order optimization, PPO achieves an approximate trust region with first-order clipping, striking the best balance of simplicity / sample efficiency / stability.

Two key PPO techniques:

  • Clipped Surrogate Objective: build the probability ratio r_t(θ)=π_θ/π_θ_old and clip it to [1−ε, 1+ε] (here clip_param=0.2). When advantage is positive, increase action probability but with an upper bound; when negative, decrease but with a lower bound.
  • Adaptive KL Penalty: add a KL-divergence regularizer to the objective; monitor the measured KL mean and dynamically adjust β: if KL exceeds desired_kl=0.01 strengthen the constraint, otherwise relax it. Here schedule="adaptive" uses this mechanism.

Common RL algorithms (reference)

Family Algorithm Core idea Traits Use
Value DQN Deep net approximates Q* Off-policy, discrete action High-dim discrete control
Policy REINFORCE Monte-Carlo policy gradient High variance, no value fn Episodic
Policy TRPO Monotonic improvement under KL Strong theory, heavy compute Stability-critical continuous control
Policy PPO Clip/penalty approximate trust region Simple, tuner-friendly Continuous control default (this project)
AC A2C/A3C Actor+Critic advantage reduces variance Parallel speedup Mid complexity
Policy SAC Maximum-entropy regularizer Off-policy, sample efficient High-perf continuous control

0.2 Robot Models: URDF → MuJoCo MJCF

URDF (Unified Robot Description Format) is an XML format describing a robot's kinematics, inertia, and geometry, as a tree (no parallel mechanisms). Two core components:

  • Joint: connects parent/child links and defines kinematics. Types: revolute (limited angle), continuous (unlimited), prismatic (translation), fixed. Key attributes: <origin> (zero pose xyz+rpy), <axis> (motion-axis unit vector, expressed in the child link frame, right-hand rule), <limit> (position/velocity/torque).
  • Link: rigid-body geometry/mass/inertia. Children: <visual> (appearance), <collision> (simplified collision), <inertial> (mass + 3×3 positive-definite symmetric inertia matrix ixx/ixy/…).

From SolidWorks to MJCF

  1. Assemble in SolidWorks; create coordinate frames at each link CoM and joint (Z up, X forward, right-hand rule), and axes at joint rotation axes.
  2. Install the sw2urdf plugin to export URDF (note: exported inertia is often wrong—recompute inertia at each part CoM in SW).
  3. Import URDF into MuJoCo: add meshdir in <compiler>, optionally balanceinertia="true" (auto-correct inertia), discardvisual="false"; use MuJoCo compile to compile URDF into MJCF (.xml). If face count is too high, decimate with MeshLab.

This project already ships compiled MJCF: src/assets/robots/<robot>/xmls/<robot>.xml (mesh + collision geometry), no manual export needed. The simulation backend is GPU-accelerated mujoco-warp, not the legacy mujoco_py.

0.3 Simulation Environment Setup

Dependency stack (Python 3.11, pinned in setup.py):

Package Version Notes
mjlab 1.2.0 RL env framework (Isaac Lab-style API + MuJoCo Warp)
mujoco 3.6.0 Physics engine
mujoco-warp 3.6.0 GPU-accelerated MuJoCo
warp-lang 1.12.1 GPU compute DSL (must be < 1.14; mjlab 1.2.0 depends on wp.context.runtime)
scipy ≥1.17.0 Scientific computing

Linux (Ubuntu 22.04) (needs NVIDIA GPU + driver 550+ + Python 3.11):

conda create -n luwu_mjlab python=3.11
conda activate luwu_mjlab
git clone https://github.com/LuwuDynamics/luwu_mjlab.git
cd luwu_mjlab
pip install -e .
python scripts/list_envs.py --keyword Mini2S      # list registered tasks
python scripts/play.py Mini2S-Walk-Flat --agent zero  # MDP sanity check

Windows (Win10/11 + NVIDIA GPU ≥ RTX 20xx + ≥16 GB RAM): install R570+ driver, verify with nvidia-smi; Miniconda env; conda install -c nvidia cuda-toolkit=13.0.2 cudnn; pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130; then pip install -e .. Smoke test: python scripts/play.py Mini2S-Walk-Flat --agent zero --num-envs 4. Use --viewer viser (web http://localhost:8080) when no display.

Google Colab: new T4 GPU notebook, install and background-train with TensorBoard (see Part 6).


Part 1 Project Architecture & Code Organization

1.1 Repository Overview

scripts/
  train.py              # Training entry (RSL-RL PPO)
  play.py               # Inference / evaluation entry
  list_envs.py          # List registered tasks
  visualize_terrain.py  # Terrain preview
src/
  assets/
    robots/             # Robot MJCF XML, mesh, constants
      mini2s/  mini2sw/  lite3/
    motions/            # Reference motions for tracking tasks (.npz)
  tasks/
    __init__.py         # import_packages() auto-registers tasks
    velocity/           # Velocity-tracking locomotion task
      velocity_env_cfg.py     # make_velocity_env_cfg() factory (MDP assembly)
      config/
        mini2s_walk/  mini2s_turn/  mini2sw_walk/  mini2sw_turn/  lite3_walk/
          env_cfgs.py         # env / reward / termination / curriculum overrides
          rl_cfg.py           # PPO network & algorithm hyperparameters
      mdp/              # observations / rewards / terminations / curriculums / velocity_command
      rl/runner.py      # VelocityOnPolicyRunner (ONNX export)
    tracking/           # Motion-imitation tracking task (mirrors velocity/)
deploy/
  run.py                # Unified real-robot deployment CLI entry
  xgo_lib2/             # Serial comms library (RL-independent, reusable by ROS2)
    protocol.py         # Frame pack / checksum / parse (header 55 00, tail 00 AA)
    client.py           # XgoDog high-level API (version/battery, write servo angles, enable feedback)
    feedback.py         # SendStateParser: ~50Hz, 254B push-state frame
    registry.py         # Field registry; feedback_slot_of(motor_id) maps feedback slots
    protocol.md         # Full protocol docs
  xgo_rl/               # RL control layer (built on top of the comms library)
    model_cfg.py        # [config] RobotConfig + device factories + WALK/TURN constants
    robot.py            # [hardware] Joint/JointSet/RobotState/Robot (read feedback, write servos, gravity LPF)
    gait.py             # [gait] GaitLogic: tick phase, map user command -> model command
    policy.py           # [policy] OnnxPolicy: assemble obs -> infer -> target joint angles
    controller.py       # [facade] Controller (alias Robot) + mode machine + 50Hz loop
    models/             # policy_<model>_<gait>.onnx

1.2 Task Auto-Registration

src/tasks/__init__.py auto-registers tasks via import_packages() when submodules are imported. scripts/train.py imports mjlab.tasks and src.tasks at startup, then the registry (mjlab.tasks.registry) provides list_tasks() / load_env_cfg() / load_rl_cfg() / load_runner_cls(). Task IDs look like Mini2S-Walk-Flat, Mini2S-Turn-Flat, Mini2SW-Walk-Flat, Lite3-Walk-Flat, etc.

Execution flow: main() → tyro parses the first arg to pick the task + remaining args override TrainConfig(env+agent) → launch_training() → run_train(). Single GPU runs directly; multi-GPU is dispatched via torchrunx.

1.3 velocity Task Structure

  • Factory make_velocity_env_cfg() (velocity_env_cfg.py): returns ManagerBasedRlEnvCfg that centrally defines sensors, observations, actions, commands, events (domain randomization), rewards, terminations, curriculum, and metrics as the common baseline for all robots/tasks.
  • Robot/task overrides config/<robot>_<task>/env_cfgs.py: customize on top of the factory (timestep, decimation, contact sensors, reward weights, per-joint std dicts, command ranges, reset randomization).
  • config/<robot>_<task>/rl_cfg.py: defines Actor/Critic network structure and PPO hyperparameters.
  • mdp/: reusable implementations of observations, rewards, terminations, curriculum, commands.
  • rl/runner.py: extends MjlabOnPolicyRunner; on save() additionally exports ONNX and writes metadata.

1.4 Asset Organization

  • Robots: src/assets/robots/<robot>/xmls/<robot>.xml (MJCF) + mesh + *_constants.py (initial pose INIT_STATE, collision config MINI2S_WALK_COLLISION/MINI2S_TURN_COLLISION, actuators). E.g. Mini2S default initial joint angles: thigh=1.0, calf=0.0, hip=0.0, arm_yaw=0, arm_thigh=−1.57, arm_calf=1.35 (rad). Walk task collides only with calf feet; Turn task lets the whole body touch ground (for get-up).
  • Reference motions: src/assets/motions/*.npz (tracking tasks).

1.5 Deployment Layers

Real-robot deployment is split by responsibility into a comms library and an RL control layer built on top of it:

  • deploy/xgo_lib2/ — comms library (RL-independent, reused directly by ROS2): the XGO serial protocol — frame pack/checksum, the XgoDog client, ~50Hz push-state parsing, and a field registry (feedback_slot_of(motor_id)). This is the only hard dependency on the robot side.
  • deploy/xgo_rl/ — RL control layer, internally five layers:
    • [config] model_cfg.py: RobotConfig + device factories (mini2s()/mini2sw()/lite3()) + WALK/TURN constants.
    • [hardware] robot.py: Joint/JointSet/RobotState/Robot — read feedback, write servo angles, gravity LPF.
    • [gait] gait.py: GaitLogic — tick gait phase, map user command → model command.
    • [policy] policy.py: OnnxPolicy — assemble observation → infer → target joint angles.
    • [facade] controller.py: Controller (alias Robot) — mode machine + 50Hz control loop; the single external entry with a thread-safe state snapshot.

Part 2 MDP Design

2.1 Observation Space

Observations are split into actor and critic groups, concatenate_terms=True concatenates into a vector; actor enables noise and corruption (enable_corruption=True), critic does not (more stable training).

actor observation terms (with uniform noise UniformNoise):

Term Function Dim Noise
base_ang_vel IMU angular velocity 3 ±1.0
projected_gravity Projected gravity (body frame) 3 ±0.1
command Velocity command (vx,vy,yaw) 3 —
phase Gait phase (sin, cos) 2 —
joint_pos Joint angle (rel. default) J ±0.1
joint_vel Joint angular velocity J ±2.0
actions Previous action J —
height_scan* Terrain ray height 187 ±0.1

critic observation terms = all actor terms + base_lin_vel(3) + foot_height(J) + foot_air_time(J) + foot_contact(J) + foot_contact_forces(3J).

* height_scan exists only in tasks with terrain scanning; the Flat config deletes the height_scan term from both actor and critic (see del cfg.observations[...].terms["height_scan"] in mini2s_flat_walk_env_cfg), so the flat policy input dimension excludes terrain scan. The exported ONNX input dimension matches this trimmed observation set.

phase observation: global_phase = (episode_length_buf * step_dt) % period / period, take (sin, cos); zeroed when standing (command norm < 0.1).

2.2 Action Space

  • walk: JointPositionActionCfg(scale=0.25, use_default_offset=True) — output is the joint-angle offset relative to default pose, scaled and added to the default angle.
  • turn (get-up): HeldJointPositionActionCfg(hold_duration_s=1.0) — each action is held for 1s to avoid high-frequency jitter.

2.3 Commands

UniformVelocityCommandCfg: velocity command sampled uniformly over [lin_vel_x, lin_vel_y, ang_vel_z, heading]; resampling_time_range=(3.0, 8.0)s periodically resamples; heading_command=True adds heading control; rel_standing_envs=0.05 (~5% of envs stay standing). Curriculum can progressively widen the command range (see 2.4).

2.4 Domain Randomization, Termination & Curriculum

  • Events (domain randomization): reset_base (reset pose/velocity), reset_robot_joints (joint offset/velocity), push_robot (interval 5–6s random push for disturbance robustness), foot_friction (startup, all foot geoms share random friction 0.2–3.0), encoder_bias (±0.04 rad), base_com (CoM offset).
  • Terminations: time_out (episode timeout), fell_over (bad_orientation, body-vs-up angle > 70°).
  • Curriculum: terrain_levels_vel (terrain difficulty advances with performance), commands_vel (progressively widen speed range, e.g. step 0→5000×24 from (±0.5,1.0) to (±1.0,2.0)).

Part 3 Reward Function Design

Rewards are implemented in src/tasks/velocity/mdp/rewards.py, organized into four groups: velocity tracking / posture / gait shaping / penalties. Each term is a RewardTermCfg(func=..., weight=..., params=...); total reward is the weighted sum. The base weights below come from velocity_env_cfg.py; override weights come from mini2s_walk (robot-specific tasks tune further on top).

3.1 Velocity Tracking

Reward Form Base wt Override wt Notes
track_linear_velocity exp(-(‖cmd_xy−v_xy‖² + 2·v_z²)/std²) 1.0 (std=√0.25) 4.0 (std=0.15) Track horizontal lin. vel; penalize vertical
track_angular_velocity exp(-((cmd_z−ω_z)² + 0.05·‖ω_xy‖²)/std²) 1.0 (std=√0.5) 4.0 (std=0.15) Track yaw rate

Gaussian form exp(-err²/std²): reward → 1 as error → 0; std controls tolerance (smaller = stricter). Override raises weight to 4.0 and tightens std to 0.15, prioritizing tracking.

3.2 Posture

  • pose / variable_posture: by command speed in three regimes, per-joint std computes exp(-mean((q−q₀)²/std²)):
    • standing std_standing: hip 0.05 / thigh 0.1 / calf 0.15
    • walking std_walking: hip 0.15 / thigh 0.35 / calf 0.35
    • running std_running: hip 0.15 / thigh 0.5 / calf 0.5
    • arm joints fixed 0.1; thresholds walking_threshold=0.05, running_threshold=0.3. Higher speed allows larger deviation from default pose.
  • gravity_gated_variable_posture (turn only): multiplies variable_posture by a "non-upright gate" (projected_gravity_z < −0.75), encouraging correct get-up posture only when not upright.

3.3 Gait Shaping

Reward Weight Notes
foot_gait 0.5 → 1.0 Phase-based: leg_phase=(t/period+offset)%1; stance when leg_phase<threshold(0.56); reward when matches actual contact; offset=[0,0.5,0,0.5] gives diagonal gait, period=0.4
foot_clearance −1.0 → −0.5 Penalize foot-height deviation from target (walk override 0.02m), weighted by foot speed, active only when command is present
feet_air_time — Reward when swing time near threshold (0.4s); at single-leg stance take min of contact/air time
feet_slip −0.25 → −0.05 Penalize foot horizontal velocity while in contact (slipping)
soft_landing −1e-3 Penalize high impact force at first contact; encourage soft landing

3.4 Penalties

Reward Weight Notes
body_orientation_l2 −1.0 Penalize projected-gravity xy (keep body level)
body_ang_vel −0.05 → −0.08 Penalize body roll/pitch rate
angular_momentum −0.025 → −0.03 Penalize whole-body angular momentum (natural arm swing)
nonfoot_contact −3 Illegal contact (non-foot touch), force_threshold=0.5
joint_acc_l2 −2.5e-7 Penalize joint acceleration (smoothness)
joint_pos_limits −10.0 Penalize exceeding joint limits
action_rate_l2 −0.05 → −0.2 Penalize action change rate (smoothness)
stand_still −1.0 Penalize joint deviation from default when command≈0
is_terminated −200.0 Termination penalty (dominant, avoids frequent falls)

3.5 walk vs turn Reward Differences

turn (get-up) task key adjustments:

  • Remove track_linear_velocity / track_angular_velocity / foot_gait / foot_clearance / feet_slip / soft_landing / stand_still (get-up needs no velocity tracking / specific gait).
  • pose becomes gravity_gated_variable_posture, weight 10.0, gravity gate threshold −0.75.
  • Add recovery_progress (weight 10.0): linear "uprightness progress" signal (1−z)·0.5 from projected gravity z, guiding rise from lying pose.
  • Add joint_vel_l2 (−3e-2) for smooth joint velocity; body_ang_vel/angular_momentum become gravity-gated variants.
  • Remove fell_over/illegal_contact terminations (get-up naturally touches ground); clear curriculum.
  • Episode 8s, timestep 0.002, decimation 10; reset_base randomizes roll/pitch/yaw fully to create initial lying poses.

Part 4 Policy Network & Training Algorithm Parameters

4.1 Policy / Value Network Structure

From mini2s_walk/rl_cfg.py RslRlModelCfg:

Config Actor Critic
hidden_dims (512, 256, 128) (512, 256, 128)
activation elu elu
obs_normalization True True
Output dist. Gaussian, init_std=1.0, std_type="scalar" — (value)
  • Three fully-connected layers (512→256→128) + ELU activation: moderate parameter count, sufficient expressiveness.
  • obs_normalization=True: maintains running mean/std of observations during training; the normalization stats are baked into the exported ONNX, so deployment only feeds raw (relative-to-default-pose / scaled) observations consistent with training.
  • Actor outputs Gaussian mean; standard deviation init_std=1.0 (scalar, shared across actions), adapted by PPO during training.

4.2 PPO Hyperparameter Details

From RslRlPpoAlgorithmCfg + RslRlOnPolicyRunnerCfg:

Param Value Meaning
value_loss_coef 1.0 Value-loss weight
use_clipped_value_loss True Also clip value loss (stable)
clip_param 0.2 Trust-region half-width ε
entropy_coef 0.01 Entropy regularizer (exploration)
num_learning_epochs 5 Update passes per batch
num_mini_batches 4 Mini-batches per pass
learning_rate 1e-3 Initial LR
schedule adaptive KL-based adaptive LR
gamma 0.99 Discount factor
lam 0.95 GAE λ (advantage bias/variance trade-off)
desired_kl 0.01 KL target triggering LR adjustment
max_grad_norm 1.0 Gradient clip norm
num_steps_per_env 24 Sample steps per env per round
max_iterations 1500 Total training iterations
save_interval 100 Checkpoint every 100 iters
experiment_name mini2s_velocity_walk Log / experiment name

4.3 Key Parameter Tuning Guide

Reward weights (relative scale matters most)

  • High tracking weight (4.0) signals velocity-following as the primary goal; is_terminated=−200 dominates termination decisions, preventing the policy from "learning to fall".
  • Gait terms (foot_gait/foot_clearance/feet_slip) tune naturalness: if slipping is obvious, increase foot_slip penalty; if feet lift too high / scrape, adjust foot_clearance target and weight.
  • variable_posture per-joint std is a "soft-constraint" knob: smaller std stays closer to default pose (steadier but stiffer), larger allows bigger motion (faster but less stable).

Action distribution & exploration

  • init_std=1.0 is a common start; if early training is violently jittery/divergent, lower LR or raise max_grad_norm first; if it converges prematurely to a suboptimal policy, briefly raise init_std or entropy_coef.
  • The per-reward std (e.g. track 0.15) controls tracking tolerance, tuned together with the weight.

Learning rate & KL

  • schedule="adaptive" + desired_kl=0.01: if measured KL exceeds target, LR drops; below, LR rises. If the loss curve oscillates, check that KL monitoring sits near target; if it never drops, nudge desired_kl.
  • clip_param=0.2 is the trust-region width: larger = more aggressive (fast but risky), smaller = more conservative.

Sampling efficiency

  • Single-batch sample count = num_steps_per_env(24) × num_envs. num_envs=4096 gives massive parallel samples — key to fast training; lower num_envs if VRAM is tight.
  • num_learning_epochs=5 / num_mini_batches=4: more epochs reuse data more but risk overfitting a batch; more mini-batches make each update more stable but slower.

Discount & advantage

  • gamma=0.99: near 1 helps long-horizon planning (locomotion needs some look-ahead). lam=0.95 trades bias vs variance; higher leans MC (low bias, high variance), lower leans TD (low variance, high bias).

Part 5 Model Export & Real-Robot Deployment

Deployment environment note: The RL part of this repo is a training framework. "Training" runs on a workstation/server with an NVIDIA GPU (Ubuntu 22.04, Python 3.11) and produces .onnx policy files; "real-robot deployment" runs on the robot dog's onboard compute unit (Ubuntu 26.04 LTS aarch64, Python 3.14.4, onnxruntime CPU inference).

The deployment side is split into two layers with a clean boundary:

Layer Dir Responsibility
Comms library deploy/xgo_lib2/ XGO serial protocol: frame pack/checksum, XgoDog client, ~50Hz push-state parsing, field registry
RL control layer deploy/xgo_rl/ Built on top of the comms library: read feedback → pick gait → ONNX inference → write joint angles

The comms library is independently reusable — the ROS2 side talks to the dog straight through it (see ../ros_tutorial/tutorial_en.md), with no dependency on the training framework.

5.1 Training → ONNX Export

velocity/rl/runner.py's VelocityOnPolicyRunner.save() exports the .pt then calls export_policy_to_onnx(), and attach_metadata_to_onnx() writes metadata:

  • joint_names (all actuated joints, in policy output order)
  • action_scale (default 0.25)
  • default_joint_pos (rad)
  • observation_names (observation term order)

These metadata let the deployment-side OnnxPolicy auto-derive dimensions and mapping without manual inspection.

5.2 Deployment Code Layering

(The directory layout is shown in Part 1.1; responsibilities per layer below.)

Layering discipline: policy never touches hardware/gait; gait does not import policy nor assemble observations; robot only handles bytes and units; mode switching and loop orchestration live only in controller.

OnnxPolicy (policy layer)

  • Loads the ONNX session with CPUExecutionProvider.
  • Derives obs_dim, num_joints from input/output tensor shapes.
  • Reads joint_names / action_scale / default_joint_pos / observation_names from metadata and cross-validates against model_cfg — a mismatch fails fast at startup.
  • step(state, ctx): RobotState + GaitContext → a JointSet of target angles (degrees). Internally three steps: assemble obs → infer → + default pose / clamp / × action_scale.
  • PolicySpec is the per-gait normalization recipe (scales, gravity source), shipped with the model — not in config.

Observation layout build_obs (strictly consistent with training actor obs):

obs = [ang_vel(3) | projected_gravity(3) | command(3) | phase(2) | joint_pos(J) | joint_vel(J) | prev_action(J)]
       first 11 dims fixed                  + 3 dims per joint (J = number of actuated joints)
obs_dim = 11 + 3*J

5.3 Real-Robot Control Loop

Controller (external alias Robot) runs a background daemon thread at ctrl_hz=50 (period 0.02s); each tick runs five steps:

  1. Read feedback robot.read_state(): pull the latest feedback frame from the comms library → RobotState (joint angles/vel, pose, filtered gravity); feedback slots are auto-derived by xgo_lib2.registry.feedback_slot_of(motor_id) — no hand-written remap table needed.
  2. Advance mode _advance_mode: _ModeMachine does stand/fall hysteresis on the filtered gravity's grav_z, switching TURN↔WALK.
  3. Advance gait _gait_step: per mode pick walk_gait/turn_gait, emit GaitContext(command, phase).
  4. Infer policy _infer: pick walk/turn policy per mode, policy.step() → target joint angles.
  5. Write + publish robot.set_joint_angles() + _publish_state(): write servos; locked snapshot for any-thread get_state().

Feedback freshness guard: if no new frame arrives for feedback_stale_timeout=0.5s, stop writing; servos hold the last target angle.

Per-gait normalization recipe differences (policy.py / gait.py factories)

walk turn
ang_vel_scale 1.0 1.0
joint_vel_scale 0.2 1.0
projected_gravity constant [0,0,−1] (training distribution) LPF-filtered measured gravity
phase (sin,cos) of a 0.4s period; zeroed when command norm < 0.1 always zero
command [vx,vy,yaw] only [0,0,yaw]
clamp arm joints (indices 12,13,14) forced to 0 none

Note ang_vel is NOT read from a raw gyro — the protocol feedback frame carries no raw gyro; angular velocity comes from frame-to-frame differencing of the orientation angles (euler2).

Mode FSM (_ModeMachine in controller.py)

  • Starts in TURN (treated as get-up/recover until upright is confirmed).
  • TURN → WALK: grav_z < stand_z_threshold(−0.75) for stand_count_required=10 consecutive ticks.
  • WALK → TURN: grav_z > fall_z_threshold(−0.4) for fall_count_required=5 consecutive ticks.
  • Gravity LPF in the hardware layer: grav_filtered = 0.3·raw + 0.7·prev, then normalize (gravity_lpf_alpha=0.3).

Serial: XgoDog default /dev/ttyAMA0, then switches to run baudrate 2_000_000 (~50Hz × 254B feedback needs a high-speed UART); motor_ids order must match the ONNX output order (Mini2S: fl,fr,br,bl,arm).

5.4 Device Config Factory

RobotConfig holds only "deployment/platform/control" facts (default pose, joint order, motor mapping — these are authoritative; the ONNX is only cross-validated); the model normalization recipe is NOT here.

Device Joints Policy Notes
mini2s() 15 walk + turn Legged; motor_ids=[13,12,11,23,22,21,33,32,31,43,42,41,54,53,52], default pose [0,57.3,0]×4 + [0,−90,77.3]
mini2sw() 15 walk + turn Same hardware as mini2s, only the ONNX policy differs
lite3() 14 walk only turn_policy_path=None (stand_up() unavailable), default pose [0,57.3,0]×4 + [−88.8,74.5], stand_z_threshold=−0.9

walk and turn must be trained in the same batch (action_scale consistent), otherwise the startup cross-check fails immediately.


Part 6 Practical Commands & Troubleshooting

6.1 Training / Evaluation Commands

# Train (walk)
python scripts/train.py Mini2S-Walk-Flat --env.scene.num-envs 4096
tensorboard --logdir logs/rsl_rl/mini2s_velocity_walk

# Train (get-up)
python scripts/train.py Mini2S-Turn-Flat --env.scene.num-envs 4096

# Common overrides
python scripts/train.py Mini2S-Walk-Flat --agent.max-iterations 10000 --agent.resume
python scripts/train.py <task> --env.scene.num-envs 4096 --gpu-ids all --video

# Evaluate (latest / specific checkpoint)
python scripts/play.py Mini2S-Walk-Flat
python scripts/play.py Mini2S-Walk-Flat --checkpoint-file logs/rsl_rl/mini2s_velocity_walk/2026-xx-xx_xx-xx-xx/model_xx.pt
python scripts/play.py Mini2S-Walk-Flat --agent zero   # MDP sanity check (zero action)

# List tasks / preview terrain
python scripts/list_envs.py --keyword Mini2S
python scripts/visualize_terrain.py

Colab train + TensorBoard: clone repo, pip install -e ., restart runtime; background python scripts/train.py Mini2S-Walk-Flat --env.scene.num-envs 4096 > output.log 2>&1 &, %load_ext tensorboard + %tensorboard --logdir ./logs/rsl_rl/mini2s_velocity_walk.

Real-robot deployment

python deploy/run.py --device mini2s
python deploy/run.py --device mini2sw --port /dev/ttyUSB0
python deploy/run.py --device lite3 --port /dev/ttyAMA0 --motor-ids 11,12,13,...

On start the ONNX auto-prints model info; robot.set_velocity(vx, vy, yaw) sends velocity; Ctrl+C shuts down safely (return to default pose + close serial).

6.2 Common Issues

Symptom Cause / Fix
warp init fails pip show warp-lang → confirm 1.12.1; mjlab 1.2.0 depends on wp.context.runtime, warp-lang must be < 1.14
CUDA unavailable Check driver + nvidia-smi; on Windows ensure %CONDA_PREFIX%\Library\bin precedes PATH
DLL load failed Ensure %CONDA_PREFIX%\Library\bin is early in PATH
No motion on real robot Check motor_ids order matches ONNX joint_names; lite3 needs --motor-ids (feedback slots are auto-derived from it)
Obs dimension mismatch Ensure the deployed ONNX was exported from the Flat config (height_scan removed), obs_dim = 11 + 3J
Large deployment jitter Check action_scale, joint-velocity scales; turn action hold_duration_s appropriateness

This tutorial is compiled from the repository code (src/, scripts/, deploy/) and Luwu's official Yuque docs (RL&PPO concepts, URDF, MuJoCo install, mujoco_py, model import). The code is authoritative; for version differences refer to the corresponding setup.py and config source files.