Luwu Dynamics Quadruped — Reinforcement Learning Training & Deployment Tutorial

Audience: Mini2S / Mini2SW / Lite3 quadruped robots, trained in MuJoCo simulation with RSL-RL (PPO) and deployed to real hardware. Code base: luwu_mjlab (extension of mjlab v1.2.0).

Table of Contents

• Part 0 Fundamentals
  • 0.1 Reinforcement Learning & PPO
  • 0.2 Robot Models: URDF → MuJoCo MJCF
  • 0.3 Simulation Environment Setup
• Part 1 Project Architecture & Code Organization
• Part 2 MDP (Markov Decision Process) Design
• Part 3 Reward Function Design
• Part 4 Policy Network & Training Algorithm (PPO) Parameters
• Part 5 Model Export & Real-Robot Deployment
• Part 6 Practical Commands & Troubleshooting

Part 0 Fundamentals

0.1 Reinforcement Learning & PPO

Reinforcement Learning (RL) is a computational paradigm for sequential decision-making. The learner is embodied as an Agent that learns through continuous interaction with an Environment: at each step it senses the State, picks an Action according to its Policy, the environment transitions to a new state and emits a Reward as feedback. The objective is not to maximize single-step reward but to optimize the policy and value function so that the long-term cumulative Return over the whole trajectory is maximized.

Five core elements

Discount factor γ: without discount, returns diverge in infinite-horizon tasks; economically, immediate reward is worth more than uncertain future reward. γ→0 is myopic, γ→1 is far-sighted. Here gamma=0.99.

PPO (Proximal Policy Optimization) is a model-free, on-policy, Actor-Critic policy-gradient algorithm. Its core idea is to build a trust region for policy updates, keeping the new policy π_θ close to the old policy π_θ_old so that overly large update steps cannot collapse performance. Compared with TRPO's second-order optimization, PPO achieves an approximate trust region with first-order clipping, striking the best balance of simplicity / sample efficiency / stability.

Two key PPO techniques:

• Clipped Surrogate Objective: build the probability ratio r_t(θ)=π_θ/π_θ_old and clip it to [1−ε, 1+ε] (here clip_param=0.2). When advantage is positive, increase action probability but with an upper bound; when negative, decrease but with a lower bound.
• Adaptive KL Penalty: add a KL-divergence regularizer to the objective; monitor the measured KL mean and dynamically adjust β: if KL exceeds desired_kl=0.01 strengthen the constraint, otherwise relax it. Here schedule="adaptive" uses this mechanism.

Common RL algorithms (reference)

0.2 Robot Models: URDF → MuJoCo MJCF

URDF (Unified Robot Description Format) is an XML format describing a robot's kinematics, inertia, and geometry, as a tree (no parallel mechanisms). Two core components:

• Joint: connects parent/child links and defines kinematics. Types: revolute (limited angle), continuous (unlimited), prismatic (translation), fixed. Key attributes: <origin> (zero pose xyz+rpy), <axis> (motion-axis unit vector, expressed in the child link frame, right-hand rule), <limit> (position/velocity/torque).
• Link: rigid-body geometry/mass/inertia. Children: <visual> (appearance), <collision> (simplified collision), <inertial> (mass + 3×3 positive-definite symmetric inertia matrix ixx/ixy/…).

From SolidWorks to MJCF

1. Assemble in SolidWorks; create coordinate frames at each link CoM and joint (Z up, X forward, right-hand rule), and axes at joint rotation axes.
2. Install the sw2urdf plugin to export URDF (note: exported inertia is often wrong—recompute inertia at each part CoM in SW).
3. Import URDF into MuJoCo: add meshdir in <compiler>, optionally balanceinertia="true" (auto-correct inertia), discardvisual="false"; use MuJoCo compile to compile URDF into MJCF (.xml). If face count is too high, decimate with MeshLab.

This project already ships compiled MJCF: src/assets/robots/<robot>/xmls/<robot>.xml (mesh + collision geometry), no manual export needed. The simulation backend is GPU-accelerated mujoco-warp, not the legacy mujoco_py.

0.3 Simulation Environment Setup

Dependency stack (Python 3.11, pinned in setup.py):

Linux (Ubuntu 22.04) (needs NVIDIA GPU + driver 550+ + Python 3.11):

conda create -n luwu_mjlab python=3.11
conda activate luwu_mjlab
git clone https://github.com/LuwuDynamics/luwu_mjlab.git
cd luwu_mjlab
pip install -e .
python scripts/list_envs.py --keyword Mini2S      # list registered tasks
python scripts/play.py Mini2S-Walk-Flat --agent zero  # MDP sanity check

Windows (Win10/11 + NVIDIA GPU ≥ RTX 20xx + ≥16 GB RAM): install R570+ driver, verify with nvidia-smi; Miniconda env; conda install -c nvidia cuda-toolkit=13.0.2 cudnn; pip install torch torchvision --index-url https://download.pytorch.org/whl/cu130; then pip install -e .. Smoke test: python scripts/play.py Mini2S-Walk-Flat --agent zero --num-envs 4. Use --viewer viser (web http://localhost:8080) when no display.

Google Colab: new T4 GPU notebook, install and background-train with TensorBoard (see Part 6).

Part 1 Project Architecture & Code Organization

1.1 Repository Overview

scripts/
  train.py              # Training entry (RSL-RL PPO)
  play.py               # Inference / evaluation entry
  list_envs.py          # List registered tasks
  visualize_terrain.py  # Terrain preview
src/
  assets/
    robots/             # Robot MJCF XML, mesh, constants
      mini2s/  mini2sw/  lite3/
    motions/            # Reference motions for tracking tasks (.npz)
  tasks/
    __init__.py         # import_packages() auto-registers tasks
    velocity/           # Velocity-tracking locomotion task
      velocity_env_cfg.py     # make_velocity_env_cfg() factory (MDP assembly)
      config/
        mini2s_walk/  mini2s_turn/  mini2sw_walk/  mini2sw_turn/  lite3_walk/
          env_cfgs.py         # env / reward / termination / curriculum overrides
          rl_cfg.py           # PPO network & algorithm hyperparameters
      mdp/              # observations / rewards / terminations / curriculums / velocity_command
      rl/runner.py      # VelocityOnPolicyRunner (ONNX export)
    tracking/           # Motion-imitation tracking task (mirrors velocity/)
deploy/
  run.py                # Unified real-robot deployment CLI entry
  xgo_lib2/             # Serial comms library (RL-independent, reusable by ROS2)
    protocol.py         # Frame pack / checksum / parse (header 55 00, tail 00 AA)
    client.py           # XgoDog high-level API (version/battery, write servo angles, enable feedback)
    feedback.py         # SendStateParser: ~50Hz, 254B push-state frame
    registry.py         # Field registry; feedback_slot_of(motor_id) maps feedback slots
    protocol.md         # Full protocol docs
  xgo_rl/               # RL control layer (built on top of the comms library)
    model_cfg.py        # [config] RobotConfig + device factories + WALK/TURN constants
    robot.py            # [hardware] Joint/JointSet/RobotState/Robot (read feedback, write servos, gravity LPF)
    gait.py             # [gait] GaitLogic: tick phase, map user command -> model command
    policy.py           # [policy] OnnxPolicy: assemble obs -> infer -> target joint angles
    controller.py       # [facade] Controller (alias Robot) + mode machine + 50Hz loop
    models/             # policy_<model>_<gait>.onnx

1.2 Task Auto-Registration

src/tasks/__init__.py auto-registers tasks via import_packages() when submodules are imported. scripts/train.py imports mjlab.tasks and src.tasks at startup, then the registry (mjlab.tasks.registry) provides list_tasks() / load_env_cfg() / load_rl_cfg() / load_runner_cls(). Task IDs look like Mini2S-Walk-Flat, Mini2S-Turn-Flat, Mini2SW-Walk-Flat, Lite3-Walk-Flat, etc.

Execution flow: main() → tyro parses the first arg to pick the task + remaining args override TrainConfig(env+agent) → launch_training() → run_train(). Single GPU runs directly; multi-GPU is dispatched via torchrunx.

1.3 velocity Task Structure

• Factory make_velocity_env_cfg() (velocity_env_cfg.py): returns ManagerBasedRlEnvCfg that centrally defines sensors, observations, actions, commands, events (domain randomization), rewards, terminations, curriculum, and metrics as the common baseline for all robots/tasks.
• Robot/task overrides config/<robot>_<task>/env_cfgs.py: customize on top of the factory (timestep, decimation, contact sensors, reward weights, per-joint std dicts, command ranges, reset randomization).
• config/<robot>_<task>/rl_cfg.py: defines Actor/Critic network structure and PPO hyperparameters.
• mdp/: reusable implementations of observations, rewards, terminations, curriculum, commands.
• rl/runner.py: extends MjlabOnPolicyRunner; on save() additionally exports ONNX and writes metadata.

1.4 Asset Organization

• Robots: src/assets/robots/<robot>/xmls/<robot>.xml (MJCF) + mesh + *_constants.py (initial pose INIT_STATE, collision config MINI2S_WALK_COLLISION/MINI2S_TURN_COLLISION, actuators). E.g. Mini2S default initial joint angles: thigh=1.0, calf=0.0, hip=0.0, arm_yaw=0, arm_thigh=−1.57, arm_calf=1.35 (rad). Walk task collides only with calf feet; Turn task lets the whole body touch ground (for get-up).
• Reference motions: src/assets/motions/*.npz (tracking tasks).

1.5 Deployment Layers

Real-robot deployment is split by responsibility into a comms library and an RL control layer built on top of it:

• deploy/xgo_lib2/ — comms library (RL-independent, reused directly by ROS2): the XGO serial protocol — frame pack/checksum, the XgoDog client, ~50Hz push-state parsing, and a field registry (feedback_slot_of(motor_id)). This is the only hard dependency on the robot side.
• deploy/xgo_rl/ — RL control layer, internally five layers:
  • [config] model_cfg.py: RobotConfig + device factories (mini2s()/mini2sw()/lite3()) + WALK/TURN constants.
  • [hardware] robot.py: Joint/JointSet/RobotState/Robot — read feedback, write servo angles, gravity LPF.
  • [gait] gait.py: GaitLogic — tick gait phase, map user command → model command.
  • [policy] policy.py: OnnxPolicy — assemble observation → infer → target joint angles.
  • [facade] controller.py: Controller (alias Robot) — mode machine + 50Hz control loop; the single external entry with a thread-safe state snapshot.

Part 2 MDP Design

2.1 Observation Space

Observations are split into actor and critic groups, concatenate_terms=True concatenates into a vector; actor enables noise and corruption (enable_corruption=True), critic does not (more stable training).

actor observation terms (with uniform noise UniformNoise):

critic observation terms = all actor terms + base_lin_vel(3) + foot_height(J) + foot_air_time(J) + foot_contact(J) + foot_contact_forces(3J).

* height_scan exists only in tasks with terrain scanning; the Flat config deletes the height_scan term from both actor and critic (see del cfg.observations[...].terms["height_scan"] in mini2s_flat_walk_env_cfg), so the flat policy input dimension excludes terrain scan. The exported ONNX input dimension matches this trimmed observation set.

phase observation: global_phase = (episode_length_buf * step_dt) % period / period, take (sin, cos); zeroed when standing (command norm < 0.1).

2.2 Action Space

• walk: JointPositionActionCfg(scale=0.25, use_default_offset=True) — output is the joint-angle offset relative to default pose, scaled and added to the default angle.
• turn (get-up): HeldJointPositionActionCfg(hold_duration_s=1.0) — each action is held for 1s to avoid high-frequency jitter.

2.3 Commands

UniformVelocityCommandCfg: velocity command sampled uniformly over [lin_vel_x, lin_vel_y, ang_vel_z, heading]; resampling_time_range=(3.0, 8.0)s periodically resamples; heading_command=True adds heading control; rel_standing_envs=0.05 (~5% of envs stay standing). Curriculum can progressively widen the command range (see 2.4).

2.4 Domain Randomization, Termination & Curriculum

• Events (domain randomization): reset_base (reset pose/velocity), reset_robot_joints (joint offset/velocity), push_robot (interval 5–6s random push for disturbance robustness), foot_friction (startup, all foot geoms share random friction 0.2–3.0), encoder_bias (±0.04 rad), base_com (CoM offset).
• Terminations: time_out (episode timeout), fell_over (bad_orientation, body-vs-up angle > 70°).
• Curriculum: terrain_levels_vel (terrain difficulty advances with performance), commands_vel (progressively widen speed range, e.g. step 0→5000×24 from (±0.5,1.0) to (±1.0,2.0)).

Part 3 Reward Function Design

Rewards are implemented in src/tasks/velocity/mdp/rewards.py, organized into four groups: velocity tracking / posture / gait shaping / penalties. Each term is a RewardTermCfg(func=..., weight=..., params=...); total reward is the weighted sum. The base weights below come from velocity_env_cfg.py; override weights come from mini2s_walk (robot-specific tasks tune further on top).

3.1 Velocity Tracking

Gaussian form exp(-err²/std²): reward → 1 as error → 0; std controls tolerance (smaller = stricter). Override raises weight to 4.0 and tightens std to 0.15, prioritizing tracking.

3.2 Posture

• pose / variable_posture: by command speed in three regimes, per-joint std computes exp(-mean((q−q₀)²/std²)):
  • standing std_standing: hip 0.05 / thigh 0.1 / calf 0.15
  • walking std_walking: hip 0.15 / thigh 0.35 / calf 0.35
  • running std_running: hip 0.15 / thigh 0.5 / calf 0.5
  • arm joints fixed 0.1; thresholds walking_threshold=0.05, running_threshold=0.3. Higher speed allows larger deviation from default pose.
• gravity_gated_variable_posture (turn only): multiplies variable_posture by a "non-upright gate" (projected_gravity_z < −0.75), encouraging correct get-up posture only when not upright.

3.3 Gait Shaping

3.4 Penalties

3.5 walk vs turn Reward Differences

turn (get-up) task key adjustments:

• Remove track_linear_velocity / track_angular_velocity / foot_gait / foot_clearance / feet_slip / soft_landing / stand_still (get-up needs no velocity tracking / specific gait).
• pose becomes gravity_gated_variable_posture, weight 10.0, gravity gate threshold −0.75.
• Add recovery_progress (weight 10.0): linear "uprightness progress" signal (1−z)·0.5 from projected gravity z, guiding rise from lying pose.
• Add joint_vel_l2 (−3e-2) for smooth joint velocity; body_ang_vel/angular_momentum become gravity-gated variants.
• Remove fell_over/illegal_contact terminations (get-up naturally touches ground); clear curriculum.
• Episode 8s, timestep 0.002, decimation 10; reset_base randomizes roll/pitch/yaw fully to create initial lying poses.

Part 4 Policy Network & Training Algorithm Parameters

4.1 Policy / Value Network Structure

From mini2s_walk/rl_cfg.py RslRlModelCfg:

• Three fully-connected layers (512→256→128) + ELU activation: moderate parameter count, sufficient expressiveness.
• obs_normalization=True: maintains running mean/std of observations during training; the normalization stats are baked into the exported ONNX, so deployment only feeds raw (relative-to-default-pose / scaled) observations consistent with training.
• Actor outputs Gaussian mean; standard deviation init_std=1.0 (scalar, shared across actions), adapted by PPO during training.

4.2 PPO Hyperparameter Details

From RslRlPpoAlgorithmCfg + RslRlOnPolicyRunnerCfg:

4.3 Key Parameter Tuning Guide

Reward weights (relative scale matters most)

• High tracking weight (4.0) signals velocity-following as the primary goal; is_terminated=−200 dominates termination decisions, preventing the policy from "learning to fall".
• Gait terms (foot_gait/foot_clearance/feet_slip) tune naturalness: if slipping is obvious, increase foot_slip penalty; if feet lift too high / scrape, adjust foot_clearance target and weight.
• variable_posture per-joint std is a "soft-constraint" knob: smaller std stays closer to default pose (steadier but stiffer), larger allows bigger motion (faster but less stable).

Action distribution & exploration

• init_std=1.0 is a common start; if early training is violently jittery/divergent, lower LR or raise max_grad_norm first; if it converges prematurely to a suboptimal policy, briefly raise init_std or entropy_coef.
• The per-reward std (e.g. track 0.15) controls tracking tolerance, tuned together with the weight.

Learning rate & KL

• schedule="adaptive" + desired_kl=0.01: if measured KL exceeds target, LR drops; below, LR rises. If the loss curve oscillates, check that KL monitoring sits near target; if it never drops, nudge desired_kl.
• clip_param=0.2 is the trust-region width: larger = more aggressive (fast but risky), smaller = more conservative.

Sampling efficiency

• Single-batch sample count = num_steps_per_env(24) × num_envs. num_envs=4096 gives massive parallel samples — key to fast training; lower num_envs if VRAM is tight.
• num_learning_epochs=5 / num_mini_batches=4: more epochs reuse data more but risk overfitting a batch; more mini-batches make each update more stable but slower.

Discount & advantage

• gamma=0.99: near 1 helps long-horizon planning (locomotion needs some look-ahead). lam=0.95 trades bias vs variance; higher leans MC (low bias, high variance), lower leans TD (low variance, high bias).

Part 5 Model Export & Real-Robot Deployment

Deployment environment note: The RL part of this repo is a training framework. "Training" runs on a workstation/server with an NVIDIA GPU (Ubuntu 22.04, Python 3.11) and produces .onnx policy files; "real-robot deployment" runs on the robot dog's onboard compute unit (Ubuntu 26.04 LTS aarch64, Python 3.14.4, onnxruntime CPU inference).

The deployment side is split into two layers with a clean boundary:

Layer	Dir	Responsibility
Comms library	deploy/xgo_lib2/	XGO serial protocol: frame pack/checksum, XgoDog client, ~50Hz push-state parsing, field registry
RL control layer	deploy/xgo_rl/	Built on top of the comms library: read feedback → pick gait → ONNX inference → write joint angles

The comms library is independently reusable — the ROS2 side talks to the dog straight through it (see ../ros_tutorial/tutorial_en.md), with no dependency on the training framework.

5.1 Training → ONNX Export

velocity/rl/runner.py's VelocityOnPolicyRunner.save() exports the .pt then calls export_policy_to_onnx(), and attach_metadata_to_onnx() writes metadata:

• joint_names (all actuated joints, in policy output order)
• action_scale (default 0.25)
• default_joint_pos (rad)
• observation_names (observation term order)

These metadata let the deployment-side OnnxPolicy auto-derive dimensions and mapping without manual inspection.

5.2 Deployment Code Layering

(The directory layout is shown in Part 1.1; responsibilities per layer below.)

Layering discipline: policy never touches hardware/gait; gait does not import policy nor assemble observations; robot only handles bytes and units; mode switching and loop orchestration live only in controller.

OnnxPolicy (policy layer)

• Loads the ONNX session with CPUExecutionProvider.
• Derives obs_dim, num_joints from input/output tensor shapes.
• Reads joint_names / action_scale / default_joint_pos / observation_names from metadata and cross-validates against model_cfg — a mismatch fails fast at startup.
• step(state, ctx): RobotState + GaitContext → a JointSet of target angles (degrees). Internally three steps: assemble obs → infer → + default pose / clamp / × action_scale.
• PolicySpec is the per-gait normalization recipe (scales, gravity source), shipped with the model — not in config.

Observation layout build_obs (strictly consistent with training actor obs):

obs = [ang_vel(3) | projected_gravity(3) | command(3) | phase(2) | joint_pos(J) | joint_vel(J) | prev_action(J)]
       first 11 dims fixed                  + 3 dims per joint (J = number of actuated joints)
obs_dim = 11 + 3*J

5.3 Real-Robot Control Loop

Controller (external alias Robot) runs a background daemon thread at ctrl_hz=50 (period 0.02s); each tick runs five steps:

1. Read feedback robot.read_state(): pull the latest feedback frame from the comms library → RobotState (joint angles/vel, pose, filtered gravity); feedback slots are auto-derived by xgo_lib2.registry.feedback_slot_of(motor_id) — no hand-written remap table needed.
2. Advance mode _advance_mode: _ModeMachine does stand/fall hysteresis on the filtered gravity's grav_z, switching TURN↔WALK.
3. Advance gait _gait_step: per mode pick walk_gait/turn_gait, emit GaitContext(command, phase).
4. Infer policy _infer: pick walk/turn policy per mode, policy.step() → target joint angles.
5. Write + publish robot.set_joint_angles() + _publish_state(): write servos; locked snapshot for any-thread get_state().

Feedback freshness guard: if no new frame arrives for feedback_stale_timeout=0.5s, stop writing; servos hold the last target angle.

Per-gait normalization recipe differences (policy.py / gait.py factories)

Note ang_vel is NOT read from a raw gyro — the protocol feedback frame carries no raw gyro; angular velocity comes from frame-to-frame differencing of the orientation angles (euler2).

Mode FSM (_ModeMachine in controller.py)

• Starts in TURN (treated as get-up/recover until upright is confirmed).
• TURN → WALK: grav_z < stand_z_threshold(−0.75) for stand_count_required=10 consecutive ticks.
• WALK → TURN: grav_z > fall_z_threshold(−0.4) for fall_count_required=5 consecutive ticks.
• Gravity LPF in the hardware layer: grav_filtered = 0.3·raw + 0.7·prev, then normalize (gravity_lpf_alpha=0.3).

Serial: XgoDog default /dev/ttyAMA0, then switches to run baudrate 2_000_000 (~50Hz × 254B feedback needs a high-speed UART); motor_ids order must match the ONNX output order (Mini2S: fl,fr,br,bl,arm).

5.4 Device Config Factory

RobotConfig holds only "deployment/platform/control" facts (default pose, joint order, motor mapping — these are authoritative; the ONNX is only cross-validated); the model normalization recipe is NOT here.

walk and turn must be trained in the same batch (action_scale consistent), otherwise the startup cross-check fails immediately.

Part 6 Practical Commands & Troubleshooting

6.1 Training / Evaluation Commands

# Train (walk)
python scripts/train.py Mini2S-Walk-Flat --env.scene.num-envs 4096
tensorboard --logdir logs/rsl_rl/mini2s_velocity_walk

# Train (get-up)
python scripts/train.py Mini2S-Turn-Flat --env.scene.num-envs 4096

# Common overrides
python scripts/train.py Mini2S-Walk-Flat --agent.max-iterations 10000 --agent.resume
python scripts/train.py <task> --env.scene.num-envs 4096 --gpu-ids all --video

# Evaluate (latest / specific checkpoint)
python scripts/play.py Mini2S-Walk-Flat
python scripts/play.py Mini2S-Walk-Flat --checkpoint-file logs/rsl_rl/mini2s_velocity_walk/2026-xx-xx_xx-xx-xx/model_xx.pt
python scripts/play.py Mini2S-Walk-Flat --agent zero   # MDP sanity check (zero action)

# List tasks / preview terrain
python scripts/list_envs.py --keyword Mini2S
python scripts/visualize_terrain.py

Colab train + TensorBoard: clone repo, pip install -e ., restart runtime; background python scripts/train.py Mini2S-Walk-Flat --env.scene.num-envs 4096 > output.log 2>&1 &, %load_ext tensorboard + %tensorboard --logdir ./logs/rsl_rl/mini2s_velocity_walk.

Real-robot deployment

python deploy/run.py --device mini2s
python deploy/run.py --device mini2sw --port /dev/ttyUSB0
python deploy/run.py --device lite3 --port /dev/ttyAMA0 --motor-ids 11,12,13,...

On start the ONNX auto-prints model info; robot.set_velocity(vx, vy, yaw) sends velocity; Ctrl+C shuts down safely (return to default pose + close serial).

6.2 Common Issues

This tutorial is compiled from the repository code (src/, scripts/, deploy/) and Luwu's official Yuque docs (RL&PPO concepts, URDF, MuJoCo install, mujoco_py, model import). The code is authoritative; for version differences refer to the corresponding setup.py and config source files.