Chapter 6: AI Model Replacement

Luwu-OS vision, speech and emotion recognition all rely on ONNX models. This chapter lists the existing models with a “which app uses which model” mapping, then covers the generic inference API and replacement steps, finishing with a troubleshooting table — so you can safely swap in your own model and confirm it works.

In this chapter:

Section What it covers Difficulty
6.1 ONNX Model List Existing model list + model→app mapping (incl. 6.1.1) ★☆☆
6.2 Generic Inference API onnxruntime / FaceDetectorYN / MediaPipe wrapper calls ★★★
6.3 Replacing the Face Detection Model Swap-in four steps + restart ★★☆
6.4 Replacing the Gesture Recognition Model Swap-in + adapt the wrapper tensor signature ★★★
6.5 Common Problems After Replacing Troubleshooting table (tensor mismatch, no detection, low fps) ★★☆
6.6 Useful Query Commands Print model I/O signature, check disk usage ★☆☆

🧭 Reading path: To only “swap a ready model”, see 6.3/6.4; for model-level development, first check the 6.1.1 mapping and 6.2 inference API. Before any swap, run the command in 6.6 to save the original signature.

Prerequisites:

  • Done Chapter 1, you can SSH into the robot;
  • A replacement ONNX model file (keep its input size/preprocessing close to the existing model to save effort);
  • Know which app and model you’re replacing (see 6.1.1).

6.1 ONNX Model List

Model files in /opt/luwu-os/model/ (names may vary across firmware versions; these are the common/public names — always trust the actual directory on the robot):

Model File Purpose Source
face_detection_yunet_2023mar.onnx Face detection YuNet (OpenCV)
palm_detection_mediapipe_2023feb.onnx Palm detection MediaPipe
handpose_estimation_mediapipe_2023feb.onnx Hand 21-keypoint pose MediaPipe
person_detection_mediapipe_2023mar.onnx Person detection MediaPipe
pose_estimation_mediapipe_2023mar.onnx Human pose estimation (33 keypoints) MediaPipe
emotion.onnx Emotion recognition (AI chat) Custom training
gender_age.onnx Gender / age estimation Custom training
yolo_coco.onnx Generic object detection (COCO 80 classes) YOLO
embedding_model.onnx / melspectrogram.onnx Speech / audio embedding & spectrogram (names vary by version) Custom training
Wake-word model (hi_luka.onnx / hi_luka_qwen.onnx / hi_luwu.onnx / xiaolu_classmate.onnx / xiaolu_tongxue.onnx etc.) Wake-word detection (English "Hi Luka" / Chinese "Xiao Lu Tong Xue", chosen by the system language) Custom training
msyh.ttc Chinese font (UI rendering, not a model) Microsoft YaHei
mp_handpose.py / mp_palmdet.py / mp_persondet.py / mp_pose.py Python wrapper classes for the MediaPipe models (not models; MPPalmDet/MPHandPose/MPPersonDet/MPPose) Built-in

⚠️ The custom-trained models above (emotion / gender-age / audio-embedding) may be named differently across firmware versions; ls /opt/luwu-os/model/ is the authoritative list. The replacement logic stays the same—just use the actual file names in that directory.

6.1.1 Model → App Mapping

The ONNX models are not loaded centrally; each is loaded on demand by a specific app at startup. Before replacing a model, confirm which app references it (variable names and paths below follow the actual source; they may change across versions):

Model App that uses it Loading location
face_detection_yunet_2023mar.onnx Face follow apps/face_follow/main.py (FACE_MODEL_PATH)
palm_detection_* / handpose_estimation_* Gesture command apps/gesture/main.py (PALM_MODEL / HAND_MODEL)
person_detection_* Person follow apps/person_follow/main.py
pose_estimation_* Pose recognition related demo apps
emotion.onnx Old AI chat (emotion recognition) apps/ai/emotion_manager.py
yolo_coco.onnx Generic object recognition / demos related apps
Speech / audio embedding & spectrogram models Wake-word / custom ASR apps/ai/wakeword_manager.py (MEL_ONNX / EMBEDDING_ONNX), reused by the new AI Chat Pro
Wake-word model Wake-word activation ("Hi Luka" / "Xiao Lu Tong Xue") apps/ai/wakeword_manager.py (EN_MODEL_PATH=hi_luka_qwen.onnx, MODEL_PATH=xiaolu_classmate.onnx), reused by the new AI Chat Pro
gender_age.onnx Feature display related demo apps

💡 All ONNX models run through onnxruntime. The MediaPipe family (palm/hand/person/pose) is additionally wrapped by the mp_*.py classes, which handle input preprocessing and output post-processing. The wrapper classes are imported with sys.path.insert(0, model dir) then from mp_palmdet import MPPalmDet (see apps/gesture/main.py).

How to quickly find which app references a model: instead of digging through source one by one, grep the file name under /opt/luwu-os (limit to .py):

grep -rl "face_detection_yunet" /opt/luwu-os/apps --include="*.py"
grep -rl "palm_detection" /opt/luwu-os/apps --include="*.py"

💡 The result lists the app paths containing that file name / variable, i.e. its loading location; most apps also define a model-path variable (e.g. FACE_MODEL_PATH) — grep that variable name to pinpoint the exact assignment line.

6.2 Generic Inference API (onnxruntime wrapper)

Every app loads an ONNX model the same way: create an onnxruntime.InferenceSession, preprocess the frame into the model input, then take the output. For face detection the real code in apps/face_follow/main.py uses OpenCV's FaceDetectorYN:

import cv2

# Load the YuNet face detector
face_detector = cv2.FaceDetectorYN.create(
    model_path.replace("model/", "/opt/luwu-os/model/"),  # absolute path
    "",                        # config (YuNet has no config file)
    (320, 240),                # input size (width, height)
    scoreThreshold=0.7,        # confidence threshold
    nmsThreshold=0.3,          # NMS threshold
    topK=5000,                 # max faces to return
)

# Per-frame inference: convert BGR to RGB first (YuNet expects RGB)
rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
# setInputSize lets you pass the size per frame; the input is scaled proportionally
face_detector.setInputSize((frame.shape[1], frame.shape[0]))
ok, faces = face_detector.detect(rgb)
# faces: Nx15 array; the first 4 cols are the bbox, the next 10 cols are the (x,y) of the right-eye/left-eye/nose/right-mouth/left-mouth keypoints (2 cols each), and the last col is the confidence

The MediaPipe pipeline uses the wrapper classes, which also infer via onnxruntime (see apps/gesture/main.py):

import sys
sys.path.insert(0, "/opt/luwu-os/model")  # wrappers live in the model/ dir

from mp_palmdet import MPPalmDet
from mp_handpose import MPHandPose

palm = MPPalmDet()   # reads model/palm_detection_mediapipe_2023feb.onnx internally
hand = MPHandPose()  # reads model/handpose_estimation_mediapipe_2023feb.onnx internally

# Per-frame inference: preprocess (resize/normalize), infer, then post-process (decode keypoints/box)
palm_out = palm.detect(frame)
hand_pts = hand.estimate(frame, palm_out)   # outputs 21 keypoints

💡 palm.detect(...) / hand.estimate(...) above are illustrative calls; the wrapper methods/args/signatures follow the actual mp_*.py. If unsure, run cat /opt/luwu-os/model/mp_handpose.py to inspect the real interface.

The general onnxruntime flow (reference when replacing a custom model that is neither OpenCV nor a MediaPipe wrapper): ```python import onnxruntime as ort, numpy as np sess = ort.InferenceSession("/opt/luwu-os/model/your_model.onnx") inp = sess.get_inputs()[0] # input signature: name/shape/type

image preprocessing: resize + normalize + CHW → NCHW + to float32

x = preprocess(frame) # shape=(1, C, H, W) out = sess.run(None, {inp.name: x})[0] # take the first output out = postprocess(out) # decode per the model's output definition ```

6.3 Replacing the Face Detection Model

  1. Place the model: put the new ONNX file into /opt/luwu-os/model/
  2. Change the path: point the FACE_MODEL_PATH variable in apps/face_follow/main.py to the new file (currently model/face_detection_yunet_2023mar.onnx)
  3. Adapt the parameters: unless the new model output is identical to YuNet, also adjust the cv2.FaceDetectorYN.create() parameters — input size (320, 240), scoreThreshold, nmsThreshold, topK all need retuning for the new model. If you switch to a non-YuNet architecture (e.g. another detection network), use the generic onnxruntime approach in 6.2 instead
  4. Restart the app: face-follow is a persistent process; restart it so the new model is reloaded
# Restart the face-follow app (launched by the launcher)
echo "apps/face_follow/main.py" > /tmp/luwu_run.fifo

⚠️ To verify, temporarily print the number of detection results to the log/terminal, make sure faces are detected, then tune the thresholds step by step.

6.4 Replacing the Gesture Recognition Model

  1. Replace the files: replace palm_detection_*.onnx and handpose_estimation_*.onnx under /opt/luwu-os/model/
  2. Change the paths: update the PALM_MODEL / HAND_MODEL variables in apps/gesture/main.py
  3. Adapt the wrapper classes: MPPalmDet / MPHandPose do preprocessing and post-processing based on a fixed input/output tensor signature. If the new model's input shape, normalization or output tensor names differ, update the wrappers (mp_palmdet.py / mp_handpose.py) accordingly, otherwise you'll get "tensor name not found" or a garbled decode
  4. Restart the app: restart with echo "apps/gesture/main.py" > /tmp/luwu_run.fifo

6.5 Common Problems After Replacing

If detection misbehaves after replacement, check in this order (usually a "tensor signature mismatch"):

Symptom Possible cause Fix
Crashes at startup with Invalid graph / No op named ... Model input/output tensor names differ from the hard-coded wrapper Open the wrapper, print the actual signature via sess.get_inputs()/get_outputs(), and align them
Loads but never detects the target Wrong input preprocessing (size / normalization / BGR↔RGB / channel order) Check each preprocessing step against the original model
Detects but boxes are offset Output post-processing decode formula doesn't match the model Rewrite the decode per the new model's output (scale back to the original size)
Frame rate drops noticeably Model too large / input size too large / not using setInputSize scaling Lower the input resolution, use a smaller model, or warm up the session
Only works for specific targets Swapped to a model of a different class Confirm the app logic matches the model output classes (e.g. yolo_coco 80 classes)

Performance tip: The robot main board is a Raspberry Pi 5 (CM5); prefer a quantized ONNX (e.g. int8) or a smaller model. Keep inference to 1–2 runs per frame at most, otherwise the main UI frame rate will be dragged down. You can also cache a reused session in a global variable instead of rebuilding it each frame.

6.6 Useful Query Commands

# Print the input/output signature of an ONNX model (replace the path first)
python3 - <<'EOS'
import onnxruntime as ort, glob, os
for p in glob.glob('/opt/luwu-os/model/*.onnx'):
    s = ort.InferenceSession(p)
    ins = [(i.name, i.shape, i.type) for i in s.get_inputs()]
    outs = [(o.name, o.shape) for o in s.get_outputs()]
    print(os.path.basename(p), ins, outs)
EOS

# Check the size and files of the model directory
du -sh /opt/luwu-os/model && ls -la /opt/luwu-os/model

6.7 Checklist Before & After Replacing

Run 6.6 first to save a baseline, then check off the items below as you go:

Stage Check
Before ① record the original model's input/output signature (6.6); ② back up the original file (cp); ③ know the new model's preprocessing (size / normalization / BGR-RGB / channel order)
After ① the app starts without error; ② it can stably detect the target; ③ frame rate is acceptable; ④ results are positioned correctly
On problems go through the 6.5 table in order; suspect tensor signature → preprocessing → decode → performance first

📌 If you just want to try a custom model without touching the source yet, drop an ONNX with the same input/output signature into model/ and only change the path — you don't write code, and you can validate the whole pipeline with a minimal change.