Chapter 6: AI Model Replacement

Luwu-OS vision, speech and emotion recognition all rely on ONNX models. This chapter lists the existing models with a “which app uses which model” mapping, then covers the generic inference API and replacement steps, finishing with a troubleshooting table — so you can safely swap in your own model and confirm it works.

In this chapter:

🧭 Reading path: To only “swap a ready model”, see 6.3/6.4; for model-level development, first check the 6.1.1 mapping and 6.2 inference API. Before any swap, run the command in 6.6 to save the original signature.

Prerequisites:

• Done Chapter 1, you can SSH into the robot;
• A replacement ONNX model file (keep its input size/preprocessing close to the existing model to save effort);
• Know which app and model you’re replacing (see 6.1.1).

6.1 ONNX Model List

Model files in /opt/luwu-os/model/ (names may vary across firmware versions; these are the common/public names — always trust the actual directory on the robot):

⚠️ The custom-trained models above (emotion / gender-age / audio-embedding) may be named differently across firmware versions; ls /opt/luwu-os/model/ is the authoritative list. The replacement logic stays the same—just use the actual file names in that directory.

6.1.1 Model → App Mapping

The ONNX models are not loaded centrally; each is loaded on demand by a specific app at startup. Before replacing a model, confirm which app references it (variable names and paths below follow the actual source; they may change across versions):

💡 All ONNX models run through onnxruntime. The MediaPipe family (palm/hand/person/pose) is additionally wrapped by the mp_*.py classes, which handle input preprocessing and output post-processing. The wrapper classes are imported with sys.path.insert(0, model dir) then from mp_palmdet import MPPalmDet (see apps/gesture/main.py).

How to quickly find which app references a model: instead of digging through source one by one, grep the file name under /opt/luwu-os (limit to .py):

grep -rl "face_detection_yunet" /opt/luwu-os/apps --include="*.py"
grep -rl "palm_detection" /opt/luwu-os/apps --include="*.py"

💡 The result lists the app paths containing that file name / variable, i.e. its loading location; most apps also define a model-path variable (e.g. FACE_MODEL_PATH) — grep that variable name to pinpoint the exact assignment line.

6.2 Generic Inference API (onnxruntime wrapper)

Every app loads an ONNX model the same way: create an onnxruntime.InferenceSession, preprocess the frame into the model input, then take the output. For face detection the real code in apps/face_follow/main.py uses OpenCV's FaceDetectorYN:

import cv2

# Load the YuNet face detector
face_detector = cv2.FaceDetectorYN.create(
    model_path.replace("model/", "/opt/luwu-os/model/"),  # absolute path
    "",                        # config (YuNet has no config file)
    (320, 240),                # input size (width, height)
    scoreThreshold=0.7,        # confidence threshold
    nmsThreshold=0.3,          # NMS threshold
    topK=5000,                 # max faces to return
)

# Per-frame inference: convert BGR to RGB first (YuNet expects RGB)
rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
# setInputSize lets you pass the size per frame; the input is scaled proportionally
face_detector.setInputSize((frame.shape[1], frame.shape[0]))
ok, faces = face_detector.detect(rgb)
# faces: Nx15 array; the first 4 cols are the bbox, the next 10 cols are the (x,y) of the right-eye/left-eye/nose/right-mouth/left-mouth keypoints (2 cols each), and the last col is the confidence

The MediaPipe pipeline uses the wrapper classes, which also infer via onnxruntime (see apps/gesture/main.py):

import sys
sys.path.insert(0, "/opt/luwu-os/model")  # wrappers live in the model/ dir

from mp_palmdet import MPPalmDet
from mp_handpose import MPHandPose

palm = MPPalmDet()   # reads model/palm_detection_mediapipe_2023feb.onnx internally
hand = MPHandPose()  # reads model/handpose_estimation_mediapipe_2023feb.onnx internally

# Per-frame inference: preprocess (resize/normalize), infer, then post-process (decode keypoints/box)
palm_out = palm.detect(frame)
hand_pts = hand.estimate(frame, palm_out)   # outputs 21 keypoints

💡 palm.detect(...) / hand.estimate(...) above are illustrative calls; the wrapper methods/args/signatures follow the actual mp_*.py. If unsure, run cat /opt/luwu-os/model/mp_handpose.py to inspect the real interface.

The general onnxruntime flow (reference when replacing a custom model that is neither OpenCV nor a MediaPipe wrapper): ```python import onnxruntime as ort, numpy as np sess = ort.InferenceSession("/opt/luwu-os/model/your_model.onnx") inp = sess.get_inputs()[0] # input signature: name/shape/type

image preprocessing: resize + normalize + CHW → NCHW + to float32

x = preprocess(frame) # shape=(1, C, H, W) out = sess.run(None, {inp.name: x})[0] # take the first output out = postprocess(out) # decode per the model's output definition ```

6.3 Replacing the Face Detection Model

1. Place the model: put the new ONNX file into /opt/luwu-os/model/
2. Change the path: point the FACE_MODEL_PATH variable in apps/face_follow/main.py to the new file (currently model/face_detection_yunet_2023mar.onnx)
3. Adapt the parameters: unless the new model output is identical to YuNet, also adjust the cv2.FaceDetectorYN.create() parameters — input size (320, 240), scoreThreshold, nmsThreshold, topK all need retuning for the new model. If you switch to a non-YuNet architecture (e.g. another detection network), use the generic onnxruntime approach in 6.2 instead
4. Restart the app: face-follow is a persistent process; restart it so the new model is reloaded

# Restart the face-follow app (launched by the launcher)
echo "apps/face_follow/main.py" > /tmp/luwu_run.fifo

⚠️ To verify, temporarily print the number of detection results to the log/terminal, make sure faces are detected, then tune the thresholds step by step.

6.4 Replacing the Gesture Recognition Model

1. Replace the files: replace palm_detection_*.onnx and handpose_estimation_*.onnx under /opt/luwu-os/model/
2. Change the paths: update the PALM_MODEL / HAND_MODEL variables in apps/gesture/main.py
3. Adapt the wrapper classes: MPPalmDet / MPHandPose do preprocessing and post-processing based on a fixed input/output tensor signature. If the new model's input shape, normalization or output tensor names differ, update the wrappers (mp_palmdet.py / mp_handpose.py) accordingly, otherwise you'll get "tensor name not found" or a garbled decode
4. Restart the app: restart with echo "apps/gesture/main.py" > /tmp/luwu_run.fifo

6.5 Common Problems After Replacing

If detection misbehaves after replacement, check in this order (usually a "tensor signature mismatch"):

Performance tip: The robot main board is a Raspberry Pi 5 (CM5); prefer a quantized ONNX (e.g. int8) or a smaller model. Keep inference to 1–2 runs per frame at most, otherwise the main UI frame rate will be dragged down. You can also cache a reused session in a global variable instead of rebuilding it each frame.

6.6 Useful Query Commands

# Print the input/output signature of an ONNX model (replace the path first)
python3 - <<'EOS'
import onnxruntime as ort, glob, os
for p in glob.glob('/opt/luwu-os/model/*.onnx'):
    s = ort.InferenceSession(p)
    ins = [(i.name, i.shape, i.type) for i in s.get_inputs()]
    outs = [(o.name, o.shape) for o in s.get_outputs()]
    print(os.path.basename(p), ins, outs)
EOS

# Check the size and files of the model directory
du -sh /opt/luwu-os/model && ls -la /opt/luwu-os/model

6.7 Checklist Before & After Replacing

Run 6.6 first to save a baseline, then check off the items below as you go:

📌 If you just want to try a custom model without touching the source yet, drop an ONNX with the same input/output signature into model/ and only change the path — you don't write code, and you can validate the whole pipeline with a minimal change.