# MB-001 — VisionProTeleop deep dive (source-verified) + OpenTeleVision comparison

**Date:** 2026-11-03 · **Idea:** MB-001 (remote 2-axis pan-tilt rig, two matched UVC webcams, Raspberry Pi 5, Wi-Fi 6 headset link)
**Method:** both repos cloned read-only to the box and read from source; aiortc 1.15.0 sdist unpacked and read; published benchmark JSON read directly. Nothing was run on the user's Mac; nothing bought; nothing sent/posted. Nothing was run on a Pi or a Vision Pro, so every performance figure for *this rig* is "not measured".

Sources (local clones, box):
- `VisionProTeleop` @ `4c549905c2a8b214d79f7cd88e535101a1ce32af` (2025-12-28) — https://github.com/Improbable-AI/VisionProTeleop — MIT
- `TeleVision` @ `e6e25afdb16c1b326b5bf37bd0ae79919bf79f26` (2024-09-26) — https://github.com/OpenTeleVision/TeleVision — Apache-2.0
- Paths below are relative to `/workspace/makerbot/research/repos/<repo>/`. `streamer.py` = `avp_stream/streamer.py`.

Legend: **[V]** verified in source/doc (file:line) · **[D]** derived by me from verified source (not run) · **[?]** unknown / not published.

---

## 0. What VisionProTeleop is

Three parts (README top, `setup.py`): (a) a visionOS **"Tracking Streamer"** app (App Store; source in `Tracking Streamer/`), (b) the Python `avp_stream` library (`VisionProStreamer`; PyPI 2.51 dated 2025-12-28, repo `setup.py` says 2.50), (c) an iOS "Tracking Manager". Python side runs on the *robot/computer* side — i.e. on the Pi in our rig; the headset side is the unmodified App Store app. **[V]**

Two network modes (README L137–178): same-LAN by headset IP, or **External Network Mode** with a room code (Cloudflare signaling worker + TURN, default `wss://visionpro-signaling.parkyh9492.workers.dev/ws`, `streamer.py` L901). README example for a camera: `configure_video(device="/dev/video0", format="v4l2", size="1280x720", fps=30)` (L168–170). **[V]** Relevant to our "far away" rig: the remote mode is built in, but the README warns of slightly higher latency via relay (L175–178), and we'd depend on a third-party-hosted signaling/TURN (may be rate-limited later, per README).

---

## 1. How SBS stereo video reaches the headset

| Item | Finding | Cite |
|---|---|---|
| Stack | **WebRTC.** Python: `aiortc` (+PyAV). visionOS: LiveKit's WebRTC fork (`LKRTC*` types). | `streamer.py` L671–827, L4238–4654; `Tracking Streamer/WebRTCClient.swift` L72–73 (factories), L410–475 (`connectToServer`) |
| Who offers | Python is the **offerer**/server; headset connects to it. Signaling is a JSON-line SDP offer/answer over a **plain TCP socket on :9999** (local mode). So the Pi must accept inbound TCP :9999 from the headset. An HTTP `/webrtc_info` on :8888 also exists. | `streamer.py` L4625 (listen 9999), L4536–4558 (offer/answer) |
| Encoder (Python side) | aiortc's built-in **software** encoders: `libx264` (Baseline, level 3.1, `tune=zerolatency`, **no `preset` set → libx264 default "medium"**) or `libvpx` VP8. No hardware encoder path in the repo. | aiortc 1.15.0 `codecs/h264.py` L20–24, L268–282; `codecs/vpx.py` L19–23 |
| Codec choice | aiortc advertises **VP8 first, then H264**. The repo does **not** force a codec, so the negotiated one depends on the Vision Pro's SDP answer. **[?] negotiated codec is not visible from source** — must be read from logs on first connection. | aiortc `codecs/__init__.py` L95–110; `streamer.py` L4511–4558 |
| Bitrate | aiortc caps: **H.264 max 3 Mbps (default 1 Mbps), VP8 max 1.5 Mbps (default 0.5)**, driven by receiver REMB. The repo inserts `b=AS:15000` (15 Mbps) into the SDP, but aiortc's encoder setter still clamps to the module `MAX_BITRATE`. Whether the headset sends REMB at all is **[?]**. | aiortc `h264.py` L313, `vpx.py` L264, `rtcrtpsender.py` L284–290; `streamer.py` L4511–4516 |
| fps | `MAX_FRAME_RATE = 30` inside both aiortc encoders; the repo's synthetic track is paced to the requested `fps` by a `MediaClock`, but encoder timebase is 30. README `configure_video` table allows higher values; treat **30 fps as the realistic cap** with stock aiortc. **[D]** | aiortc `h264.py` L24, L275–276; `streamer.py` L416–550 (MediaClock), L788 |
| Resolution | Arbitrary; README: "Wireless ≤720p <100 ms; wired stereo 4K ~50 ms". Benchmarked up to 7680×2160 SBS. | README L661–672; `docs/benchmark.md` |
| **SBS handling** | **`stereo=True` is metadata only.** It sets `stereoVideo` in the signaling message (L1868–1883) and the headset app interprets the frame as `[left | right]`. **The library never composites two sources — you must hand it one already-side-by-side frame** (width = 2 × eye width). | `streamer.py` L1868–1889, L4319–4351; `docs/benchmark.md` (stereo sizes are 2W×H) |
| Headset rendering | App receives the I420 frame, converts to BGRA on the CPU, then **CIImage → CGImage → TextureResource every frame**, splits it in `splitSideBySideImage`, and feeds left/right halves to a ShaderGraph material (`stereo` material) on a **flat video plane** that is head-anchored by default (optional "fixed to world"). | `ImmersiveView.swift` L288–331, L851–893, L903, L1045–1140; plane L133–175, L460–483 |
| Transport | Video over RTP/SRTP via WebRTC (UDP via ICE; TURN fallback in remote mode). | `WebRTCClient.swift` L91–99 |

**Consequence for our rig:** one process on the Pi must (1) grab two UVC frames, (2) stitch them, (3) software-encode a wide frame with libx264 at ≤3 Mbps (unless patched), at 30 fps. Compositing (`np.hstack`) is cheap; the **BGR→yuv420p conversion + libx264 encode on the Pi 5 CPU is the load**. Raspberry Pi's white paper (cited in ideas.md, *not re-verified in this pass*) says 1080p30 low-latency x264 uses ~60–90% of one core — but that was with their settings; **aiortc runs x264 at default "medium" preset with zerolatency** so real cost is **[?]** and must be measured.

---

## 2. How head and hand tracking come back

**Two backends** (`ht_backend` ctor arg):

1. **gRPC (default).** The *headset hosts* a gRPC server on **:12345**; Python is the **client** and calls `StreamHandUpdates` (server-streaming, plaintext HTTP/2). Headset loop: ~5 ms sleep ⇒ **≈200 Hz** send attempts, `bufferingNewest(1)` (drops stale). Python discovery: it sends a request whose `Head.m00 = 888` marker and encodes its own IP + library version in other matrix cells. **[V]** `Tracking Streamer/GRPCServer.swift` L12, L69, L107, L205–227; `streamer.py` L2056–2141.
2. **WebRTC data channel `"hand-tracking"`** (`ht_backend="webrtc"`). In local mode it's **unordered, `maxRetransmits=0`** (best for stale-data-hostile pose); in signaling/remote mode it's **ordered**. Swift sender loop is **2 ms** (`WebRTCClient.swift` L30, L477–511); Python side `streamer.py` L4457–4463 (local, unordered), L1778 (signaling, ordered), handlers L1298–1333. **[V]**

**Message** (`avp_stream/grpc_msg/handtracking.proto`): protobuf `HandUpdate { left_hand, right_hand, Head }`. Each pose is a `Matrix4x4` message with 16 float fields `m00…m33` (**a full 4×4 transform, not a quaternion**). Hands add wrist matrix + 25–27 joint matrices (+ pinch). **No timestamp field** — so you cannot compute one-way pose age from the message. **[V]**

**Sample rates at the source** (`🥽AppModel.swift`): head via ARKit `WorldTrackingProvider.queryDeviceAnchor` polled at **120 Hz** (L573 comment, L613–636); hands via predictive `handAnchors(at:)` at 120 Hz (L639–652). README L485–541 documents a hand "prediction offset" option (0/5/33/100 ms, default 5 ms); that applies to hands, **I did not find the head being predicted** (**[?]** whether `queryDeviceAnchor` is called with a predicted time — read L613–636 again before relying on it). End-to-end delivered head-pose rate on Wi-Fi is **[?]** (`avp_stream/benchmark_rate.py` exists, unread).

**Coordinate frames:** ARKit/visionOS is Y-up. Python applies `YUP2ZUP` (`streamer.py` L71) then **`rotate_head(R, -90°)`** — `R @ Rx(-90°)` (`avp_stream/utils/grpc_utils.py` L15–26, applied at `streamer.py` L1056–1110) — so head returns in the Z-up world frame. The world origin is set **at app launch**; **long-press the Digital Crown to recenter** (`assets/short_paper_new.pdf`, README L642 axis image). The library does **no filtering/smoothing** (paper). Paper also warns about moving vehicles (tracking assumes a static world) — irrelevant here since the *wearer* is static. **[V]**

Python access: `s.get_latest()["head"]` → `(1,4,4)`, or `.head` → `(4,4)` (`streamer.py` L2143–2175, L335). `get_latest(use_cache=True, cache_ms=10)` is available.

---

## 3. Hardware used by the project

- **Headset:** Apple Vision Pro (visionOS). **Dev/benchmark client:** **MacBook Pro M2**, wired (USB-C developer strap) or wireless Wi-Fi 6 (`docs/benchmark.md`). **[V]**
- **Cameras:** the *library* is camera-agnostic. Website/README claim "Arbitrary camera setup via UVC" (`website/index.html`). **[V]** The example scripts use a Mac webcam (`examples/06_stream_from_camera.py`, avfoundation) or synthetic/simulated frames. **The authors' physical teleop camera rigs are not specified in the repo → [?]**.
- **Robot:** none is bundled. `examples/15_franka_visionpro_teleop.py` is a **Franka sim in Isaac Lab**; other examples are MuJoCo/viewers. **[V]** No pan-tilt/gimbal hardware or servo code exists in this repo. **[V]** (grep of `avp_stream/` & `examples/` for servo/gimbal/dynamixel found nothing relevant.)
- **Compute:** a Mac for benchmarking; docs mention general PC/Linux/Mac. PyPI **aarch64 wheels exist for `av` 19.0.2, `opencv-python`, `grpcio`, `mujoco`** (checked on PyPI 2026-11-03), so a Pi 5 install is plausible; **[?] untested here**, especially PyAV v4l2/x264 build features on Raspberry Pi OS.

---

## 4. Measured latency — what's published

Source: `docs/benchmark.md` + `benchmarks/{wired,wireless}-{mono,stereo}.json` (1000 trials each, 30 fps, client = MacBook Pro M2, Vision Pro; synthetic 8×9 binary-grid frame; round-trip = Python sends frame → headset decodes → **detects the pattern on the decoded frame (`ImmersiveView.swift` L941 `detectBenchmarkPayload`) → returns result over the tracking channel**). **It is NOT glass-to-glass and NOT head-to-photon: it excludes camera capture, texture upload and display scan-out, and uses a trivially compressible image, so real camera video will be worse.** Means, ms:

| Config | resolution (SBS = 2W×H) → mean ms |
|---|---|
| wired mono | 426×240 **17.3** · 640×360 **17.5** · 1280×720 **20.5** · 1920×1080 **24.5** · 3840×2160 **51.0** |
| wired stereo | 852×240 **21.8** · 1280×360 **22.3** · **2560×720 19.2** · 3840×1080 **23.0** · 7680×2160 **61.4** |
| wireless mono | 426×240 **57.2** · 640×360 **55.0** · 1280×720 **64.3** (p50 56, p95 123) · 1920×1080 **62.7** (p50 49, p95 121, max 663) · 3840×2160 **97.2** (p50 92, p95 151) |
| wireless stereo | 852×240 **54.4** · 1280×360 **57.7** · **2560×720 59.9** (p50 52, p95 117) · 3840×1080 **56.6** (p50 64, p95 113) · 7680×2160 **117.2** (p50 84, p95 238, max 932) |

README headline: "Wireless, ≤720p <100 ms; Wired stereo 4K ~50 ms stable" (L661–672). `docs/source/video_streaming.rst` has a generic "50–150 ms" and an outdated API (`start_streaming`) — I treat README/benchmark.md as authoritative.

**Not published → [?]:** glass-to-glass with real cameras; head-pose-to-servo latency; anything on Pi/UVC/x264-Pi; one-way pose age (no timestamp in message).
Rough planning number only **[D]**: our wireless round trip ~55–60 ms at 2560×720 from the benchmark, **plus** UVC capture (~1–2 frames), Pi x264 encode, servo travel + mechanical settle — total motion-to-photon likely well over 100 ms; measure with a phone slow-mo camera.

---

## 5. Exactly what to change for this rig

Principle: **keep the stock App Store Tracking Streamer app; run a modified/driver script on the Pi using `avp_stream` as a library.** Avoid forking Swift. Files below are in `avp_stream/`.

### 5.1 Two-UVC capture → SBS frame (replace `MediaPlayer`)
- Hook: use **`configure_video(device=None, size="<2W>x<H>", fps=30, stereo=True)`** (`streamer.py` L2515) so `ProcessedVideoTrack` takes the **synthetic path** (L785–825), then feed frames with **`update_frame(frame)`** (L2486) from your own thread. Rationale **[V]**:
  - The synthetic path is clock-paced, copies only the latest `user_frame` each tick, so no queue → no latency build-up (duplicates a frame if capture is slower).
  - The camera path (`device="/dev/videoX"`) uses aiortc `MediaPlayer` → PyAV, which for live v4l2 input uses an **unbounded asyncio queue** (aiortc `contrib/media.py`, from my read) so any encoder stall accumulates delay; and it is **single device** — no stereo.
  - `register_frame_callback` (L2419) runs your function **synchronously on the asyncio loop inside `recv()`** (L759–763 / L806–814) — heavy work there stalls signaling/RTP. Do capture/stitch in your own threads; keep callbacks empty.
- Capture (own threads, **OpenCV V4L2 `cv2.VideoCapture(idx, cv2.CAP_V4L2)`** with `CAP_PROP_FOURCC=MJPG`, W×H, 30 fps, `CAP_PROP_BUFFERSIZE=1`). MJPEG is required: the Quanrui USB48MP01 family is USB 2.0 UVC with MJPEG/YUY2 (ideas.md); raw YUYV at 1080p over USB2 won't fit two cameras. MJPEG decode costs Pi CPU (libjpeg-turbo in OpenCV) — **[?] cost at 2×1080p30; start at 1280×720 per eye** (SBS 2560×720, which is a point benchmarked at 19 ms wired / 60 ms wireless mean).
- Sync: two free-running USB cams are **not hardware-synced**. Grab both with `cap.grab()` back-to-back then `cap.retrieve()` (OpenCV recommended pattern for multi-cam) and use the latest pair; residual skew ≤ one frame period (33 ms) **[D]**. Matched exposure/white balance: set manual exposure/WB identically via `v4l2-ctl` on both.
- Stitch: `sbs = np.hstack((left, right))` (BGR24, contiguous) → `streamer.update_frame(sbs)`. Left cam must be the **left** eye (swapped = pseudo-stereo).
- Alternative: GStreamer pipeline (`v4l2src ! jpegdec ! videoscale ! compositor`) → appsink → numpy is lighter than OpenCV on some systems, but adds a dependency; only if OpenCV is the bottleneck **[?]**.

### 5.2 Encoding on the Pi 5 (no hardware encoder)
- Stock path = aiortc **software** x264/VP8 (§1). Quick wins **[D]** (all untested on Pi):
  1. **Lower the pixel count first** (SBS 2560×720 = 1.84 MP, vs 1080p-per-eye 4.1 MP). Latency benchmark shows wireless 1280×720×2 ≈ wired-class cost on the headset side; the Pi encode is the unknown.
  2. Raise caps: set `aiortc.codecs.h264.MAX_BITRATE` (and `DEFAULT_BITRATE`) at import time, e.g. 8–12 Mbps for 2560×720, otherwise REMB/default stays ≤3 Mbps → blocky SBS. Note aiortc **recreates the encoder (new keyframe) when target changes >10%** (h264.py L255–259) — avoid REMB thrash by pinning default = max.
  3. **x264 preset:** aiortc doesn't set one (→ "medium"). Patch `H264Encoder._encode_frame` options (h264.py L278–281) to add `"preset": "ultrafast"` (+ keep `tune=zerolatency`) — this is a monkeypatch/fork of aiortc, not in the repo.
  4. Keep 30 fps (aiortc cap).
- Alternatives (all require work beyond stock): (a) **MJPEG-passthrough is not possible** — the headset app expects a WebRTC video track; (b) use a board **with a hardware H.264 encoder** (e.g. Jetson/RK3588 class) and feed encoded frames — aiortc has `H264Encoder.pack(packet)` (h264.py L298) that accepts pre-encoded `av.Packet`s, a plausible hook for hardware-encoded input **[D]**; (c) keep Pi 5 and accept x264 CPU cost.
- Level flag: aiortc hard-codes Baseline L3.1 (≈0.92 MP nominal) while the repo benchmarks went to 7680×2160 and worked, so the headset decoder evidently tolerates it **[V by benchmark]**, but it's a flag to check if SBS >1280×720 stutters.

### 5.3 Head pose → servo (quaternion → yaw/pitch → pan/tilt)
Library gives a **4×4 matrix, not a quaternion** (§2). Skip quaternions; use the forward vector:

```python
# [D] DERIVED, NOT RUN. Verify axes by printing yaw/pitch while nodding/turning.
import numpy as np
H = streamer.get_latest()["head"][0]        # 4x4, Z-up world, after rotate_head
f = H[:3, 1]                                 # assumed "forward" column in this frame (verify!)
yaw   = np.arctan2(-f[0], f[1])              # +left, 0 = facing initial heading   (verify sign)
pitch = np.arcsin(np.clip(f[2], -1, 1))      # +up
```
Reason for the assumption [D]: ARKit camera/device forward is −Z (Y up); after `YUP2ZUP` and `Rx(-90°)` the visualization in README L642 shows the head axes with Z up, which should place forward on a horizontal axis — **I could not confirm which column; print `H[:3,:3]` while looking straight ahead (should be near a permutation matrix) to pin it**. Roll is ignored (2-axis rig).
- **Recenter:** the global origin is set at app launch; heading drifts relative to the rig's "zero". Use either long-press Digital Crown (app side) or software: store `yaw0, pitch0` on a button/pinch and subtract. Wrap yaw to (−π, π].
- **Limits:** clamp to the servos' *mechanical* range (e.g. pan ±90°, tilt −30…+60°, set to your gimbal), **before** sending; add soft limits 5° inside the stops. Pi-side deadband ~0.3–0.5° to stop jitter.
- **Smoothing / rate:** the library does none. Recommended: One-Euro or critically-damped filter on yaw/pitch (cutoff ~5–8 Hz, tunable), then a **slew-rate limiter** (deg/s cap from servo spec). Command servos at **50–100 Hz** (hobby PWM servos update at 50 Hz; Dynamixel/serial bus servos can take more). **[D]** Incoming data is ≈120 Hz at source (§2); network jitter is unknown **[?]**. No prediction/lead compensation exists; for the round-trip lag in §4 the operator will see the view arrive late — expect to tune filter vs. lag.
- Servo hardware drive: **not in either repo for hobby servos** — use `pigpio`/`gpiozero`/PCA9685 (I2C) from the Pi, or a serial bus servo library.

### 5.4 Concrete hook points
| Purpose | Where | Note |
|---|---|---|
| Config stereo wide frame | `VisionProStreamer.configure_video` `streamer.py` L2515 | `device=None, size="2560x720", fps=30, stereo=True` |
| Push composited frame | `update_frame` L2486 | called from your capture/stitch thread, ≤30 Hz |
| Avoid | `register_frame_callback` L2419 | runs on asyncio loop (L759–814) |
| Start | `start_webrtc` L4656 (or `start_streaming` L4949, older API) | → `serve()` L4238 on :9999 |
| Head pose read | `get_latest()` L2143 (`use_cache=True, cache_ms=10`) | call from your servo thread @ 50–100 Hz |
| Cheaper pose path | `_process_hand_update` L1056–1110 | every update converts both hands + ~54 joint matrices with NumPy even if you want only the head; on a Pi at up to 200 Hz this may waste CPU **[?]**. Optional: add early `if self._head_only:` branch computing only `head`. Rough wire estimate **[D]**: ≤~5 KB/update × 200 Hz ≲ 8 Mbps uplink from the headset — competes with video on the same Wi-Fi |
| Unreliable pose channel | `ht_backend="webrtc"` (L4457–4463) | local mode is unordered/maxRetransmits=0 → no head-of-line stale poses; gRPC/TCP can queue stale poses under Wi-Fi loss |
| Encoder caps/preset | aiortc `codecs/h264.py` L20–24, L278–281 | monkeypatch at import |
| Hardware-encoder idea | aiortc `H264Encoder.pack` L298; replace `ProcessedVideoTrack` (`streamer.py` L671–827) | only if abandoning x264 |
| Remote (non-LAN) | `VisionProStreamer(ip="ABC-1234")` + `start_webrtc()` (README L137–178) | Cloudflare signaling/TURN default `streamer.py` L901 |

Minimal driver sketch **[D, untested]**:
```python
from avp_stream import VisionProStreamer
import aiortc.codecs.h264 as h264; h264.MAX_BITRATE = h264.DEFAULT_BITRATE = 10_000_000
s = VisionProStreamer(ip="<AVP IP or room code>", ht_backend="webrtc")
s.configure_video(device=None, size="2560x720", fps=30, stereo=True)
s.start_webrtc()
threading.Thread(target=capture_stitch_loop, args=(s,), daemon=True).start()   # cv2 x2 -> hstack -> s.update_frame(sbs)
while True:                                   # servo loop
    d = s.get_latest(use_cache=True, cache_ms=10)
    if d: pan, tilt = map_head_to_servo(d["head"][0]); drive_servos(pan, tilt)
    time.sleep(0.01)
```

---

## 6. OpenTeleVision (TeleVision) comparison

| Aspect | VisionProTeleop | OpenTeleVision (TeleVision) |
|---|---|---|
| Headset side | Native visionOS app (App Store) | **Safari WebXR page** served by **Vuer** from your PC; needs HTTPS (mkcert root CA installed on headset), port 8012 (README) |
| Streaming (default) | WebRTC, software x264/VP8 | **JPEG (quality 80) per eye over websocket** via Vuer `ImageBackground`, loop ~30 ms sleep, vertically decimated `[::2]` (`teleop/TeleVision.py` L128–196). Optional WebRTC mode: aiortc **H264 forced** (`teleop/webrtc/zed_server.py` L71–77) into `WebRTCStereoVideoPlane` (`TeleVision.py` L114–126) — hardcoded IP/port in the shipped example; issues #43/#52 show users struggle with it |
| Resolution / fps | Arbitrary; ≤30 fps encoder cap | Paper: **480×640 per eye, whole loop 60 Hz** (arXiv 2407.01512); code: ZED Mini HD720 @ 60 (`teleop/teleop_active_cam.py` L28–31) |
| Pose return | gRPC/WebRTC datachannel, full 4×4 head + hands @ ~120–200 Hz | Vuer events (`CAMERA_MOVE` head 4×4, hand events) over websocket, handled `TeleVision.py` L80–112, head matrix L225–228 |
| Cameras | any (incl. UVC) | **ZED Mini (stereolabs) via pyzed/ZED SDK** hard-wired (`teleop_active_cam.py` L13, L26–79); replacing with 2 UVC means rewriting capture |
| Gimbal | none | **Yes: 2-DoF yaw/pitch active neck**, Dynamixel (paper: XL330-M288-T, 3D-printed mount); `teleop/dynamixel/active_cam.py` L59–71, `driver.py` L96–175 (57600 baud, `int(angle*2048/π)` goal-position ticks), `dynamixel_robot.py` L140–150. **Head→gimbal mapping:** `teleop_active_cam.py` L56–64 (yaw/pitch from head rotation, YUP→ZUP rotation). **No limits, no filtering, no rate limiting** in this path (`map_to_valid_range` only wraps mod 2π). |
| Published latency | benchmark.md (§4) | **Not published** (paper gives only the 60 Hz loop). Third-party issue #52: **>500 ms** on default LAN image mode (anecdotal, Quest/AVP, Feb 2025). |
| Deps / maintenance | Active (last commit 2025-12-28), PyPI package | Last commit 2024-09-26; pinned old deps (`vuer==0.0.32rc7`, `aiortc==1.8.0`, `av==11.0.0`, `numpy==1.23.0`, `torch==2.3.0`, Python 3.8 via conda) — awkward on Pi 5 aarch64 |
| Licence | MIT | Apache-2.0 |

**Pros/cons for this rig**
- VisionProTeleop: ✅ real WebRTC video with published latency, 120 Hz pose in full 4×4, no certificate dance, library on the Pi, LAN or remote mode; ❌ you must build the dual-UVC stitch + servo layer; stock aiortc caps 30 fps/3 Mbps and software x264; unknown codec negotiation; flat head-anchored video plane (no per-eye lens/FOV matching).
- OpenTeleVision: ✅ proven *concept* of head-following 2-DoF active neck with stereo (exactly our UX) and shows the head→Dynamixel mapping; ❌ default video path is JPEG/websocket (reports of high delay), WebXR in Safari + local HTTPS, ZED-specific capture, old pinned deps, no safety limits/smoothing, no latency data.

---

## 7. Recommendation

**Base on VisionProTeleop** (video + pose + remote mode + published latency); **borrow from OpenTeleVision only the head→yaw/pitch→active-neck idea** (`teleop_active_cam.py` L56–64 and the XL330 gimbal hardware choice), adding the limits/filtering it lacks. Don't run OpenTeleVision's Vuer stack on the Pi.

Cheapest path to evidence (all reversible): (1) install `avp_stream` on a Pi 5 and stream a synthetic 2560×720 SBS pattern to the headset; log negotiated codec & Pi CPU; (2) add two-UVC hstack; (3) print head matrix to pin the forward column and signs; (4) servo; (5) measure motion-to-photon with a slow-mo phone camera.

## 8. Unknowns / flags
1. **Negotiated codec** (VP8 vs H264) with the real headset — [?].
2. Whether the headset sends **REMB**, i.e. whether bitrate will actually exceed 1–3 Mbps without patching — [?].
3. **End-to-end head-pose rate/jitter over Wi-Fi 6** and whether head pose is predicted — [?] (`benchmark_rate.py` unread).
4. **Forward-axis column & sign** after `rotate_head` — derived only; must be verified empirically.
5. **Pi 5 cost** of MJPEG decode ×2 + hstack + x264 "medium" at 2560×720@30 — not measured; aarch64 PyAV/x264 behaviour untested.
6. UVC camera sync and USB 2.0 bandwidth for two cameras — untested (ideas.md notes it).
7. Published latency is round-trip with synthetic frames — not glass-to-glass; no head→servo latency anywhere.
8. Authors' physical teleop camera hardware — not in the repo. XL330 tick resolution/limits — not verified (driver uses 2048 ticks/π).
9. Remote mode depends on a third-party Cloudflare signaling/TURN (may be limited later).
10. TeleVision latency — not published (third-party anecdote only).