# MB-001 — Live per-eye video into Apple Vision Pro (research)

> Sibling (TouchDesigner-focused): [`MB-001-touchdesigner-avp-stream.md`](./MB-001-touchdesigner-avp-stream.md)

Researched 2026-11-03 (Asia/Shanghai). Scope: servo pan-tilt rig + 2 cameras (USB webcams on Pi-class board, or 2× Insta360 Go Ultra) -> Wi-Fi -> native visionOS app, one camera per eye.
Legend: **[S]** = backed by a cited source. **[I]** = my inference or engineering judgement, not sourced. **[?]** = not verified, test it.

---
## 0. TL;DR
* Wi-Fi: the M5 Vision Pro is still Wi-Fi 6 (802.11ax, 2.4/5 GHz only, 2×2, 1,200 Mbps PHY max on 80 MHz). A Wi-Fi 7 router will not help the headset hop. [S] https://support.apple.com/en-us/125436 , https://support.apple.com/guide/deployment/wi-fi-ethernet-specifications-apple-devices-dep268652e6c/web . M5 display modes: 90/96/100/120 Hz [S] same Apple page.
* Bandwidth is not the problem (2 × 720p H.264 is ~10-30 Mbps). The problem is the **latency floor of the cameras** (~100 ms for USB webcams) plus Wi-Fi jitter. Transport choice (UDP vs WebRTC) only moves things by ~10-30 ms. [S] https://transitiverobotics.com/blog/webrtc-latency-breakdown/
* **Insta360 Go Ultra: Insta360 says it cannot livestream and cannot be a webcam** [S] https://onlinemanual.insta360.com/goultra/en-us/faq/functionality/livestreaming . Only an unofficial reverse-engineered Wi-Fi TCP:6666 H.265 pull exists [S] https://github.com/Daiki-Iijima/insta360-go-ultra-sdk . It also can't be a clean stereo pair (separate Wi-Fi hotspots per camera [I]). **Use USB webcams (or CSI cameras).**
* **Raspberry Pi 5 has no hardware H.264 encoder** (libx264 on CPU only; ~60% of one core for 1080p30 `ultrafast/zerolatency`). Pi 4 and earlier do have one (h264_v4l2m2m, hard bitrate ceiling ~6-14 Mbps). [S] https://pip-assets.raspberrypi.com/categories/685-app-notes-guides-whitepapers/documents/RP-010033-WP-1-H.264%20encoding%20performance%20on%20Raspberry%20Pi%205_series%20computers.pdf , https://forums.raspberrypi.com/viewtopic.php?t=367583
* Cheapest correct first prototype: **webcams output MJPEG natively -> Pi just forwards JPEG frames (no encode) -> UDP/WebSocket -> visionOS decode -> two textures -> RealityKit ShaderGraph Camera Index Switch.** Then upgrade to H.264 + VideoToolbox, then to CompositorServices.

---
## 1. visionOS display paths for a different image per eye

| # | Path | How per-eye works | Latency notes | Verdict |
|---|------|-------------------|---------------|---------|
| A | **RealityKit + ShaderGraph `Camera Index Switch`** (Reality Composer Pro material) | Node outputs `Left`/`Right` input depending on which eye is rendering (Mono elsewhere). Wire two texture params (or one SBS texture + per-eye UV offset) into an Unlit surface; set via `ShaderGraphMaterial.setParameter(name:value: .textureResource(...))`. [S] https://apple-docs.everest.mt/docs/shadergraph/realitykit/camera-index-switch-(realitykit)/ ; https://apple-docs.everest.mt/docs/visionos/displaying-a-stereoscopic-image-in-visionos/ ; demo https://github.com/halmueller/ShaderGraphStereo (MIT) | Content goes through RealityKit -> render server -> compositor (Shared Space/RealityKit path) [S] https://apple-docs.everest.mt/docs/visionos/understanding-the-visionos-render-pipeline/ . Likely 1-2 compositor frames extra vs direct Metal [I] | **Best fast start.** Works in Shared Space window-less volumes or ImmersiveSpace |
| A1 | Live texture feed for A: `TextureResource.DrawableQueue` | `queue = try TextureResource.DrawableQueue(.init(pixelFormat:.bgra8Unorm,width:,height:,usage:[.renderTarget,.shaderRead,.shaderWrite],mipmapsMode:.none))`, `resource.replace(withDrawables: queue)`, each frame `nextDrawable()` -> Metal blit/render CVPixelBuffer -> `present()`. Use one queue per eye, or one SBS queue. [S] https://developer.apple.com/forums/thread/742567 (as summarised by search; page returned 403 to my fetcher), https://apple-docs.everest.mt/docs/visionos/generating-procedural-textures-in-visionos/ | Cheapest per-frame path; needs CVMetalTextureCache for zero-copy [I] | Recommended with A |
| A2 | `LowLevelTexture` (visionOS 2+) | Metal texture RealityKit samples; update with `texture.replace(using: commandBuffer)` in a RealityKit `System`; then `TextureResource(from:)` [S] https://apple-docs.everest.mt/docs/realitykit/lowleveltexture/ , WWDC24 "Build a spatial drawing app with RealityKit" (session 10104, cited from search results, not re-read) | Same as A1; newer, compute-friendly (e.g. do YUV->RGB, crop, lens undistort in a compute shader) | Good alternative to A1 |
| B | **CompositorServices + Metal** (full control) | `ImmersiveSpace { CompositorLayer(configuration:) { layerRenderer in ... } }`; per frame `queryNextFrame()`, `startUpdate/endUpdate`, `predictTiming()`, wait `optimalInputTime`, `startSubmission()`, `queryDrawables()`, render each `drawable.views` (left/right) from your own textures, `endSubmission()`. Layout `.layered` (one texture, 2 slices; keeps foveation + vertex amplification) or `.shared`; `.dedicated` = 2 textures but no vertex amplification. [S] https://developer.apple.com/videos/play/wwdc2023/10089/ , https://developer.apple.com/videos/play/wwdc2025/294/ , https://github.com/gnikoloff/drawing-graphics-on-apple-vision-with-metal-rendering-api | Bypasses the render server [S, render-pipeline doc above]. You set `drawable.deviceAnchor` (pose) right before encoding; the compositor reprojects with it. Lowest-latency, most work. Pace frames at display rate; long stalls -> system terminates app [S] render-pipeline doc | **End-state for lowest latency & custom warping** |
| C | **AVSampleBufferDisplayLayer / `AVSampleBufferVideoRenderer`** | Decoded or even *compressed* CMSampleBuffers enqueued; this is a *mono* image in a UIView (window) — no per-eye by itself. For stereo you'd put two layers... only useful in a window, not per-eye in a space. Apple sample for UVC cameras uses it: https://apple-docs.everest.mt/docs/sample-code/visionos/displaying-video-from-connected-devices/ . Also `VideoPlayerComponent(videoRenderer:)` exists, which lets RealityKit show a renderer's output [S] https://apple-docs.everest.mt/docs/realitykit/videoplayercomponent/ | Good decode/queue latency if you set `alwaysDiscardsLateVideoFrames`, drop late frames, no display timestamps [S sample; I for the details]. | Use for mono preview / debugging; not for per-eye |
| D | **AVPlayer + MV-HEVC + `VideoPlayerComponent`** | `component.desiredViewingMode = .stereo` shows MV-HEVC stereo; immersive modes `.portal/.progressive/.full` [S] https://developer.apple.com/videos/play/wwdc2025/296/ , VideoPlayerComponent doc above | AVPlayer/HLS buffering = seconds. LL-HLS ~1-5 s typical [S] https://nighthawk.tv/blog/low-latency-hls-llhls-vs-webrtc-vs-srt-choosing-the-right-protocol-for-your-late . Pi has no MV-HEVC encoder; VideoToolbox MV-HEVC encode only on Apple HW (sample converting SBS -> MV-HEVC: https://developer.apple.com/documentation/avfoundation/converting-side-by-side-3d-video-to-multiview-hevc-and-spatial-video ) | **Not for live teleop**. Fine for recording/playback of rig footage |
| E | `VideoMaterial` (RealityKit) | Wraps an AVPlayer/renderer as a material; mono per material — no eye switch node on its own [I]. | | Not useful for per-eye |
| F | **WebXR in Safari** (no native app) | visionOS 2+ Safari supports `immersive-vr` by default [S] https://www.uploadvr.com/visionos-2-apple-vision-pro-webxr/ , https://webkit.org/blog/15443/news-from-wwdc24-webkit-in-safari-18-beta/ ; stereo via WebGL per-view rendering; Open-TeleVision does exactly this with a stereo camera at 480x640 per eye, 60 Hz loop [S] https://arxiv.org/html/2407.01512v1 | Browser decode + WebGL; no control of pose timing as in B | Fast demo, not your goal (you build native) |
| G | **Wired UVC via Developer Strap** | visionOS lets apps read UVC cameras attached to the Developer Strap using `AVCaptureDevice.DiscoverySession(deviceTypes:[.external])`, needs `NSCameraUsageDescription` [S] https://apple-docs.everest.mt/docs/sample-code/visionos/displaying-video-from-connected-devices/ | No network, camera latency only. One USB-C port -> hub or a single stereo UVC module for two eyes [I, ?] | Interesting *wired baseline/benchmark* and sanity check for latency of the cameras alone |

### visionOS 26 / 27 additions (relevant parts only)
* visionOS 26: Apple Projected Media Profile (APMP) for 180/360/wide-FOV video, native in AVKit/RealityKit/Safari incl. HLS; stereo APMP must be MV-HEVC [S] https://developer.apple.com/videos/play/wwdc2025/317/ , /297/ , https://developer.apple.com/streaming/hls-authoring-specification-for-apple-devices.html . All playback-oriented, not live-low-latency. `Compositor Services` gained `queryDrawables()` returning one or two drawables [S] WWDC25/294.
* visionOS 27 (WWDC26): RealityKit lighting/projective textures/cloth/navigation/Gaussian splats; Apple Immersive Video live preview from Mac (`ImmersivePreviewRenderer`), camera presentation overrides [S] https://developer.apple.com/videos/play/wwdc2026/279/ , https://developer.apple.com/videos/play/wwdc2026/287/ , https://developer.apple.com/visionos/whats-new/ (via search summary; I did not read the full sessions). **I found no new first-party "receive a live stereo network stream" API.** [?]

---
## 2. Transport Pi -> AVP (Wi-Fi 6)
Reality check: glass-to-glass ≈ camera (≈100 ms with 30 fps USB cams over USB bus) + encode (~10 ms) + network + jitter buffer (~7-10 ms) + decode (~10 ms) + display. Transitive measured: direct 110 ms, local WebRTC 130 ms, remote 170 ms, all with USB cameras at 30 fps. [S] https://transitiverobotics.com/blog/webrtc-latency-breakdown/ . Therefore **use 60 fps cameras / short exposure to cut the 33 ms frame interval** [I].

| Rank (est. lowest first) | Transport | Rough glass-to-glass (est.) | Visionos receiver | Notes |
|---|---|---|---|---|
| 1 | **Custom RTP-ish/UDP with H.264 (or HEVC) + VideoToolbox `VTDecompressionSession`** | camera floor + ~30-60 ms [I] | native, Network.framework `NWConnection` UDP; decode -> `CVPixelBuffer` -> Metal | Full control: no jitter buffer, drop late frames. You must do packetization/FEC. Apple low-latency VT encode flags exist for the *sender* on Apple HW (`kVTVideoEncoderSpecification_EnableLowLatencyRateControl`) [S] https://developer.apple.com/videos/play/wwdc2021/10158/ ; Pi side uses libx264 `zerolatency`, no B-frames |
| 2 | **WebRTC** | +~10 ms over raw in LAN tests [S Transitive]; VisionProTeleop: wireless sub-100 ms (round-trip metric) at 720p over Wi-Fi 6; wired 50 ms at 4K stereo [S] https://github.com/Improbable-AI/VisionProTeleop/releases , methodology https://github.com/Improbable-AI/VisionProTeleop/blob/main/docs/benchmark.md | LiveKit's WebRTC xcframework supports visionOS (`LKRTC*` types) [S] https://github.com/livekit/webrtc-xcframework (MIT), guide https://medium.com/@can-dindar/streaming-with-webrtc-to-vision-pro-a-custom-video-rendering-guide-5f0bc74c4cee ; `stasel/WebRTC` is **iOS+macOS only**, no visionOS release (visionOS PR open) [S] https://github.com/stasel/WebRTC . LiveKit Swift SDK supports visionOS (current versions need visionOS 26) [S] https://github.com/livekit/client-sdk-swift (Apache-2.0) | Pi sender: MediaMTX (WHEP/RTSP/SRT) supports `rpiCamera` source and WebRTC out, https://github.com/bluenviron/mediamtx . Congestion control + NAT + stats for free; frames arrive as `LKRTCVideoFrame`, you convert to CVPixelBuffer -> texture |
| 3 | **SRT** | LAN 120 ms default latency buffer, tunable; it's a retransmission buffer, not end-to-end [S] https://www.forasoft.com/learn/video-streaming/articles-streaming/srt-deep-dive | no first-party visionOS lib; would need building libsrt [?] | Good for loss but no advantage for a clean LAN |
| 4 | **MJPEG** over UDP/HTTP/WebSocket | camera floor + ~20-50 ms [I, unsourced]; high bandwidth (~10-40 Mbps per eye at 720p-1080p) fine on Wi-Fi 6 | `VTDecompressionSession` supports JPEG [?] or ImageIO; trivial in WebSocket | UVC webcams can output MJPG natively -> **no encode on Pi**. Frames independent -> lost packet = lost frame only. Best *prototype* path |
| 5 | **RTSP (UDP)** | ~100-300 ms typical; TCP interleaved can buffer far more [S] Nighthawk/FastoCloud posts above | no native RTSP client on visionOS; use FFmpeg/libVLC-like (build yourself) [?] | |
| 6 | **NDI** | few frames, ~100+ Mbps for full-NDI [I, ?] | NDI SDK added visionOS support in 6.2 and fixed missing receiver-advertiser symbols in 6.3.2 [S] https://docs.ndi.video/all/developing-with-ndi/sdk/release-notes ; Unity demo https://github.com/gwangyu-lee/KlakNDI-visionOS-Demo (no licence file). Needs `NSBonjourServices _ndi._tcp` + local-network permission, multicast entitlement if using multicast discovery [S] | Pi sender story is weak; SDK terms apply; overkill |
| 7 | **HLS/LL-HLS** | 1-5 s [S] | AVPlayer | Not usable for head-tracked teleop |
| n/a | ALVR / Moonlight | host-GPU -> headset streamers; they **solve the wrong direction** but show the decode/render/pose-timing pipeline on visionOS | see §3 | Study, don't use |

### Pi-side encoding options
* **Pi 5**: software x264 only; `ultrafast`+`zerolatency` 1080p30 ≈ 60% of one core, high-quality preset 100-150% of a core [S Pi whitepaper]. A side-by-side 2560×720 frame is ~1.84 MP (vs 2.07 MP for 1080p), so ~1 core [I]. Two separate 1280×720 encodes ≈ same load.
* **Pi 4**: hardware H.264 (h264_v4l2m2m): ~0 CPU, but bitrate ceiling ~6-14 Mbps, min ~240 kb/s, no 1080p60 [S]. Only one CSI port (Pi 5 has two) [I, ?].
* **Jetson Orin Nano**: no NVENC hardware encoder; software x264 only [S] https://docs.nvidia.com/jetson/archives/r38.2.1/DeveloperGuide/SD/Multimedia/SoftwareEncodeInOrinNano.html . (Orin NX/AGX do have NVENC [S: NVIDIA forum](https://forums.developer.nvidia.com/t/encoder-in-orin-nx-vs-orin-nano/288951).)
* **RK3588 boards** (Orange Pi 5, Rock 5B): VPU hardware H.264/H.265 encode up to 8K@30; GStreamer `mpph264enc` [S datasheets, e.g. https://groundplanestudio.com/silicon/rk3588 ] — best "small board with real HW encoder" option [I].
* Webcam MJPG passthrough: zero encode, any board.

### Insta360 Go Ultra
* Official: **no livestreaming, no webcam** [S above]. RTMP guide only for ONE X/X2/X3/X4/X5/ONE R/ONE RS [S https://onlinemanual.insta360.com/app/en-us/operation-tutorial/live/rtmp-publish-streaming-facebook-youtube]. Reddit users note GO 3 can webcam in its pod; Ultra can't [S reddit threads, https://www.reddit.com/r/Insta360/comments/1n6975e/insta360_go_ultra_as_webcam/].
* Unofficial: raw H.265 over Wi-Fi TCP 6666 (protobuf); Pi Zero 2W + second Wi-Fi adapter recommended as bridge [S https://github.com/Daiki-Iijima/insta360-go-ultra-sdk ; protocol dependency https://github.com/RigacciOrg/insta360-wifi-api GPL-3.0]. Each camera is its own AP, so two cameras need two radios; sync and latency unknown, camera onboard processing adds more [I]. Not recommended.

---
## 3. Existing open source / sample code
(licenses from GitHub API; stars as of 2026-11-03)

| Repo | What it demonstrates | License |
|---|---|---|
| https://github.com/Improbable-AI/VisionProTeleop (836★) | **Closest match.** visionOS app + Python lib: WebRTC video/audio **to** Vision Pro from `/dev/videoN` (v4l2) incl. **side-by-side stereo** (`configure_video(..., stereo=True)`), streams hand/head tracking back, latency benchmark scripts, wired (dev strap) & remote (TURN) modes. Known issue: "Connected (Negotiating)" on relaunch https://github.com/Improbable-AI/VisionProTeleop/issues/27 | MIT |
| https://github.com/alvr-org/alvr-visionos (534★) | Production-grade **CompositorServices + Metal** client, decode of streamed HEVC/H.264 with VideoToolbox, pose-sync with compositor, foveation, hand tracking [S: repo README/wiki; transport details not verified by me]; macOS + Xcode build (`zsh build_and_repack.sh`) [S] https://github.com/alvr-org/alvr-visionos/wiki/Building | MIT |
| https://github.com/livekit/webrtc-xcframework (37★) + https://github.com/livekit/client-sdk-swift (456★) | WebRTC binaries/SDK that actually compile for visionOS | MIT / Apache-2.0 |
| https://github.com/halmueller/ShaderGraphStereo (14★) | Camera Index Switch stereo-image demo in RealityKit | MIT |
| https://github.com/gnikoloff/drawing-graphics-on-apple-vision-with-metal-rendering-api (304★) | Tutorial on CompositorServices layouts (.layered/.shared/.dedicated), foveation, per-eye views | no licence file (GitHub shows none) |
| https://github.com/OpenTeleVision/TeleVision (1,321★) | Active neck gimbal + stereo (ZED Mini) streamed to AVP via WebXR in Safari, 480×640/eye, 60 Hz; Vuer server; paper https://arxiv.org/html/2407.01512v1 . **Exactly your concept (2-DoF Dynamixel XL330 yaw/pitch gimbal following the operator's head).** Paper shows stereo beat mono in task time/success | Apache-2.0 (LICENSE file) |
| https://github.com/RikuKunMS2/moonlight-ios-vision (111★) | Moonlight client fork for visionOS (RealityKit) | GPL-3.0 |
| https://github.com/Daiki-Iijima/insta360-go-ultra-sdk | Unofficial Go Ultra live H.265 pull | not checked (rate-limited) |
| https://github.com/bluenviron/mediamtx | Pi-side server: rpiCamera -> WebRTC/WHEP/RTSP/SRT | MIT |
| Apple: "Displaying video from connected devices" https://apple-docs.everest.mt/docs/sample-code/visionos/displaying-video-from-connected-devices/ ; "Drawing fully immersive content using Metal"; "Converting side-by-side 3D video to multiview HEVC" https://developer.apple.com/documentation/avfoundation/converting-side-by-side-3d-video-to-multiview-hevc-and-spatial-video ; "Connecting iPadOS and visionOS apps over the local network" https://developer.apple.com/documentation/visionos/connecting-ipados-and-visionos-apps-over-the-local-network | AVSampleBufferVideoRenderer pipeline; Metal immersive; MV-HEVC authoring; local networking | Apple sample licence |

Not found / not verified: a ready-made visionOS "UDP H.264 stereo receiver" sample. `because-why-not/webrtc_visionpro_workspace` (20★, 2024) is a custom WebRTC build scripts repo, superseded by LiveKit's builds [I].

---
## 4. Platform constraints & gotchas
1. **Wi-Fi 6 only, 5 GHz 2×2 80 MHz, 1,200 Mbps PHY** [S]. Real-world is a fraction; stream needs only ~10-40 Mbps. Jitter from channel contention is the actual risk: use a dedicated 5 GHz SSID/channel, or put the **Pi on wired Ethernet to the AP** (Pi onboard Wi-Fi is 802.11ac, not Wi-Fi 6 [I, ?]). Disable power-save on the Pi radio. Tools: iperf3 + ping jitter test from AVP-adjacent device first.
2. **Local network permission**: add `NSLocalNetworkUsageDescription`; for Bonjour discovery add `NSBonjourServices` with exact service types; multicast/broadcast needs `com.apple.developer.networking.multicast` (Apple approval) on iOS-family [S] https://developer.apple.com/documentation/technotes/tn3179-understanding-local-network-privacy , https://developer.apple.com/news/?id=0oi77447 . Easiest: type the Pi's static IP or use mDNS names (`.local`) with one declared Bonjour type.
3. **Foreground / Full Space**: ARKit head pose (`WorldTrackingProvider.queryDeviceAnchor`) is only available while a **Full Space (ImmersiveSpace)** is open; stop/recreate the provider when the space closes [S] https://apple-docs.everest.mt/docs/visionos/setting-up-access-to-arkit-data/ , https://developer.apple.com/forums/thread/761167 . No documented background streaming for third-party apps; assume the app is suspended when not frontmost and the rig must have a failsafe (servo hold/park on link loss) [I].
4. **Render rate**: render pipeline targets the display rate (90 Hz typical, 96/100/120 on M5), "maintain a consistent Metal rendering frame rate equal to the display refresh rate"; query pose prediction **immediately before encoding GPU work**; long stalls terminate the app [S render-pipeline doc]. Your video arrives at 30-60 fps; compositor reprojects only the *pose* difference for the layer you give it, it knows nothing about the remote camera.
5. **Camera access**: `NSCameraUsageDescription` only needed for local UVC cameras (G). Main cameras (passthrough) not available to normal apps (Enterprise API) — irrelevant here [I].
6. **Thermal/battery**: no authoritative numbers found. Hardware decode (VideoToolbox) + one Metal pass is light; WebRTC stack and software JPEG decode cost more CPU [I]. Battery is ~2-2.5 h in general use per Apple (not re-verified).
7. **Motion-to-photon with head-driven servos** (core design issue; engineering recommendations [I], no source):
   * Loop = head pose -> network -> Pi -> servo -> camera -> encode -> network -> decode -> display. Expect 150-250 ms with hobby servos + USB cams. Servo response (slew) often dominates; consider digital-servo/Dynamixel (Open-TeleVision uses Dynamixel XL330 [S paper]) with position feedback.
   * **Do not head-lock the video plane naively**: any delay appears as the whole picture swimming against vestibular cues -> discomfort.
   * Mitigation 1: **world-lock a virtual window/sphere section**, and place the quad each frame at the *camera's pose at capture time* (send the commanded/measured pan/tilt angle in the frame metadata; pose of the head at the time the command was issued). When the user turns, the window stays put while the content catches up — rotational latency becomes "late-arriving window edge" instead of swim. This is "late warping" using capture-time pose.
   * Mitigation 2: Use the angle error (actual servo angle vs current head yaw/pitch) to reproject the quad each frame; mask with soft vignette; clamp velocities.
   * Mitigation 3: low-pass/lead (predict head pose ~100-150 ms ahead for servo commands), limit servo speed, high fps (60) cameras, short exposure, low-latency encoder tune, drop (not buffer) late frames.
   * Stereo geometry: IPD of cameras (~63 mm), toe-in vs parallel (use parallel + shift), matched exposure/white balance, hardware sync not available on cheap webcams (use one Pi, same-instant capture, send both in one SBS packet to keep pairs together). FOV mismatch: webcams ~70-90° vs AVP ~100°+ per eye -> black borders; fine with world-locked window.
   * Frame-pairing: single SBS stream guarantees L/R from same instant of encode; two streams need timestamp pairing.
   * Quick latency test: Transitive's infinite-mirror method https://transitiverobotics.com/blog/webrtc-latency-breakdown/ .

---
## 5. Recommended build path
**Stage 0 (a weekend, risk-reduction):** Run VisionProTeleop's server on a laptop or the Pi with the two webcams SBS (`stereo=True`) and install its Tracking Streamer on the AVP. Gives a working stereo + head-pose-back loop and a benchmark. https://github.com/Improbable-AI/VisionProTeleop (MIT).
**Stage 1 (own app, simplest):** Pi (Pi 5 or Pi 4 if you want HW encode) + 2 USB webcams at 1280×720 @30-60 MJPG -> either MJPEG passthrough or ffmpeg/GStreamer `hstack` -> x264 `ultrafast zerolatency`, one SBS stream -> UDP (or WebRTC via MediaMTX). visionOS: `NWConnection` UDP (or LiveKit WebRTC) -> `VTDecompressionSession` -> `CVMetalTextureCache` -> `TextureResource.DrawableQueue` -> RealityKit ShaderGraph with Camera Index Switch (SBS UV offset per eye) on a world-locked plane. Head pose from `WorldTrackingProvider` at 90 Hz -> small UDP packet (e.g. 16 bytes of quaternion/timestamp) -> Pi -> servos.
**Stage 2 (latency polish):** Move to CompositorServices (`.layered`), render quad with capture-time pose, tune. Consider RK3588 board or Pi 4 HW encoder, CSI cameras (Pi 5 has 2 CSI ports [I,?]) to drop USB latency, or global-shutter/hardware-synced stereo modules.

## 6. Open questions to test early
* Real camera latency of your webcams at 720p30/60 (infinite mirror).
* Pi 5 x264 SBS 2560×720 CPU/latency at your target fps vs Pi 4 HW encoder (does 2560 width work on v4l2m2m? [?]).
* Wi-Fi 6 jitter in your environment (AVP <-> AP <-> Pi) measured with ping/iperf.
* RealityKit (render server) vs CompositorServices latency difference on your device (measure; I could not find published numbers).
* Whether the Developer Strap + UVC path (G) is useful as a wired baseline.