IQ.Pilot Release Commit @ 0798119
This commit is contained in:
54
tools/iqmacvisiond/README.md
Normal file
54
tools/iqmacvisiond/README.md
Normal file
@@ -0,0 +1,54 @@
|
||||
# iqmacvisiond — IQ Vision offload server
|
||||
|
||||
Runs the iqvd perception model on an Apple-Silicon Mac and serves 2D detections
|
||||
to one IQ device over wifi. The device (`iqvd`) ships the camera frame, the Mac
|
||||
runs YOLO on the Metal GPU/NPU, and only boxes come back — the device does no
|
||||
inference, so vision dots no longer contend with the driving model.
|
||||
|
||||
## Wire path
|
||||
|
||||
```
|
||||
IQ device (auto-hotspot AP) ──wifi──▶ Mac (IQ Vision.app)
|
||||
iqvd VisionClient iqmacvisiond server
|
||||
read frame → downscale 640w cv2 decode → YOLOv8n (Metal)
|
||||
JPEG encode → INFER ───────────────▶ detect
|
||||
RESULT ◀─────────────────────────── {tracks: [2D boxes]}
|
||||
publish iqVehicleTracks (dots)
|
||||
publish iqEnvironment (3D via ground-plane + calibration)
|
||||
```
|
||||
|
||||
- Discovery: the device UDP-broadcasts `IQVISION_DISCOVER_V1` on the subnet; the
|
||||
Mac replies `IQVISION_HERE_V1:<tcp_port>`. No config, no pairing.
|
||||
- Protocol: `iqvd_private_src/offload/protocol.py` (length-prefixed frames, JSON
|
||||
header + optional binary blob). Shared verbatim by both sides — it is the ABI.
|
||||
- Ports: tcp/51998 inference, udp/51999 discovery, tcp/51995 localhost status.
|
||||
|
||||
## The Mac app
|
||||
|
||||
- Menu-bar app (`◎` waiting, `◉` connected). Menu shows device, frames served,
|
||||
inference p50/p99, and **Quit**.
|
||||
- Keeps the Mac awake while running (`caffeinate`).
|
||||
- Ships the model in the dmg — no download on first run.
|
||||
- First launch creates a small venv (numpy, opencv-headless, rumps); tinygrad is
|
||||
bundled.
|
||||
|
||||
## Build
|
||||
|
||||
```
|
||||
tools/iqmacvisiond/macos/build_dmg.sh
|
||||
```
|
||||
|
||||
Produces `IQ Vision.app` and `IQVision.dmg`. See `macos/SIGNING.md` for signing +
|
||||
notarization.
|
||||
|
||||
## Run from source (dev)
|
||||
|
||||
```
|
||||
DEV=METAL python3 tools/iqmacvisiond/server.py # server only
|
||||
python3 tools/iqmacvisiond/menubar.py # menu-bar wrapper
|
||||
python3 tools/iqmacvisiond/test_offload.py # protocol/geometry/loopback
|
||||
```
|
||||
|
||||
Gating on the device: `VisionVehicleTracks` enables iqvd; when `maciqmodeld` (the
|
||||
eMac driving offload) is running, iqvd is Mac-or-nothing — it never runs local
|
||||
inference. On non-eMac setups iqvd falls back to on-device YOLO if no Mac is found.
|
||||
23
tools/iqmacvisiond/macos/SIGNING.md
Normal file
23
tools/iqmacvisiond/macos/SIGNING.md
Normal file
@@ -0,0 +1,23 @@
|
||||
# Signing & notarization — IQ Vision.app
|
||||
|
||||
Unsigned, the app runs after a right-click ▸ Open (Gatekeeper first-run). For
|
||||
distribution, sign + notarize:
|
||||
|
||||
```bash
|
||||
APP="tools/iqmacvisiond/macos/dist/IQ Vision.app"
|
||||
ENT="tools/iqmacvisiond/macos/entitlements.plist"
|
||||
IDENTITY="Developer ID Application: <YOUR NAME> (<TEAMID>)"
|
||||
|
||||
codesign --force --deep --options runtime --entitlements "$ENT" \
|
||||
--sign "$IDENTITY" "$APP"
|
||||
|
||||
hdiutil create -volname "IQ Vision" -srcfolder "$(dirname "$APP")" -ov -format UDZO \
|
||||
tools/iqmacvisiond/macos/dist/IQVision.dmg
|
||||
|
||||
xcrun notarytool submit tools/iqmacvisiond/macos/dist/IQVision.dmg \
|
||||
--apple-id "<APPLE_ID>" --team-id "<TEAMID>" --password "<APP_PW>" --wait
|
||||
xcrun stapler staple tools/iqmacvisiond/macos/dist/IQVision.dmg
|
||||
```
|
||||
|
||||
The entitlements cover: JIT + unsigned exec memory + library-validation off
|
||||
(tinygrad Metal JIT) and network server/client (LAN discovery + inference).
|
||||
59
tools/iqmacvisiond/macos/build_dmg.sh
Executable file
59
tools/iqmacvisiond/macos/build_dmg.sh
Executable file
@@ -0,0 +1,59 @@
|
||||
#!/bin/bash
|
||||
# Copyright © IQ.Lvbs, apart of Project Teal Lvbs, All Rights Reserved, licensed under https://konn3kt.com/tos/
|
||||
set -euo pipefail
|
||||
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
|
||||
REPO="$(cd "$HERE/../../.." >/dev/null && pwd)"
|
||||
OUT="${1:-$HERE/dist}"
|
||||
APP="$OUT/IQ Vision.app"
|
||||
|
||||
rm -rf "$OUT"
|
||||
mkdir -p "$APP/Contents/MacOS" "$APP/Contents/Resources"
|
||||
|
||||
cat > "$APP/Contents/Info.plist" <<'PLIST'
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>CFBundleName</key><string>IQ Vision</string>
|
||||
<key>CFBundleIdentifier</key><string>com.iqpilot.iqvision</string>
|
||||
<key>CFBundleVersion</key><string>1.0</string>
|
||||
<key>CFBundleShortVersionString</key><string>1.0</string>
|
||||
<key>CFBundlePackageType</key><string>APPL</string>
|
||||
<key>CFBundleExecutable</key><string>iqvision</string>
|
||||
<key>LSMinimumSystemVersion</key><string>13.0</string>
|
||||
<key>LSUIElement</key><true/>
|
||||
</dict>
|
||||
</plist>
|
||||
PLIST
|
||||
|
||||
cat > "$APP/Contents/MacOS/iqvision" <<'LAUNCH'
|
||||
#!/bin/bash
|
||||
RES="$(cd "$(dirname "$0")/../Resources" && pwd)"
|
||||
if [ ! -x "$HOME/Library/Application Support/IQVision/venv/bin/python" ]; then
|
||||
osascript -e "tell application \"Terminal\"
|
||||
activate
|
||||
do script \"bash '$RES/macos/setup.sh'\"
|
||||
end tell"
|
||||
else
|
||||
exec bash "$RES/macos/setup.sh" >/tmp/iqvision.log 2>&1
|
||||
fi
|
||||
LAUNCH
|
||||
chmod +x "$APP/Contents/MacOS/iqvision"
|
||||
|
||||
RES="$APP/Contents/Resources"
|
||||
SRC="$RES/openpilot/iqpilot/iqvd_private_src"
|
||||
mkdir -p "$RES/tools/iqmacvisiond" "$RES/macos" "$SRC/offload" "$SRC/models"
|
||||
|
||||
cp "$REPO/tools/iqmacvisiond/server.py" "$REPO/tools/iqmacvisiond/menubar.py" "$RES/tools/iqmacvisiond/"
|
||||
cp "$HERE/setup.sh" "$RES/macos/"
|
||||
cp "$REPO/iqpilot/iqvd_private_src/__init__.py" "$REPO/iqpilot/iqvd_private_src/yolov8_net.py" "$SRC/"
|
||||
cp "$REPO/iqpilot/iqvd_private_src/offload/"*.py "$SRC/offload/"
|
||||
cp "$REPO/iqpilot/iqvd_private_src/models/yolov8n.safetensors" "$SRC/models/"
|
||||
touch "$RES/openpilot/__init__.py" "$RES/openpilot/iqpilot/__init__.py" "$RES/tools/__init__.py" \
|
||||
"$RES/tools/iqmacvisiond/__init__.py"
|
||||
rsync -a --exclude=".git" --exclude="__pycache__" --exclude="extra" --exclude="test" \
|
||||
--exclude="examples" --exclude="docs" "$REPO/tinygrad_repo/" "$RES/tinygrad_repo/"
|
||||
|
||||
hdiutil create -volname "IQ Vision" -srcfolder "$OUT" -ov -format UDZO "$OUT/IQVision.dmg" >/dev/null
|
||||
echo "built: $OUT/IQVision.dmg ($(du -h "$OUT/IQVision.dmg" | cut -f1))"
|
||||
11
tools/iqmacvisiond/macos/entitlements.plist
Normal file
11
tools/iqmacvisiond/macos/entitlements.plist
Normal file
@@ -0,0 +1,11 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>com.apple.security.cs.allow-jit</key><true/>
|
||||
<key>com.apple.security.cs.allow-unsigned-executable-memory</key><true/>
|
||||
<key>com.apple.security.cs.disable-library-validation</key><true/>
|
||||
<key>com.apple.security.network.server</key><true/>
|
||||
<key>com.apple.security.network.client</key><true/>
|
||||
</dict>
|
||||
</plist>
|
||||
21
tools/iqmacvisiond/macos/setup.sh
Executable file
21
tools/iqmacvisiond/macos/setup.sh
Executable file
@@ -0,0 +1,21 @@
|
||||
#!/bin/bash
|
||||
# Copyright © IQ.Lvbs, apart of Project Teal Lvbs, All Rights Reserved, licensed under https://konn3kt.com/tos/
|
||||
set -euo pipefail
|
||||
|
||||
RES="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." >/dev/null && pwd)"
|
||||
SUPPORT="$HOME/Library/Application Support/IQVision"
|
||||
VENV="$SUPPORT/venv"
|
||||
PY="$VENV/bin/python"
|
||||
|
||||
mkdir -p "$SUPPORT"
|
||||
|
||||
if [ ! -x "$PY" ]; then
|
||||
echo "Creating IQ Vision environment (one time)…"
|
||||
/usr/bin/python3 -m venv "$VENV"
|
||||
"$PY" -m pip install --upgrade --quiet pip
|
||||
"$PY" -m pip install --quiet numpy "opencv-python-headless>=4.8" rumps
|
||||
fi
|
||||
|
||||
export PYTHONPATH="$RES:$RES/tinygrad_repo"
|
||||
export DEV=METAL
|
||||
exec "$PY" "$RES/tools/iqmacvisiond/menubar.py"
|
||||
80
tools/iqmacvisiond/menubar.py
Executable file
80
tools/iqmacvisiond/menubar.py
Executable file
@@ -0,0 +1,80 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Copyright © IQ.Lvbs, apart of Project Teal Lvbs, All Rights Reserved, licensed under https://konn3kt.com/tos/
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
import rumps
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
STATUS_URL = "http://127.0.0.1:51995/status.json"
|
||||
DASHBOARD_URL = "http://127.0.0.1:51995"
|
||||
|
||||
|
||||
class IQVisionApp(rumps.App):
|
||||
def __init__(self):
|
||||
super().__init__("IQ Vision", title="◎", quit_button=None)
|
||||
self.item_state = rumps.MenuItem("Starting…")
|
||||
self.item_device = rumps.MenuItem("Device: —")
|
||||
self.item_frames = rumps.MenuItem("Frames: —")
|
||||
self.item_exec = rumps.MenuItem("Inference: —")
|
||||
self.item_awake = rumps.MenuItem("Keep awake: —")
|
||||
self.menu = [
|
||||
self.item_state, None,
|
||||
self.item_device, self.item_frames, self.item_exec, self.item_awake, None,
|
||||
rumps.MenuItem("Open Dashboard", callback=self.open_dashboard),
|
||||
rumps.MenuItem("Quit IQ Vision", callback=self.quit_app),
|
||||
]
|
||||
self.proc: subprocess.Popen | None = None
|
||||
self._start_server()
|
||||
self.timer = rumps.Timer(self.refresh, 1)
|
||||
self.timer.start()
|
||||
|
||||
def _start_server(self) -> None:
|
||||
env = dict(os.environ)
|
||||
self.proc = subprocess.Popen([sys.executable, str(HERE / "server.py")], env=env)
|
||||
|
||||
def refresh(self, _) -> None:
|
||||
if self.proc is not None and self.proc.poll() is not None:
|
||||
self.title = "◎!"
|
||||
self.item_state.title = "Server stopped — reopen the app"
|
||||
return
|
||||
try:
|
||||
with urllib.request.urlopen(STATUS_URL, timeout=0.8) as r:
|
||||
s = json.load(r)
|
||||
except Exception:
|
||||
self.title = "◎"
|
||||
self.item_state.title = "Warming up…"
|
||||
return
|
||||
live = s.get("connected") and s.get("fresh")
|
||||
self.title = "◉" if live else "◎"
|
||||
self.item_state.title = "Connected" if live else "Waiting for device"
|
||||
self.item_device.title = f"Device: {s.get('peer') or '—'}"
|
||||
self.item_frames.title = f"Frames: {s.get('infer_count', 0):,}"
|
||||
p50, p99 = s.get("exec_p50_ms", 0.0), s.get("exec_p99_ms", 0.0)
|
||||
self.item_exec.title = f"Inference: {p50:.0f} / {p99:.0f} ms" if p50 else "Inference: —"
|
||||
self.item_awake.title = f"Keep awake: {'on' if s.get('awake') else 'off'}"
|
||||
|
||||
def open_dashboard(self, _) -> None:
|
||||
subprocess.Popen(["open", DASHBOARD_URL])
|
||||
|
||||
def quit_app(self, _) -> None:
|
||||
if self.proc is not None:
|
||||
self.proc.terminate()
|
||||
try:
|
||||
self.proc.wait(timeout=3)
|
||||
except subprocess.TimeoutExpired:
|
||||
self.proc.kill()
|
||||
rumps.quit_application()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
IQVisionApp().run()
|
||||
290
tools/iqmacvisiond/server.py
Executable file
290
tools/iqmacvisiond/server.py
Executable file
@@ -0,0 +1,290 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Copyright © IQ.Lvbs, apart of Project Teal Lvbs, All Rights Reserved, licensed under https://konn3kt.com/tos/
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import signal
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
os.environ.setdefault("DEV", "METAL")
|
||||
os.environ.setdefault("JIT_BATCH_SIZE", "0")
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
from openpilot.iqpilot.iqvd_private_src.offload.protocol import (
|
||||
DISCOVERY_MAGIC, DISCOVERY_REPLY, DISCOVERY_PORT_DEFAULT, MSG_HELLO, MSG_HELLO_ACK, MSG_INFER,
|
||||
MSG_PING, MSG_PONG, MSG_RESULT, ProtocolError, recv_msg, send_msg,
|
||||
)
|
||||
from openpilot.iqpilot.iqvd_private_src.offload.perception import Detector
|
||||
|
||||
log = logging.getLogger("iqmacvisiond")
|
||||
|
||||
DEFAULT_PORT = 51998
|
||||
STATUS_PORT = 51995
|
||||
MODEL_NAME = "yolov8n"
|
||||
SESSION_IDLE_TIMEOUT_S = 8.0
|
||||
|
||||
STATUS: dict = {
|
||||
"connected": False, "peer": "", "model": MODEL_NAME,
|
||||
"exec_p50_ms": 0.0, "exec_p99_ms": 0.0, "infer_count": 0,
|
||||
"last_seen": 0.0, "awake": False,
|
||||
}
|
||||
|
||||
|
||||
class KeepAwake:
|
||||
def __init__(self):
|
||||
self._proc: subprocess.Popen | None = None
|
||||
|
||||
def start(self) -> None:
|
||||
if sys.platform != "darwin" or self._proc is not None:
|
||||
return
|
||||
try:
|
||||
self._proc = subprocess.Popen(["caffeinate", "-dimsu"])
|
||||
STATUS["awake"] = True
|
||||
log.info("keep-awake active (caffeinate pid=%d)", self._proc.pid)
|
||||
except OSError:
|
||||
log.warning("caffeinate unavailable; display may sleep")
|
||||
|
||||
def stop(self) -> None:
|
||||
if self._proc is not None:
|
||||
self._proc.terminate()
|
||||
self._proc = None
|
||||
STATUS["awake"] = False
|
||||
|
||||
|
||||
class DiscoveryResponder:
|
||||
def __init__(self, tcp_port: int, disc_port: int = DISCOVERY_PORT_DEFAULT):
|
||||
self.tcp_port = tcp_port
|
||||
self.disc_port = disc_port
|
||||
|
||||
def start(self) -> None:
|
||||
threading.Thread(target=self._serve, daemon=True).start()
|
||||
|
||||
def _serve(self) -> None:
|
||||
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
|
||||
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
try:
|
||||
sock.bind(("0.0.0.0", self.disc_port))
|
||||
except OSError:
|
||||
log.exception("discovery bind failed on udp/%d", self.disc_port)
|
||||
return
|
||||
reply = DISCOVERY_REPLY + f":{self.tcp_port}".encode()
|
||||
log.info("discovery responder on udp/%d -> tcp/%d", self.disc_port, self.tcp_port)
|
||||
while True:
|
||||
try:
|
||||
data, addr = sock.recvfrom(256)
|
||||
except OSError:
|
||||
continue
|
||||
if data.startswith(DISCOVERY_MAGIC):
|
||||
try:
|
||||
sock.sendto(reply, addr)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
class StatusServer:
|
||||
def __init__(self, port: int = STATUS_PORT):
|
||||
self.port = port
|
||||
|
||||
def start(self) -> None:
|
||||
threading.Thread(target=self._serve, daemon=True).start()
|
||||
|
||||
def _serve(self) -> None:
|
||||
from http.server import BaseHTTPRequestHandler, HTTPServer
|
||||
|
||||
class H(BaseHTTPRequestHandler):
|
||||
def log_message(self, *a):
|
||||
pass
|
||||
|
||||
def _send(self, code, ctype, body):
|
||||
self.send_response(code)
|
||||
self.send_header("Content-Type", ctype)
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def do_GET(self):
|
||||
if self.path.startswith("/status.json"):
|
||||
st = dict(STATUS)
|
||||
st["fresh"] = (time.time() - st["last_seen"]) < 6 if st["last_seen"] else False
|
||||
self._send(200, "application/json", json.dumps(st).encode())
|
||||
else:
|
||||
self._send(200, "text/html; charset=utf-8", _STATUS_HTML.encode())
|
||||
|
||||
try:
|
||||
HTTPServer(("127.0.0.1", self.port), H).serve_forever()
|
||||
except OSError:
|
||||
log.exception("status server failed on %d", self.port)
|
||||
|
||||
|
||||
_STATUS_HTML = """<!doctype html><html><head><meta charset=utf-8>
|
||||
<title>IQ Vision</title><meta name=viewport content="width=device-width,initial-scale=1">
|
||||
<style>
|
||||
:root{color-scheme:dark}
|
||||
body{margin:0;font:15px -apple-system,system-ui,sans-serif;background:#0b0d10;color:#e6e9ef}
|
||||
.wrap{max-width:520px;margin:0 auto;padding:32px 20px}
|
||||
h1{font-size:20px;margin:0 0 20px;display:flex;align-items:center;gap:10px}
|
||||
.dot{width:12px;height:12px;border-radius:50%;background:#555}
|
||||
.dot.green{background:#28d2c8;box-shadow:0 0 10px #28d2c8}
|
||||
.dot.red{background:#e74c3c}
|
||||
.row{display:flex;justify-content:space-between;padding:12px 0;border-bottom:1px solid #1c2027}
|
||||
.k{color:#8a91a0}.v{font-variant-numeric:tabular-nums}
|
||||
</style></head><body><div class=wrap>
|
||||
<h1><span class=dot id=dot></span><span id=title>IQ Vision</span></h1>
|
||||
<div class=row><span class=k>Device</span><span class="v" id=peer>—</span></div>
|
||||
<div class=row><span class=k>Model</span><span class="v" id=model>—</span></div>
|
||||
<div class=row><span class=k>Inference (p50 / p99)</span><span class="v" id=exec>—</span></div>
|
||||
<div class=row><span class=k>Frames served</span><span class="v" id=count>—</span></div>
|
||||
<div class=row><span class=k>Keep awake</span><span class="v" id=awake>—</span></div>
|
||||
</div><script>
|
||||
async function tick(){
|
||||
try{
|
||||
const s=await (await fetch('/status.json')).json();
|
||||
const live=s.connected&&s.fresh;
|
||||
document.getElementById('dot').className='dot '+(live?'green':'red');
|
||||
document.getElementById('title').textContent=live?'IQ Vision — connected':'IQ Vision — waiting for device';
|
||||
document.getElementById('peer').textContent=s.peer||'not connected';
|
||||
document.getElementById('model').textContent=s.model||'—';
|
||||
document.getElementById('exec').textContent=s.exec_p50_ms?`${s.exec_p50_ms.toFixed(1)} / ${s.exec_p99_ms.toFixed(1)} ms`:'—';
|
||||
document.getElementById('count').textContent=s.infer_count?s.infer_count.toLocaleString():'—';
|
||||
document.getElementById('awake').textContent=s.awake?'on':'off';
|
||||
}catch(e){document.getElementById('dot').className='dot red';}
|
||||
}
|
||||
tick();setInterval(tick,1000);
|
||||
</script></body></html>"""
|
||||
|
||||
|
||||
class Session:
|
||||
def __init__(self, conn: socket.socket, detector: Detector):
|
||||
self.conn = conn
|
||||
self.detector = detector
|
||||
try:
|
||||
self.peer = conn.getpeername()[0]
|
||||
except OSError:
|
||||
self.peer = ""
|
||||
self.infer_count = 0
|
||||
self.exec_ms: list[float] = []
|
||||
|
||||
def handshake(self) -> bool:
|
||||
msg_type, header, _ = recv_msg(self.conn)
|
||||
if msg_type != MSG_HELLO:
|
||||
raise ProtocolError(f"expected HELLO, got {msg_type}")
|
||||
send_msg(self.conn, MSG_HELLO_ACK, {"ok": True, "model": MODEL_NAME, "hostname": socket.gethostname()})
|
||||
STATUS.update(connected=True, peer=self.peer, last_seen=time.time())
|
||||
log.info("device connected: %s dongle=%s", self.peer, header.get("dongle_id", ""))
|
||||
return True
|
||||
|
||||
def serve(self) -> None:
|
||||
while True:
|
||||
msg_type, header, blob = recv_msg(self.conn)
|
||||
if msg_type == MSG_INFER:
|
||||
self._infer(header, blob)
|
||||
elif msg_type == MSG_PING:
|
||||
send_msg(self.conn, MSG_PONG, {})
|
||||
else:
|
||||
raise ProtocolError(f"unexpected message type {msg_type}")
|
||||
|
||||
def _infer(self, header: dict, jpeg: bytes) -> None:
|
||||
st = time.perf_counter()
|
||||
tracks = []
|
||||
try:
|
||||
rgb = cv2.imdecode(np.frombuffer(jpeg, np.uint8), cv2.IMREAD_COLOR)
|
||||
if rgb is not None:
|
||||
tracks = self.detector.detect(cv2.cvtColor(rgb, cv2.COLOR_BGR2RGB))
|
||||
except Exception:
|
||||
log.exception("inference failed for frame %s", header.get("frame_id"))
|
||||
dt = (time.perf_counter() - st) * 1e3
|
||||
send_msg(self.conn, MSG_RESULT, {"frame_id": header.get("frame_id", 0), "tracks": tracks,
|
||||
"exec_ms": dt})
|
||||
self.infer_count += 1
|
||||
self.exec_ms.append(dt)
|
||||
if len(self.exec_ms) > 400:
|
||||
del self.exec_ms[:200]
|
||||
if self.infer_count % 10 == 0 or self.infer_count == 1:
|
||||
recent = self.exec_ms[-200:]
|
||||
STATUS.update(connected=True, infer_count=self.infer_count, last_seen=time.time(),
|
||||
exec_p50_ms=float(np.percentile(recent, 50)),
|
||||
exec_p99_ms=float(np.percentile(recent, 99)))
|
||||
|
||||
|
||||
def _weights_ok() -> bool:
|
||||
from openpilot.iqpilot.iqvd_private_src.offload.perception import _weights_dir
|
||||
return (_weights_dir() / "yolov8n.safetensors").exists()
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--host", default="0.0.0.0")
|
||||
parser.add_argument("--port", type=int, default=DEFAULT_PORT)
|
||||
parser.add_argument("--no-keep-awake", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(name)s %(levelname)s %(message)s")
|
||||
|
||||
if not _weights_ok():
|
||||
log.error("yolov8n.safetensors not found; the app ships the model with it")
|
||||
sys.exit(1)
|
||||
|
||||
keep_awake = KeepAwake()
|
||||
if not args.no_keep_awake:
|
||||
keep_awake.start()
|
||||
|
||||
def _shutdown(*_):
|
||||
keep_awake.stop()
|
||||
sys.exit(0)
|
||||
try:
|
||||
signal.signal(signal.SIGTERM, _shutdown)
|
||||
signal.signal(signal.SIGINT, _shutdown)
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
t0 = time.perf_counter()
|
||||
detector = Detector(None)
|
||||
for _ in range(3):
|
||||
detector.detect(np.zeros((416, 640, 3), dtype=np.uint8))
|
||||
log.info("model warm in %.1fs", time.perf_counter() - t0)
|
||||
|
||||
DiscoveryResponder(args.port).start()
|
||||
StatusServer().start()
|
||||
log.info("status dashboard on http://127.0.0.1:%d", STATUS_PORT)
|
||||
|
||||
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
server.bind((args.host, args.port))
|
||||
server.listen(1)
|
||||
log.info("READY listening on %s:%d", args.host, args.port)
|
||||
|
||||
try:
|
||||
while True:
|
||||
conn, addr = server.accept()
|
||||
conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
|
||||
conn.settimeout(SESSION_IDLE_TIMEOUT_S)
|
||||
try:
|
||||
session = Session(conn, detector)
|
||||
if session.handshake():
|
||||
session.serve()
|
||||
except (ConnectionError, ProtocolError, OSError) as e:
|
||||
log.info("session ended: %s", e)
|
||||
finally:
|
||||
conn.close()
|
||||
STATUS.update(connected=False, peer="")
|
||||
finally:
|
||||
keep_awake.stop()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
117
tools/iqmacvisiond/test_offload.py
Executable file
117
tools/iqmacvisiond/test_offload.py
Executable file
@@ -0,0 +1,117 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Copyright © IQ.Lvbs, apart of Project Teal Lvbs, All Rights Reserved, licensed under https://konn3kt.com/tos/
|
||||
"""
|
||||
import socket
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(REPO_ROOT))
|
||||
|
||||
from openpilot.iqpilot.iqvd_private_src.offload import protocol
|
||||
from openpilot.iqpilot.iqvd_private_src.offload.client import VisionClient, discover_server
|
||||
from openpilot.iqpilot.iqvd_private_src.offload.geometry import pixel_to_ground, tracks_to_objects
|
||||
|
||||
|
||||
def test_protocol_roundtrip():
|
||||
a, b = socket.socketpair()
|
||||
protocol.send_msg(a, protocol.MSG_INFER, {"frame_id": 7, "w": 640}, b"\x00\x01\x02payload")
|
||||
mt, header, blob = protocol.recv_msg(b)
|
||||
assert mt == protocol.MSG_INFER
|
||||
assert header == {"frame_id": 7, "w": 640}
|
||||
assert blob == b"\x00\x01\x02payload"
|
||||
a.close()
|
||||
b.close()
|
||||
|
||||
|
||||
def test_protocol_empty_blob():
|
||||
a, b = socket.socketpair()
|
||||
protocol.send_msg(a, protocol.MSG_HELLO_ACK, {"ok": True, "model": "yolov8n"})
|
||||
mt, header, blob = protocol.recv_msg(b)
|
||||
assert mt == protocol.MSG_HELLO_ACK and header["model"] == "yolov8n" and blob == b""
|
||||
a.close()
|
||||
b.close()
|
||||
|
||||
|
||||
def test_geometry_center_projects_forward():
|
||||
intr = np.array([[900.0, 0.0, 960.0], [0.0, 900.0, 604.0], [0.0, 0.0, 1.0]])
|
||||
device_from_calib = np.eye(3)
|
||||
p = pixel_to_ground(960.0, 900.0, intr, device_from_calib, 1.22)
|
||||
assert p is not None
|
||||
assert p[0] > 0
|
||||
assert abs(p[1]) < 1.0
|
||||
|
||||
|
||||
def test_geometry_above_horizon_none():
|
||||
intr = np.array([[900.0, 0.0, 960.0], [0.0, 900.0, 604.0], [0.0, 0.0, 1.0]])
|
||||
assert pixel_to_ground(960.0, 100.0, intr, np.eye(3), 1.22) is None
|
||||
|
||||
|
||||
def test_tracks_to_objects():
|
||||
intr = np.array([[900.0, 0.0, 960.0], [0.0, 900.0, 604.0], [0.0, 0.0, 1.0]])
|
||||
tracks = [{"x1": 0.45, "y1": 0.5, "x2": 0.55, "y2": 0.75, "prob": 0.9, "label": "car"}]
|
||||
objs = tracks_to_objects(tracks, 1928, 1208, intr, [0.0, 0.0, 0.0])
|
||||
assert len(objs) == 1
|
||||
assert objs[0]["x"] > 0 and objs[0]["label"] == "car"
|
||||
|
||||
|
||||
class _StubDetector:
|
||||
def detect(self, rgb):
|
||||
return [{"x1": 0.1, "y1": 0.2, "x2": 0.3, "y2": 0.5, "prob": 0.8, "label": "car"}]
|
||||
|
||||
|
||||
def _run_server(port, ready):
|
||||
from openpilot.iqpilot.iqvd_private_src.offload import protocol as p
|
||||
import tools.iqmacvisiond.server as srv
|
||||
disc = srv.DiscoveryResponder(port)
|
||||
disc.start()
|
||||
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
server.bind(("0.0.0.0", port))
|
||||
server.listen(1)
|
||||
ready.set()
|
||||
conn, _ = server.accept()
|
||||
conn.settimeout(5)
|
||||
session = srv.Session(conn, _StubDetector())
|
||||
session.handshake()
|
||||
try:
|
||||
session.serve()
|
||||
except (p.ProtocolError, OSError):
|
||||
pass
|
||||
|
||||
|
||||
def test_discovery_and_loopback():
|
||||
port = 52050
|
||||
ready = threading.Event()
|
||||
threading.Thread(target=_run_server, args=(port, ready), daemon=True).start()
|
||||
assert ready.wait(5)
|
||||
time.sleep(0.2)
|
||||
|
||||
found = discover_server(timeout=2.0)
|
||||
assert found is not None, "discovery failed"
|
||||
assert found[1] == port
|
||||
|
||||
import cv2
|
||||
ok, jpeg = cv2.imencode(".jpg", np.zeros((400, 640, 3), dtype=np.uint8))
|
||||
assert ok
|
||||
|
||||
client = VisionClient("test-dongle")
|
||||
assert client.connect(), "connect failed"
|
||||
meta = {"frame_id": 42, "wide": False, "w": 640, "h": 400}
|
||||
tracks = client.infer(jpeg.tobytes(), meta)
|
||||
assert tracks is not None and len(tracks) == 1
|
||||
assert tracks[0]["label"] == "car"
|
||||
client.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
fns = [v for k, v in sorted(globals().items()) if k.startswith("test_") and callable(v)]
|
||||
for fn in fns:
|
||||
fn()
|
||||
print(f"ok {fn.__name__}")
|
||||
print(f"\n{len(fns)} passed")
|
||||
Reference in New Issue
Block a user