IQ.Pilot Prebuilt Release @ ab07000

msgq_repo/.github/workflows/tests.yml

@@ -0,0 +1,24 @@
+name: tests
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref != 'refs/heads/master' && github.ref || github.run_id }}-${{ github.event_name }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    name: ./test.sh
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: ['macos-latest', 'ubuntu-latest']
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@v4
+      - run: ./test.sh

msgq_repo/.gitignore

@@ -0,0 +1,19 @@
+/gen/
+*.tmp
+*.pyc
+__pycache__
+.*.swp
+.*.swo
+*.os
+*.so
+*.o
+*.a
+uv.lock
+catch2/
+
+test_runner
+
+libmessaging.*
+libmessaging_shared.*
+.sconsign.dblite
+.mypy_cache/

msgq_repo/README.md

@@ -0,0 +1,54 @@
+# MSGQ: A lock free single producer multi consumer message queue
+
+## What is this library?
+MSGQ is a generic high performance IPC pub sub system with a single publisher and multiple subscribers. MSGQ is designed to be a high performance replacement for ZMQ-like SUB/PUB patterns. It uses a ring buffer in shared memory to efficiently read and write data. Each read requires a copy. Writing can be done without a copy, as long as the size of the data is known in advance. While MSGQ is the core of this library, this library also allows replacing the MSGQ backend with ZMQ or a spoofed implementation that can be used for deterministic testing. This library also contains visionipc, an IPC system specifically for large contiguous buffers (like images/video).
+
+## Storage
+The storage for the queue consists of an area of metadata, and the actual buffer. The metadata contains:
+
+1. A counter to the number of readers that are active
+2. A pointer to the head of the queue for writing. From now on referred to as *write pointer*
+3. A cycle counter for the writer. This counter is incremented when the writer wraps around
+4. N pointers, pointing to the current read position for all the readers. From now on referred to as *read pointer*
+5. N counters,  counting the number of cycles for all the readers
+6. N booleans, indicating validity for all the readers. From now on referred to as *validity flag*
+
+The counter and the pointer are both 32 bit values, packed into 64 bit so they can be read and written atomically.
+
+The data buffer is a ring buffer. All messages are prefixed by an 8 byte size field, followed by the data. A size of -1 indicates a wrap-around, and means the next message is stored at the beginning of the buffer.
+
+
+## Writing
+Writing involves the following steps:
+
+1. Check if the area that is to be written overlaps with any of the read pointers, mark those readers as invalid by clearing the validity flag.
+2. Write the message
+3. Increase the write pointer by the size of the message
+
+In case there is not enough space at the end of the buffer, a special empty message with a prefix of -1 is written. The cycle counter is incremented by one. In this case step 1 will check there are no read pointers pointing to the remainder of the buffer. Then another write cycle will start with the actual message.
+
+There always needs to be 8 bytes of empty space at the end of the buffer. By doing this there is always space to write the -1.
+
+## Reset reader
+When the reader is lagging too much behind the read pointer becomes invalid and no longer points to the beginning of a valid message. To reset a reader to the current write pointer, the following steps are performed:
+
+1. Set valid flag
+2. Set read cycle counter to that of the writer
+3. Set read pointer to write pointer
+
+## Reading
+Reading involves the following steps:
+
+1. Read the size field at the current read pointer
+2. Read the validity flag
+3. Copy the data out of the buffer
+4. Increase the read pointer by the size of the message
+5. Check the validity flag again
+
+Before starting the copy, the valid flag is checked. This is to prevent a race condition where the size prefix was invalid, and the read could read outside of the buffer. Make sure that step 1 and 2 are not reordered by your compiler or CPU.
+
+If a writer overwrites the data while it's being copied out, the data will be invalid. Therefore the validity flag is also checked after reading it. The order of step 4 and 5 does not matter.
+
+If at steps 2 or 5 the validity flag is not set, the reader is reset. Any data that was already read is discarded. After the reader is reset, the reading starts from the beginning.
+
+If a message with size -1 is encountered, step 3 and 4 are replaced by increasing the cycle counter and setting the read pointer to the beginning of the buffer. After that another read is performed.

msgq_repo/codecov.yml

@@ -0,0 +1,8 @@
+comment: false
+coverage:
+  status:
+    project:
+      default:
+        informational: true
+    patch: off
+

msgq_repo/lefthook.yml

@@ -0,0 +1,32 @@
+output:
+  - meta           # Print lefthook version
+  - summary        # Print summary block (successful and failed steps)
+  - empty_summary  # Print summary heading when there are no steps to run
+  - success        # Print successful steps
+  - failure        # Print failed steps printing
+  - execution      # Print any execution logs
+  #- execution_out  # Print execution output
+  #- execution_info # Print `EXECUTE > ...` logging
+  - skips          # Print "skip" (i.e. no files matched)
+
+test:
+  parallel: true
+  commands:
+    # *** static analysis ***
+    ruff:
+      run: ruff check .
+    ty:
+      run: ty check .
+    codespell:
+      run: codespell {files} -L ned,stdio,master --builtin clear,rare,informal,usage,code,names,en-GB_to_en-US -S uv.lock,*_pyx.cpp,catch2*
+      files: git ls-tree -r HEAD --name-only
+    cppcheck:
+      run: cppcheck --error-exitcode=1 --inline-suppr --language=c++ --force --quiet -j4 --check-level=exhaustive $(git ls-files '*.cc' | grep -v -E '(msgq_tests|test_runner)\.cc')
+    cpplint:
+      run: cpplint --exclude=msgq/catch2/ --exclude=msgq/ipc_pyx.cpp --exclude=msgq/visionipc/visionipc_pyx.cpp --recursive --quiet --counting=detailed --linelength=240 --filter=-build,-legal,-readability,-runtime,-whitespace,+build/include_subdir,+build/forward_decl,+build/include_what_you_use,+build/deprecated,+whitespace/comma,+whitespace/line_length,+whitespace/empty_if_body,+whitespace/empty_loop_body,+whitespace/empty_conditional_body,+whitespace/forcolon,+whitespace/parens,+whitespace/semicolon,+whitespace/tab,+readability/braces msgq/
+
+    # *** tests ***
+    test_runner:
+      run: msgq/test_runner
+    pytest:
+      run: pytest

msgq_repo/msgq/.gitignore

@@ -0,0 +1 @@
+ipc_pyx.cpp

msgq_repo/msgq/__init__.py

@@ -0,0 +1,63 @@
+# must be built with scons
+from msgq.ipc_pyx import Context, Poller, SubSocket, PubSocket, SocketEventHandle, toggle_fake_events, \
+                                set_fake_prefix, get_fake_prefix, delete_fake_prefix, wait_for_one_event, \
+                                context_is_zmq
+from msgq.ipc_pyx import MultiplePublishersError, IpcError
+
+from typing import Optional, List, Union
+
+assert MultiplePublishersError
+assert IpcError
+assert toggle_fake_events
+assert set_fake_prefix
+assert get_fake_prefix
+assert delete_fake_prefix
+assert wait_for_one_event
+assert context_is_zmq
+
+NO_TRAVERSAL_LIMIT = 2**64-1
+
+context = Context()
+
+
+def fake_event_handle(endpoint: str, identifier: Optional[Union[str, bytes]] = None, override: bool = True, enable: bool = False) -> SocketEventHandle:
+  ident = identifier if identifier is not None else get_fake_prefix()
+  handle = SocketEventHandle(endpoint, ident, override)
+  if override:
+    handle.enabled = enable
+
+  return handle
+
+def pub_sock(endpoint: str, segment_size: int = 0) -> PubSocket:
+  sock = PubSocket()
+  sock.connect(context, endpoint, segment_size)
+  return sock
+
+
+def sub_sock(endpoint: str, poller: Optional[Poller] = None, addr: str = "127.0.0.1",
+             conflate: bool = False, timeout: Optional[int] = None, segment_size: int = 0) -> SubSocket:
+  sock = SubSocket()
+  sock.connect(context, endpoint, addr.encode('utf8'), conflate, segment_size)
+
+  if timeout is not None:
+    sock.setTimeout(timeout)
+
+  if poller is not None:
+    poller.registerSocket(sock)
+  return sock
+
+def drain_sock_raw(sock: SubSocket, wait_for_one: bool = False) -> List[bytes]:
+  """Receive all message currently available on the queue"""
+  ret: List[bytes] = []
+  while 1:
+    if wait_for_one and len(ret) == 0:
+      dat = sock.receive()
+    else:
+      dat = sock.receive(non_blocking=True)
+
+    if dat is None:
+      break
+
+    ret.append(dat)
+
+  return ret

msgq_repo/msgq/conftest.py

@@ -0,0 +1,14 @@
+import os
+import pytest
+import msgq
+
+@pytest.fixture(params=[False, True], ids=["msgq", "zmq"], autouse=True)
+def zmq_mode(request):
+  if request.param:
+    os.environ["ZMQ"] = "1"
+  else:
+    os.environ.pop("ZMQ", None)
+  msgq.context = msgq.Context()
+  assert msgq.context_is_zmq() == request.param
+  yield request.param
+  os.environ.pop("ZMQ", None)

msgq_repo/msgq/tests/test_fake.py

@@ -0,0 +1,189 @@
+import pytest
+import os
+import multiprocessing
+import platform
+import msgq
+from parameterized import parameterized_class
+from typing import Optional
+
+WAIT_TIMEOUT = 5
+
+
+@pytest.mark.skipif(condition=platform.system() == "Darwin", reason="Events not supported on macOS")
+class TestEvents:
+
+  def test_mutation(self):
+    handle = msgq.fake_event_handle("carState")
+    event = handle.recv_called_event
+
+    assert not event.peek()
+    event.set()
+    assert event.peek()
+    event.clear()
+    assert not event.peek()
+
+    del event
+
+  def test_wait(self):
+    handle = msgq.fake_event_handle("carState")
+    event = handle.recv_called_event
+
+    event.set()
+    try:
+      event.wait(WAIT_TIMEOUT)
+      assert event.peek()
+    except RuntimeError:
+      pytest.fail("event.wait() timed out")
+
+  def test_wait_multiprocess(self):
+    handle = msgq.fake_event_handle("carState")
+    event = handle.recv_called_event
+
+    def set_event_run():
+      event.set()
+
+    try:
+      p = multiprocessing.Process(target=set_event_run)
+      p.start()
+      event.wait(WAIT_TIMEOUT)
+      assert event.peek()
+    except RuntimeError:
+      pytest.fail("event.wait() timed out")
+
+    p.kill()
+
+  def test_wait_zero_timeout(self):
+    handle = msgq.fake_event_handle("carState")
+    event = handle.recv_called_event
+
+    try:
+      event.wait(0)
+      pytest.fail("event.wait() did not time out")
+    except RuntimeError:
+      assert not event.peek()
+
+
+@pytest.mark.skipif(condition=platform.system() == "Darwin", reason="FakeSockets not supported on macOS")
+@parameterized_class([{"prefix": None}, {"prefix": "test"}])
+class TestFakeSockets:
+  prefix: Optional[str] = None
+
+  def setup_method(self):
+    if "ZMQ" in os.environ:
+      pytest.skip("FakeSockets not supported on ZMQ")
+    msgq.toggle_fake_events(True)
+    if self.prefix is not None:
+      msgq.set_fake_prefix(self.prefix)
+    else:
+      msgq.delete_fake_prefix()
+
+  def teardown_method(self):
+    msgq.toggle_fake_events(False)
+    msgq.delete_fake_prefix()
+
+  def test_event_handle_init(self):
+    handle = msgq.fake_event_handle("controlsState", override=True)
+
+    assert not handle.enabled
+    assert handle.recv_called_event.fd >= 0
+    assert handle.recv_ready_event.fd >= 0
+
+  def test_non_managed_socket_state(self):
+    # non managed socket should have zero state
+    _ = msgq.pub_sock("ubloxGnss")
+
+    handle = msgq.fake_event_handle("ubloxGnss", override=False)
+
+    assert not handle.enabled
+    assert handle.recv_called_event.fd == 0
+    assert handle.recv_ready_event.fd == 0
+
+  def test_managed_socket_state(self):
+    # managed socket should not change anything about the state
+    handle = msgq.fake_event_handle("ubloxGnss")
+    handle.enabled = True
+
+    expected_enabled = handle.enabled
+    expected_recv_called_fd = handle.recv_called_event.fd
+    expected_recv_ready_fd = handle.recv_ready_event.fd
+
+    _ = msgq.pub_sock("ubloxGnss")
+
+    assert handle.enabled == expected_enabled
+    assert handle.recv_called_event.fd == expected_recv_called_fd
+    assert handle.recv_ready_event.fd == expected_recv_ready_fd
+
+  def test_sockets_enable_disable(self):
+    carState_handle = msgq.fake_event_handle("ubloxGnss", enable=True)
+    recv_called = carState_handle.recv_called_event
+    recv_ready = carState_handle.recv_ready_event
+
+    pub_sock = msgq.pub_sock("ubloxGnss")
+    sub_sock = msgq.sub_sock("ubloxGnss")
+
+    try:
+      carState_handle.enabled = True
+      recv_ready.set()
+      pub_sock.send(b"test")
+      _ = sub_sock.receive()
+      assert recv_called.peek()
+      recv_called.clear()
+
+      carState_handle.enabled = False
+      recv_ready.set()
+      pub_sock.send(b"test")
+      _ = sub_sock.receive()
+      assert not recv_called.peek()
+    except RuntimeError:
+      pytest.fail("event.wait() timed out")
+
+  def test_synced_pub_sub(self):
+    def daemon_repub_process_run():
+      pub_sock = msgq.pub_sock("ubloxGnss")
+      sub_sock = msgq.sub_sock("carState")
+
+      frame = -1
+      while True:
+        frame += 1
+        msg = sub_sock.receive(non_blocking=True)
+        if msg is None:
+          print("none received")
+          continue
+
+        bts = frame.to_bytes(8, 'little')
+        pub_sock.send(bts)
+
+    carState_handle = msgq.fake_event_handle("carState", enable=True)
+    recv_called = carState_handle.recv_called_event
+    recv_ready = carState_handle.recv_ready_event
+
+    p = multiprocessing.Process(target=daemon_repub_process_run)
+    p.start()
+
+    pub_sock = msgq.pub_sock("carState")
+    sub_sock = msgq.sub_sock("ubloxGnss")
+
+    try:
+      for i in range(10):
+        recv_called.wait(WAIT_TIMEOUT)
+        recv_called.clear()
+
+        if i == 0:
+          sub_sock.receive(non_blocking=True)
+
+        bts = i.to_bytes(8, 'little')
+        pub_sock.send(bts)
+
+        recv_ready.set()
+        recv_called.wait(WAIT_TIMEOUT)
+
+        msg = sub_sock.receive(non_blocking=True)
+        assert msg is not None
+        assert len(msg) == 8
+
+        frame = int.from_bytes(msg, 'little')
+        assert frame == i
+    except RuntimeError:
+      pytest.fail("event.wait() timed out")
+    finally:
+      p.kill()

msgq_repo/msgq/tests/test_messaging.py

@@ -0,0 +1,70 @@
+import os
+import random
+import time
+import string
+import msgq
+import pytest
+
+def random_sock():
+  return ''.join(random.choices(string.ascii_uppercase + string.digits, k=10))
+
+def random_bytes(length=1000):
+  return bytes([random.randrange(0xFF) for _ in range(length)])
+
+def zmq_sleep(t=1):
+  if "ZMQ" in os.environ:
+    time.sleep(t)
+
+class TestPubSubSockets:
+
+  def setup_method(self):
+    # ZMQ pub socket takes too long to die
+    # sleep to prevent multiple publishers error between tests
+    zmq_sleep()
+
+  def test_pub_sub(self):
+    sock = random_sock()
+    pub_sock = msgq.pub_sock(sock)
+    sub_sock = msgq.sub_sock(sock, conflate=False, timeout=None)
+    zmq_sleep(3)
+
+    for _ in range(1000):
+      msg = random_bytes()
+      pub_sock.send(msg)
+      recvd = sub_sock.receive()
+      assert msg == recvd
+
+  def test_conflate(self):
+    sock = random_sock()
+    pub_sock = msgq.pub_sock(sock)
+    for conflate in [True, False]:
+      num_msgs = random.randint(3, 10)
+      sub_sock = msgq.sub_sock(sock, conflate=conflate, timeout=None)
+      zmq_sleep()
+
+      sent_msgs = []
+      for __ in range(num_msgs):
+        msg = random_bytes()
+        pub_sock.send(msg)
+        sent_msgs.append(msg)
+      time.sleep(0.1)
+      recvd_msgs = msgq.drain_sock_raw(sub_sock)
+      if conflate:
+        assert len(recvd_msgs) == 1
+        assert recvd_msgs[0] == sent_msgs[-1]
+      else:
+        assert len(recvd_msgs) == len(sent_msgs)
+        for rec_msg, sent_msg in zip(recvd_msgs, sent_msgs):
+          assert rec_msg == sent_msg
+
+  @pytest.mark.flaky(retries=3, delay=1)
+  def test_receive_timeout(self):
+    sock = random_sock()
+    timeout = random.randrange(200)
+    sub_sock = msgq.sub_sock(sock, timeout=timeout)
+    zmq_sleep()
+
+    start_time = time.monotonic()
+    recvd = sub_sock.receive()
+    assert (time.monotonic() - start_time) < (timeout + 0.1)
+    assert recvd is None

msgq_repo/msgq/tests/test_poller.py

@@ -0,0 +1,138 @@
+import pytest
+import time
+import msgq
+import concurrent.futures
+
+SERVICE_NAME = 'myService'
+
+def poller():
+  context = msgq.Context()
+
+  p = msgq.Poller()
+
+  sub = msgq.SubSocket()
+  sub.connect(context, SERVICE_NAME)
+  p.registerSocket(sub)
+
+  socks = p.poll(10000)
+  r = [s.receive(non_blocking=True) for s in socks]
+
+  return r
+
+
+class TestPoller:
+  def test_poll_once(self):
+    context = msgq.Context()
+
+    pub = msgq.PubSocket()
+    pub.connect(context, SERVICE_NAME)
+
+    with concurrent.futures.ThreadPoolExecutor() as e:
+      poll = e.submit(poller)
+
+      time.sleep(0.1)  # Slow joiner syndrome
+
+      # Send message
+      pub.send(b"a")
+
+      # Wait for poll result
+      result = poll.result()
+
+    del pub
+    context.term()
+
+    assert result == [b"a"]
+
+  def test_poll_and_create_many_subscribers(self):
+    context = msgq.Context()
+
+    pub = msgq.PubSocket()
+    pub.connect(context, SERVICE_NAME)
+
+    with concurrent.futures.ThreadPoolExecutor() as e:
+      poll = e.submit(poller)
+
+      time.sleep(0.1)  # Slow joiner syndrome
+      c = msgq.Context()
+      for _ in range(10):
+        msgq.SubSocket().connect(c, SERVICE_NAME)
+
+      time.sleep(0.1)
+
+      # Send message
+      pub.send(b"a")
+
+      # Wait for poll result
+      result = poll.result()
+
+    del pub
+    context.term()
+
+    assert result == [b"a"]
+
+  def test_multiple_publishers_exception(self):
+    context = msgq.Context()
+
+    with pytest.raises(msgq.MultiplePublishersError):
+      pub1 = msgq.PubSocket()
+      pub1.connect(context, SERVICE_NAME)
+
+      pub2 = msgq.PubSocket()
+      pub2.connect(context, SERVICE_NAME)
+
+      pub1.send(b"a")
+
+    del pub1
+    del pub2
+    context.term()
+
+  def test_multiple_messages(self):
+    context = msgq.Context()
+
+    pub = msgq.PubSocket()
+    pub.connect(context, SERVICE_NAME)
+
+    sub = msgq.SubSocket()
+    sub.connect(context, SERVICE_NAME)
+
+    time.sleep(0.1)  # Slow joiner
+
+    for i in range(1, 100):
+      pub.send(b'a'*i)
+
+    msg_seen = False
+    i = 1
+    while True:
+      r = sub.receive(non_blocking=True)
+
+      if r is not None:
+        assert b'a'*i == r
+
+        msg_seen = True
+        i += 1
+
+      if r is None and msg_seen:  # ZMQ sometimes receives nothing on the first receive
+        break
+
+    del pub
+    del sub
+    context.term()
+
+  def test_conflate(self):
+    context = msgq.Context()
+
+    pub = msgq.PubSocket()
+    pub.connect(context, SERVICE_NAME)
+
+    sub = msgq.SubSocket()
+    sub.connect(context, SERVICE_NAME, conflate=True)
+
+    time.sleep(0.1)  # Slow joiner
+    pub.send(b'a')
+    pub.send(b'b')
+
+    assert b'b' == sub.receive()
+
+    del pub
+    del sub
+    context.term()

msgq_repo/msgq/visionipc/.gitignore

@@ -0,0 +1,2 @@
+visionipc_pyx.cpp
+*.so

msgq_repo/msgq/visionipc/__init__.py

@@ -0,0 +1,6 @@
+from msgq.visionipc.visionipc_pyx import VisionBuf, VisionIpcClient, VisionIpcServer, VisionStreamType, get_endpoint_name
+assert VisionBuf
+assert VisionIpcClient
+assert VisionIpcServer
+assert VisionStreamType
+assert get_endpoint_name

msgq_repo/msgq/visionipc/tests/test_visionipc.py

@@ -0,0 +1,107 @@
+import os
+import time
+import random
+from typing import Optional
+import numpy as np
+from msgq.visionipc import VisionIpcServer, VisionIpcClient, VisionStreamType
+
+def zmq_sleep(t=1):
+  if "ZMQ" in os.environ:
+    time.sleep(t)
+
+
+class TestVisionIpc:
+  server: VisionIpcServer
+  client: Optional[VisionIpcClient]
+
+  def setup_vipc(self, name, *stream_types, num_buffers=1, width=100, height=100, conflate=False):
+    self.server = VisionIpcServer(name)
+    for stream_type in stream_types:
+      self.server.create_buffers(stream_type, num_buffers, width, height)
+    self.server.start_listener()
+
+    if len(stream_types):
+      self.client = VisionIpcClient(name, stream_types[0], conflate)
+      assert self.client.connect(True)
+    else:
+      self.client = None
+
+    zmq_sleep()
+    return self.server, self.client
+
+  def test_connect(self):
+    self.setup_vipc("camerad", VisionStreamType.VISION_STREAM_ROAD)
+    assert self.client is not None
+    assert self.client.is_connected
+    del self.client
+    del self.server
+
+  def test_available_streams(self):
+    for k in range(4):
+      stream_types = set(random.choices([x.value for x in VisionStreamType], k=k))
+      self.setup_vipc("camerad", *stream_types)
+      available_streams = VisionIpcClient.available_streams("camerad", True)
+      assert available_streams == stream_types
+      del self.client
+      del self.server
+
+  def test_buffers(self):
+    width, height, num_buffers = 100, 200, 5
+    self.setup_vipc("camerad", VisionStreamType.VISION_STREAM_ROAD, num_buffers=num_buffers, width=width, height=height)
+    assert self.client is not None
+    assert self.client.width == width
+    assert self.client.height == height
+    assert self.client.buffer_len is not None and self.client.buffer_len > 0
+    assert self.client.num_buffers == num_buffers
+    del self.client
+    del self.server
+
+  def test_send_single_buffer(self):
+    self.setup_vipc("camerad", VisionStreamType.VISION_STREAM_ROAD)
+    assert self.client is not None
+    assert self.client.buffer_len is not None
+    buf = np.zeros(self.client.buffer_len, dtype=np.uint8)
+    buf.view('<i4')[0] = 1234
+    self.server.send(VisionStreamType.VISION_STREAM_ROAD, buf, frame_id=1337)
+
+    recv_buf = self.client.recv()
+    assert recv_buf is not None
+    assert recv_buf.data.view('<i4')[0] == 1234
+    assert self.client.frame_id == 1337
+    del self.client
+    del self.server
+
+  def test_no_conflate(self):
+    self.setup_vipc("camerad", VisionStreamType.VISION_STREAM_ROAD)
+    assert self.client is not None
+    assert self.client.buffer_len is not None
+    buf = np.zeros(self.client.buffer_len, dtype=np.uint8)
+    self.server.send(VisionStreamType.VISION_STREAM_ROAD, buf, frame_id=1)
+    self.server.send(VisionStreamType.VISION_STREAM_ROAD, buf, frame_id=2)
+
+    recv_buf = self.client.recv()
+    assert recv_buf is not None
+    assert self.client.frame_id == 1
+
+    recv_buf = self.client.recv()
+    assert recv_buf is not None
+    assert self.client.frame_id == 2
+    del self.client
+    del self.server
+
+  def test_conflate(self):
+    self.setup_vipc("camerad", VisionStreamType.VISION_STREAM_ROAD, conflate=True)
+    assert self.client is not None
+    assert self.client.buffer_len is not None
+    buf = np.zeros(self.client.buffer_len, dtype=np.uint8)
+    self.server.send(VisionStreamType.VISION_STREAM_ROAD, buf, frame_id=1)
+    self.server.send(VisionStreamType.VISION_STREAM_ROAD, buf, frame_id=2)
+
+    recv_buf = self.client.recv()
+    assert recv_buf is not None
+    assert self.client.frame_id == 2
+
+    recv_buf = self.client.recv()
+    assert recv_buf is None
+    del self.client
+    del self.server

msgq_repo/pyproject.toml

@@ -0,0 +1,56 @@
+[project]
+name = "msgq"
+version = "0.0.1"
+description = "Code powering the comma.ai panda"
+readme = "README.md"
+requires-python = ">=3.11,<3.13"
+license = {text = "MIT"}
+authors = [{name = "comma.ai"}]
+classifiers = [
+  "Natural Language :: English",
+  "Programming Language :: Python :: 3",
+  "Topic :: System :: Hardware",
+]
+dependencies = [
+  "setuptools",  # for distutils
+  "Cython",
+  "scons",
+  "ruff",
+  "parameterized",
+  "coverage",
+  "numpy",
+  "pytest",
+  "pytest-retry",
+  "cppcheck",
+  "cpplint",
+  "codespell",
+  "ty",
+  "lefthook",
+]
+
+# https://beta.ruff.rs/docs/configuration/#using-pyprojecttoml
+[tool.ruff]
+lint.select = ["E", "F", "W", "PIE", "C4", "ISC", "RUF100", "A"]
+lint.ignore = ["W292", "E741", "E402", "C408", "ISC003"]
+lint.flake8-implicit-str-concat.allow-multiline=false
+
+line-length = 160
+target-version="py311"
+
+[tool.ruff.lint.flake8-tidy-imports.banned-api]
+"pytest.main".msg = "pytest.main requires special handling that is easy to mess up!"
+"unittest".msg = "Use pytest"
+
+[tool.ty.src]
+exclude = ["site_scons/"]
+
+[tool.ty.rules]
+# Cython modules are compiled at build time, not available for static analysis
+unresolved-import = "ignore"
+
+[tool.pytest.ini_options]
+addopts = "--durations=10"
+testpaths = [
+  "msgq/tests",
+  "msgq/visionipc/tests",
+]

msgq_repo/setup.sh

@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+set -e
+
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
+cd $DIR
+
+PLATFORM=$(uname -s)
+
+echo "installing dependencies"
+if [[ $PLATFORM == "Darwin" ]]; then
+  if ! command -v python3 &>/dev/null || ! pkg-config --exists libzmq 2>/dev/null; then
+    export HOMEBREW_NO_AUTO_UPDATE=1
+    brew install python3 zeromq
+  fi
+elif [[ $PLATFORM == "Linux" ]]; then
+  # for AGNOS since we clear the apt lists
+  if [[ ! -d /"var/lib/apt/" ]]; then
+    sudo apt update
+  fi
+
+  sudo apt-get install -y --no-install-recommends \
+    curl ca-certificates \
+    libzmq3-dev \
+    ocl-icd-opencl-dev opencl-headers \
+    python3-dev python3-pip python3-venv
+else
+  echo "WARNING: unsupported platform. skipping apt/brew install."
+fi
+
+# catch2
+if [ ! -d $DIR/msgq/catch2/ ]; then
+  rm -rf /tmp/catch2/ $DIR/msgq/catch2/
+  git clone -b v2.x --depth 1 https://github.com/catchorg/Catch2.git /tmp/catch2
+  pushd /tmp/catch2
+  mv single_include/* $DIR/msgq/
+  popd
+fi
+
+if ! command -v uv &>/dev/null; then
+  echo "'uv' is not installed. Installing 'uv'..."
+  curl -LsSf https://astral.sh/uv/install.sh | sh
+
+  # doesn't require sourcing on all platforms
+  set +e
+  source $HOME/.local/bin/env
+  set -e
+fi
+
+export UV_PROJECT_ENVIRONMENT="$DIR/.venv"
+uv sync --all-extras
+source "$DIR/.venv/bin/activate"

msgq_repo/site_scons/site_tools/cython.py

@@ -0,0 +1,72 @@
+import re
+import SCons
+from SCons.Action import Action
+from SCons.Scanner import Scanner
+
+pyx_from_import_re = re.compile(r'^from\s+(\S+)\s+cimport', re.M)
+pyx_import_re = re.compile(r'^cimport\s+(\S+)', re.M)
+cdef_import_re = re.compile(r'^cdef extern from\s+.(\S+).:', re.M)
+
+
+def pyx_scan(node, env, path, arg=None):
+  contents = node.get_text_contents()
+
+  # from <module> cimport ...
+  matches = pyx_from_import_re.findall(contents)
+  # cimport <module>
+  matches += pyx_import_re.findall(contents)
+
+  # Modules can be either .pxd or .pyx files
+  files = [m.replace('.', '/') + '.pxd' for m in matches]
+  files += [m.replace('.', '/') + '.pyx' for m in matches]
+
+  # cdef extern from <file>
+  files += cdef_import_re.findall(contents)
+
+  # Handle relative imports
+  cur_dir = str(node.get_dir())
+  files = [cur_dir + f if f.startswith('/') else f for f in files]
+
+  # Filter out non-existing files (probably system imports)
+  files = [f for f in files if env.File(f).exists()]
+  return env.File(files)
+
+
+pyxscanner = Scanner(function=pyx_scan, skeys=['.pyx', '.pxd'], recursive=True)
+cythonAction = Action("$CYTHONCOM")
+
+
+def create_builder(env):
+  try:
+    cython = env['BUILDERS']['Cython']
+  except KeyError:
+    cython = SCons.Builder.Builder(
+      action=cythonAction,
+      emitter={},
+      suffix=cython_suffix_emitter,
+      single_source=1
+    )
+    env.Append(SCANNERS=pyxscanner)
+    env['BUILDERS']['Cython'] = cython
+  return cython
+
+def cython_suffix_emitter(env, source):
+  return "$CYTHONCFILESUFFIX"
+
+def generate(env):
+  env["CYTHON"] = "cythonize"
+  env["CYTHONCOM"] = "$CYTHON $CYTHONFLAGS $SOURCE"
+  env["CYTHONCFILESUFFIX"] = ".cpp"
+
+  c_file, _ = SCons.Tool.createCFileBuilders(env)
+
+  c_file.suffix['.pyx'] = cython_suffix_emitter
+  c_file.add_action('.pyx', cythonAction)
+
+  c_file.suffix['.py'] = cython_suffix_emitter
+  c_file.add_action('.py', cythonAction)
+
+  create_builder(env)
+
+def exists(env):
+  return True

msgq_repo/test.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+set -e
+
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
+cd $DIR
+
+source ./setup.sh
+
+# *** build ***
+scons -j8
+
+# *** lint + test ***
+lefthook run test
+
+# *** all done ***
+GREEN='\033[0;32m'
+NC='\033[0m'
+printf "\n${GREEN}All good!${NC} Finished build, lint, and test in ${SECONDS}s\n"