pycapnp/README.md

pycapnp for IQPilot

IQ.Lvbs maintains this fork at https://git.konn3kt.com/IQ.Lvbs/pycapnp. It extends comma's minimal branch at caee18aeef0a9a54ad8f98292a325a1675a43bec. The upstream licenses and history are retained.

IQPilot additions:

• Native NumPy conversion for floating-point list readers and builders through __array__, retaining inferred dtype, explicit casts and warnings, writable independent output, and rejection of unsupported copy=False requests.
• to_float64_bytes() for an owned native-endian numeric snapshot without constructing intermediate Python scalar objects. Non-numeric values are rejected.
• Direct serialization into the final Python bytes object, preserving segment framing and byte identity while removing the intermediate flat-array copy.

NumPy is optional at runtime and loaded lazily only when its array protocol is used. The validation suite additionally requires NumPy. Serialization reference hashes were generated by the unmodified comma commit above and cover one and multiple segments through megabyte payloads. The native helpers do not borrow message memory.

A serialization-only fork of pycapnp. The minimal branch starts at upstream commit a0cb5cdf0673481f2f9850541f4d42b89698c476, including the from_dict reference-cycle fix in upstream PR #407.

The supported surface is based on openpilot at 7f6f13997c3c9b8e27e1581583f61e3fcabc5151, including its opendbc checkout:

• Explicit schema loading with capnp.load() and schema imports.
• Dynamic structs, lists, enums, unions, nested groups, and constants.
• Message construction, field access (including cached _get_by_field and _set_by_field), dictionaries, reader/builder copies, and pickling.
• Unpacked to_bytes(), from_bytes(), and read_multiple_bytes(), including traversal/nesting limits and readers that retain their underlying message.
• Schema reflection for cereal, CAN conversion, WebRTC, fuzzing, and replay tools.

Removed: RPC/capabilities, promises, KJ event loops, asyncio/network streams, packed serialization, file-descriptor I/O, segment APIs, borrowed Data views, custom allocators, orphans/resizable lists, AnyPointer wrappers, type registration, the Python schema import hook, and the Cython code generator. Their examples, tests, docs, dependencies, and unsupported-platform CI were removed too. Also removed are generated per-schema Reader/Builder classes, synthetic .Union enums, _has_by_field/_init_by_field, allocation-size overrides, from_bytes(builder=True), and to_dict ordering/base64 options. Incoming base64 Data values in from_dict remain supported. Unused schema reset/metadata helpers, schema equality, _which_str, and legacy exception arguments were removed too. remove_import_hook() remains a no-op for cereal/opendbc compatibility.

This is intentionally not a full upstream API replacement. The import and package names remain capnp and pycapnp. It must replace the installed pycapnp, not be installed alongside another distribution providing capnp.

Build and test

Targets: CPython 3.12, Linux x86_64/aarch64, and macOS arm64. A C++17 compiler and CMake are required to build the vendored library.

uv venv --python 3.12
uv pip install cython setuptools wheel pytest numpy build
.venv/bin/python setup.py build_ext --inplace
.venv/bin/python -m pytest
.venv/bin/python -m build

The C++ library is vendored under vendor/capnproto, based on Cap'n Proto 1.4.0. Every build compiles and statically links this copy; it does not search for a system Cap'n Proto installation or download sources. The source archive includes the vendored files, and wheels contain the compiled library. Runtime schema parsing remains included, so cereal schemas need no generated Python bindings. See the vendor README for provenance and the removed features.

The retained upstream tests cover message construction, schema loading, reflection, binary fixtures, serialization, and exceptions. Added lifetime tests check kwargs construction with GC disabled and readers surviving their input or iterator. Optional integration tests use real openpilot schemas and exercise messaging, CAN conversion, WebRTC reflection, LogReader, pickling, and replay:

# Run in an environment with openpilot's dependencies and this fork installed.
OPENPILOT_PATH=/path/to/openpilot python -m pytest test/test_openpilot.py
python -m pytest /path/to/openpilot/openpilot/cereal/messaging/tests \
  /path/to/openpilot/openpilot/tools/lib/tests/test_logreader.py

See LICENSE.md for the upstream BSD license and attribution.