TwoWayPipe and basic asyncio support

Note: I've tried not to break any behaviour of the previously working APIs

Python API Changes / Additions
- capnp/lib/capnp.pyx
  * class _RemotePromise
    + [Added] cpdef _wait(self)
      = Exception raising code that used to be inside of wait(self)
    + [Modified] def wait(self)
      = Same functionality as before
    + [Added] async def a_wait(self)
      = Cannot use await as that's a reserved keyword
      = Uses pollRemote and asyncio.sleep(0) to make call asynchronous
  * class _TwoPartyVatNetwork
    + [Added] cdef _init_pipe(self, _TwoWayPipe pipe, Side side,
    schema_cpp.ReaderOptions opts)
      = Instanciates a TwoPartyVatNetwork using a TwoWayPipe (instead of
      using a file handle or connection as before)
  * class TwoPartyClient
    + [Modified] def __init__(self, socket=None, restorer=None,
    traversal_limit_in_words=None, nesting_limit=None)
      = Changes the socket parameter to be optional
      = If socket is not specified, default to using a TwoWayPipe
    + [Added] async def read(self, bufsize)
      = awaitable function that blocks until data has been read
      = bufsize defines the maximum amount of data to be read back
        (e.g. 4096 bytes)
      = Reads data from TwoWayPipe
    + [Added] def write(self, data)
      = Write data to TwoWayPipe
      = Not awaitable as the write interface of the TwoWayPipe doesn't
      have poll functionality
  * class TwoPartyServer
    + [Modified] def __init__(self, socket=None, restorer=None,
    server_socket=None, bootstrap=None, traversal_limit_in_words=None,
    nesting_limit=None)
      = Changes the socket parameter to be optional
      = If socket is not specified, default to using a TwoWayPipe
      = Simplified code by removing an else (self._connect)
    + [Added] async def read(self, bufsize)
      = awaitable function that blocks until data has been read
      = bufsize defines the maximum amount of data to be read back
        (e.g. 4096 bytes)
      = Reads data from TwoWayPipe
    + [Added] def write(self, data)
      = Write data to TwoWayPipe
      = Not awaitable as the write interface of the TwoWayPipe doesn't
      have poll functionality
    + [Added] async def poll_forever(self)
      = asyncio equivalent of run_forever()
  * class _TwoWayPipe
    + Wrapper class for TwoWayPipe

Other Additions
- capnp/helpers/asyncHelper.h
  * pollWaitScope
    + Pumps the kj event handler
    + Used for the TwoWayServer
  * pollRemote
    + Polls a remote promise
    + i.e. a capnp RPC call
- capnp/helpers/asyncIoHelper.h
  * AsyncIoStreamReadHelper
    + I wasn't able to figure out Promise[size_t] using Cython so this was
    the next best thing I could think of doing
    + Was needed to handle read polling from a read promise
      = Polling is used for asyncio as kj waits need a wrapper to be
      compatible
- capnp/lib/capnp.pyx
  * makeTwoWayPipe
    + Wrapper for kj newTwoWayPipe function
  * poll_once
    + Single pump of the kj event handler (used with pollWaitScope)

TwoWayClient Usage - TwoWayPipe
- See examples/async_client.py

TwoWayServer Usage - TwoWayPipe
- See examples/async_server.py

capnp/helpers/asyncIoHelper.h

Misc Changes
- Fixed thread_server.py and thread_client.py to use bootstrap instead
of ez_restore
- async_client.py and async_server.py examples
  * Uses the same thread.capnp as thread_client.py and thread_server.py
  * They are compatible, so you can mix and match client and server for
  compatibility testing
  * async_client.py and async_server.py require <address>:<port>
  formatting (unlike autodetection from thread_client.py and
  thread_server.py)
This commit is contained in:
Jacob Alexander
2019-09-16 21:34:29 -07:00
parent e73b63ddd1
commit 8915ef79f1
11 changed files with 407 additions and 41 deletions

View File

@@ -4,6 +4,7 @@
cdef extern from "capnp/helpers/checkCompiler.h":
pass
from libcpp cimport bool
from schema_cpp cimport Node, Data, StructNode, EnumNode, InterfaceNode, MessageBuilder, MessageReader, ReaderOptions
from capnp.helpers.non_circular cimport PythonInterfaceDynamicImpl, reraise_kj_exception, PyRefCounter, PyRestorer, PyEventPort, ErrorHandler
from capnp.includes.types cimport *
@@ -45,6 +46,7 @@ cdef extern from "kj/exception.h" namespace " ::kj":
cdef extern from "kj/memory.h" namespace " ::kj":
cdef cppclass Own[T]:
T& operator*()
T* get()
Own[TwoPartyVatNetwork] makeTwoPartyVatNetwork" ::kj::heap< ::capnp::TwoPartyVatNetwork>"(AsyncIoStream& stream, Side, ReaderOptions)
Own[PromiseFulfillerPair] copyPromiseFulfillerPair" ::kj::heap< ::kj::PromiseFulfillerPair<void> >"(PromiseFulfillerPair&)
Own[PyRefCounter] makePyRefCounter" ::kj::heap< PyRefCounter >"(PyObject *)
@@ -55,6 +57,7 @@ cdef extern from "kj/async.h" namespace " ::kj":
Promise(Promise)
Promise(T)
T wait(WaitScope)
bool poll(WaitScope)
# ForkedPromise<T> fork()
# Promise<T> exclusiveJoin(Promise<T>&& other)
# Promise[T] eagerlyEvaluate()
@@ -121,16 +124,21 @@ cdef inline Duration Nanoseconds(int64_t nanos):
cdef extern from "kj/async-io.h" namespace " ::kj":
cdef cppclass AsyncIoStream:
pass
Promise[size_t] read(void*, size_t, size_t)
Promise[void] write(const void*, size_t)
cdef cppclass LowLevelAsyncIoProvider:
# Own[AsyncInputStream] wrapInputFd(int)
# Own[AsyncOutputStream] wrapOutputFd(int)
Own[AsyncIoStream] wrapSocketFd(int)
Timer& getTimer() except +reraise_kj_exception
cdef cppclass AsyncIoProvider:
pass
TwoWayPipe newTwoWayPipe()
cdef cppclass WaitScope:
pass
cdef cppclass AsyncIoContext:
AsyncIoContext(AsyncIoContext&)
Own[LowLevelAsyncIoProvider] lowLevelProvider
@@ -140,6 +148,9 @@ cdef extern from "kj/async-io.h" namespace " ::kj":
cdef cppclass TaskSet:
TaskSet(ErrorHandler &)
cdef cppclass TwoWayPipe:
Own[AsyncIoStream] ends[2]
AsyncIoContext setupAsyncIo()
cdef extern from "capnp/schema.capnp.h" namespace " ::capnp":