README,OVERVIEW: replace master -> controller
This commit is contained in:
Ran Benita
38
OVERVIEW.md
38
OVERVIEW.md
@@ -1,36 +1,36 @@
|
||||
# Overview #
|
||||
|
||||
`xdist` works by spawning one or more **workers**, which are controlled
|
||||
by the **master**. Each **worker** is responsible for performing
|
||||
a full test collection and afterwards running tests as dictated by the **master**.
|
||||
by the **controller**. Each **worker** is responsible for performing
|
||||
a full test collection and afterwards running tests as dictated by the **controller**.
|
||||
|
||||
The execution flow is:
|
||||
|
||||
1. **master** spawns one or more **workers** at the beginning of
|
||||
the test session. The communication between **master** and **worker** nodes makes use of
|
||||
[execnet](http://codespeak.net/execnet/) and its [gateways](http://codespeak.net/execnet/basics.html#gateways-bootstrapping-python-interpreters).
|
||||
1. **controller** spawns one or more **workers** at the beginning of
|
||||
the test session. The communication between **controller** and **worker** nodes makes use of
|
||||
[execnet](https://codespeak.net/execnet/) and its [gateways](https://codespeak.net/execnet/basics.html#gateways-bootstrapping-python-interpreters).
|
||||
The actual interpreters executing the code for the **workers** might
|
||||
be remote or local.
|
||||
|
||||
1. Each **worker** itself is a mini pytest runner. **workers** at this
|
||||
point perform a full test collection, sending back the collected
|
||||
test-ids back to the **master** which does not
|
||||
test-ids back to the **controller** which does not
|
||||
perform any collection itself.
|
||||
|
||||
1. The **master** receives the result of the collection from all nodes.
|
||||
At this point the **master** performs some sanity check to ensure that
|
||||
1. The **controller** receives the result of the collection from all nodes.
|
||||
At this point the **controller** performs some sanity check to ensure that
|
||||
all **workers** collected the same tests (including order), bailing out otherwise.
|
||||
If all is well, it converts the list of test-ids into a list of simple
|
||||
indexes, where each index corresponds to the position of that test in the
|
||||
original collection list. This works because all nodes have the same
|
||||
collection list, and saves bandwidth because the **master** can now tell
|
||||
collection list, and saves bandwidth because the **controller** can now tell
|
||||
one of the workers to just *execute test index 3* index of passing the
|
||||
full test id.
|
||||
|
||||
1. If **dist-mode** is **each**: the **master** just sends the full list
|
||||
1. If **dist-mode** is **each**: the **controller** just sends the full list
|
||||
of test indexes to each node at this moment.
|
||||
|
||||
1. If **dist-mode** is **load**: the **master** takes around 25% of the
|
||||
1. If **dist-mode** is **load**: the **controller** takes around 25% of the
|
||||
tests and sends them one by one to each **worker** in a round robin
|
||||
fashion. The rest of the tests will be distributed later as **workers**
|
||||
finish tests (see below).
|
||||
@@ -40,36 +40,36 @@ The execution flow is:
|
||||
1. **workers** re-implement `pytest_runtestloop`: pytest's default implementation
|
||||
basically loops over all collected items in the `session` object and executes
|
||||
the `pytest_runtest_protocol` for each test item, but in xdist **workers** sit idly
|
||||
waiting for **master** to send tests for execution. As tests are
|
||||
waiting for **controller** to send tests for execution. As tests are
|
||||
received by **workers**, `pytest_runtest_protocol` is executed for each test.
|
||||
Here it worth noting an implementation detail: **workers** always must keep at
|
||||
least one test item on their queue due to how the `pytest_runtest_protocol(item, nextitem)`
|
||||
hook is defined: in order to pass the `nextitem` to the hook, the worker must wait for more
|
||||
instructions from master before executing that remaining test. If it receives more tests,
|
||||
instructions from controller before executing that remaining test. If it receives more tests,
|
||||
then it can safely call `pytest_runtest_protocol` because it knows what the `nextitem` parameter will be.
|
||||
If it receives a "shutdown" signal, then it can execute the hook passing `nextitem` as `None`.
|
||||
|
||||
1. As tests are started and completed at the **workers**, the results are sent
|
||||
back to the **master**, which then just forwards the results to
|
||||
back to the **controller**, which then just forwards the results to
|
||||
the appropriate pytest hooks: `pytest_runtest_logstart` and
|
||||
`pytest_runtest_logreport`. This way other plugins (for example `junitxml`)
|
||||
can work normally. The **master** (when in dist-mode **load**)
|
||||
can work normally. The **controller** (when in dist-mode **load**)
|
||||
decides to send more tests to a node when a test completes, using
|
||||
some heuristics such as test durations and how many tests each **worker**
|
||||
still has to run.
|
||||
|
||||
1. When the **master** has no more pending tests it will
|
||||
1. When the **controller** has no more pending tests it will
|
||||
send a "shutdown" signal to all **workers**, which will then run their
|
||||
remaining tests to completion and shut down. At this point the
|
||||
**master** will sit waiting for **workers** to shut down, still
|
||||
**controller** will sit waiting for **workers** to shut down, still
|
||||
processing events such as `pytest_runtest_logreport`.
|
||||
|
||||
## FAQ ##
|
||||
|
||||
> Why does each worker do its own collection, as opposed to having
|
||||
the master collect once and distribute from that collection to the workers?
|
||||
the controller collect once and distribute from that collection to the workers?
|
||||
|
||||
If collection was performed by master then it would have to
|
||||
If collection was performed by controller then it would have to
|
||||
serialize collected items to send them through the wire, as workers live in another process.
|
||||
The problem is that test items are not easily (impossible?) to serialize, as they contain references to
|
||||
the test functions, fixture managers, config objects, etc. Even if one manages to serialize it,
|
||||
|
||||
27
README.rst
27
README.rst
@@ -243,11 +243,11 @@ environment this command will send each tests to all
|
||||
platforms - and report back failures from all platforms
|
||||
at once. The specifications strings use the `xspec syntax`_.
|
||||
|
||||
.. _`xspec syntax`: http://codespeak.net/execnet/basics.html#xspec
|
||||
.. _`xspec syntax`: https://codespeak.net/execnet/basics.html#xspec
|
||||
|
||||
.. _`socketserver.py`: https://raw.githubusercontent.com/pytest-dev/execnet/master/execnet/script/socketserver.py
|
||||
|
||||
.. _`execnet`: http://codespeak.net/execnet
|
||||
.. _`execnet`: https://codespeak.net/execnet
|
||||
|
||||
Identifying the worker process during a test
|
||||
--------------------------------------------
|
||||
@@ -287,17 +287,6 @@ Since version 2.0, the following functions are also available in the ``xdist`` m
|
||||
:param request_or_session: the `pytest` `request` or `session` object
|
||||
"""
|
||||
|
||||
def is_xdist_master(request_or_session) -> bool:
|
||||
"""Return `True` if this is the xdist controller, `False` otherwise
|
||||
|
||||
Note: this method also returns `False` when distribution has not been
|
||||
activated at all.
|
||||
|
||||
deprecated alias for is_xdist_controller
|
||||
|
||||
:param request_or_session: the `pytest` `request` or `session` object
|
||||
"""
|
||||
|
||||
def is_xdist_controller(request_or_session) -> bool:
|
||||
"""Return `True` if this is the xdist controller, `False` otherwise
|
||||
|
||||
@@ -307,11 +296,15 @@ Since version 2.0, the following functions are also available in the ``xdist`` m
|
||||
:param request_or_session: the `pytest` `request` or `session` object
|
||||
"""
|
||||
|
||||
def is_xdist_master(request_or_session) -> bool:
|
||||
"""Deprecated alias for is_xdist_controller."""
|
||||
|
||||
def get_xdist_worker_id(request_or_session) -> str:
|
||||
"""Return the id of the current worker ('gw0', 'gw1', etc) or 'master'
|
||||
if running on the controller node.
|
||||
|
||||
If not distributing tests (for example passing `-n0` or not passing `-n` at all) also return 'master'.
|
||||
If not distributing tests (for example passing `-n0` or not passing `-n` at all)
|
||||
also return 'master'.
|
||||
|
||||
:param request_or_session: the `pytest` `request` or `session` object
|
||||
"""
|
||||
@@ -371,10 +364,10 @@ Additionally, during a test run, the following environment variable is defined:
|
||||
|
||||
* ``PYTEST_XDIST_TESTRUNUID``: the unique id of the test run.
|
||||
|
||||
Accessing ``sys.argv`` from the master node in workers
|
||||
------------------------------------------------------
|
||||
Accessing ``sys.argv`` from the controller node in workers
|
||||
----------------------------------------------------------
|
||||
|
||||
To access the ``sys.argv`` passed to the command-line of the master node, use
|
||||
To access the ``sys.argv`` passed to the command-line of the controller node, use
|
||||
``request.config.workerinput["mainargv"]``.
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user