From 475d7c4c2a0aa087dbafe57c35d8b46c20d337c7 Mon Sep 17 00:00:00 2001 From: Ran Benita Date: Mon, 1 Nov 2021 09:36:12 +0200 Subject: [PATCH] README,OVERVIEW: replace master -> controller --- OVERVIEW.md | 38 +++++++++++++++++++------------------- README.rst | 27 ++++++++++----------------- 2 files changed, 29 insertions(+), 36 deletions(-) diff --git a/OVERVIEW.md b/OVERVIEW.md index 87b9d1b..da0d3c4 100644 --- a/OVERVIEW.md +++ b/OVERVIEW.md @@ -1,36 +1,36 @@ # Overview # `xdist` works by spawning one or more **workers**, which are controlled -by the **master**. Each **worker** is responsible for performing -a full test collection and afterwards running tests as dictated by the **master**. +by the **controller**. Each **worker** is responsible for performing +a full test collection and afterwards running tests as dictated by the **controller**. The execution flow is: -1. **master** spawns one or more **workers** at the beginning of - the test session. The communication between **master** and **worker** nodes makes use of - [execnet](http://codespeak.net/execnet/) and its [gateways](http://codespeak.net/execnet/basics.html#gateways-bootstrapping-python-interpreters). +1. **controller** spawns one or more **workers** at the beginning of + the test session. The communication between **controller** and **worker** nodes makes use of + [execnet](https://codespeak.net/execnet/) and its [gateways](https://codespeak.net/execnet/basics.html#gateways-bootstrapping-python-interpreters). The actual interpreters executing the code for the **workers** might be remote or local. 1. Each **worker** itself is a mini pytest runner. **workers** at this point perform a full test collection, sending back the collected - test-ids back to the **master** which does not + test-ids back to the **controller** which does not perform any collection itself. -1. The **master** receives the result of the collection from all nodes. - At this point the **master** performs some sanity check to ensure that +1. The **controller** receives the result of the collection from all nodes. + At this point the **controller** performs some sanity check to ensure that all **workers** collected the same tests (including order), bailing out otherwise. If all is well, it converts the list of test-ids into a list of simple indexes, where each index corresponds to the position of that test in the original collection list. This works because all nodes have the same - collection list, and saves bandwidth because the **master** can now tell + collection list, and saves bandwidth because the **controller** can now tell one of the workers to just *execute test index 3* index of passing the full test id. -1. If **dist-mode** is **each**: the **master** just sends the full list +1. If **dist-mode** is **each**: the **controller** just sends the full list of test indexes to each node at this moment. -1. If **dist-mode** is **load**: the **master** takes around 25% of the +1. If **dist-mode** is **load**: the **controller** takes around 25% of the tests and sends them one by one to each **worker** in a round robin fashion. The rest of the tests will be distributed later as **workers** finish tests (see below). @@ -40,36 +40,36 @@ The execution flow is: 1. **workers** re-implement `pytest_runtestloop`: pytest's default implementation basically loops over all collected items in the `session` object and executes the `pytest_runtest_protocol` for each test item, but in xdist **workers** sit idly - waiting for **master** to send tests for execution. As tests are + waiting for **controller** to send tests for execution. As tests are received by **workers**, `pytest_runtest_protocol` is executed for each test. Here it worth noting an implementation detail: **workers** always must keep at least one test item on their queue due to how the `pytest_runtest_protocol(item, nextitem)` hook is defined: in order to pass the `nextitem` to the hook, the worker must wait for more - instructions from master before executing that remaining test. If it receives more tests, + instructions from controller before executing that remaining test. If it receives more tests, then it can safely call `pytest_runtest_protocol` because it knows what the `nextitem` parameter will be. If it receives a "shutdown" signal, then it can execute the hook passing `nextitem` as `None`. 1. As tests are started and completed at the **workers**, the results are sent - back to the **master**, which then just forwards the results to + back to the **controller**, which then just forwards the results to the appropriate pytest hooks: `pytest_runtest_logstart` and `pytest_runtest_logreport`. This way other plugins (for example `junitxml`) - can work normally. The **master** (when in dist-mode **load**) + can work normally. The **controller** (when in dist-mode **load**) decides to send more tests to a node when a test completes, using some heuristics such as test durations and how many tests each **worker** still has to run. -1. When the **master** has no more pending tests it will +1. When the **controller** has no more pending tests it will send a "shutdown" signal to all **workers**, which will then run their remaining tests to completion and shut down. At this point the - **master** will sit waiting for **workers** to shut down, still + **controller** will sit waiting for **workers** to shut down, still processing events such as `pytest_runtest_logreport`. ## FAQ ## > Why does each worker do its own collection, as opposed to having -the master collect once and distribute from that collection to the workers? +the controller collect once and distribute from that collection to the workers? -If collection was performed by master then it would have to +If collection was performed by controller then it would have to serialize collected items to send them through the wire, as workers live in another process. The problem is that test items are not easily (impossible?) to serialize, as they contain references to the test functions, fixture managers, config objects, etc. Even if one manages to serialize it, diff --git a/README.rst b/README.rst index 5394aad..53849fb 100644 --- a/README.rst +++ b/README.rst @@ -243,11 +243,11 @@ environment this command will send each tests to all platforms - and report back failures from all platforms at once. The specifications strings use the `xspec syntax`_. -.. _`xspec syntax`: http://codespeak.net/execnet/basics.html#xspec +.. _`xspec syntax`: https://codespeak.net/execnet/basics.html#xspec .. _`socketserver.py`: https://raw.githubusercontent.com/pytest-dev/execnet/master/execnet/script/socketserver.py -.. _`execnet`: http://codespeak.net/execnet +.. _`execnet`: https://codespeak.net/execnet Identifying the worker process during a test -------------------------------------------- @@ -287,17 +287,6 @@ Since version 2.0, the following functions are also available in the ``xdist`` m :param request_or_session: the `pytest` `request` or `session` object """ - def is_xdist_master(request_or_session) -> bool: - """Return `True` if this is the xdist controller, `False` otherwise - - Note: this method also returns `False` when distribution has not been - activated at all. - - deprecated alias for is_xdist_controller - - :param request_or_session: the `pytest` `request` or `session` object - """ - def is_xdist_controller(request_or_session) -> bool: """Return `True` if this is the xdist controller, `False` otherwise @@ -307,11 +296,15 @@ Since version 2.0, the following functions are also available in the ``xdist`` m :param request_or_session: the `pytest` `request` or `session` object """ + def is_xdist_master(request_or_session) -> bool: + """Deprecated alias for is_xdist_controller.""" + def get_xdist_worker_id(request_or_session) -> str: """Return the id of the current worker ('gw0', 'gw1', etc) or 'master' if running on the controller node. - If not distributing tests (for example passing `-n0` or not passing `-n` at all) also return 'master'. + If not distributing tests (for example passing `-n0` or not passing `-n` at all) + also return 'master'. :param request_or_session: the `pytest` `request` or `session` object """ @@ -371,10 +364,10 @@ Additionally, during a test run, the following environment variable is defined: * ``PYTEST_XDIST_TESTRUNUID``: the unique id of the test run. -Accessing ``sys.argv`` from the master node in workers ------------------------------------------------------- +Accessing ``sys.argv`` from the controller node in workers +---------------------------------------------------------- -To access the ``sys.argv`` passed to the command-line of the master node, use +To access the ``sys.argv`` passed to the command-line of the controller node, use ``request.config.workerinput["mainargv"]``.