IQ.Lvbs/pytest-xdist
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Drop "node" from "workers" and "master"

Browse Source
This commit is contained in:
Bruno Oliveira
2015-12-10 20:00:43 -02:00
parent c62effdd79
commit 1716767a1c
1 changed files with 21 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
OVERVIEW.md
View File

@@ -1,25 +1,25 @@
# Overview # # Overview #
`xdist` works by spawning one or more **worker nodes**, which are controlled `xdist` works by spawning one or more **workers**, which are controlled
by the **master node**. Each **worker node** is responsible for performing by the **master**. Each **worker** is responsible for performing
a full test collection and afterwards running tests as dictated by the **master node**. a full test collection and afterwards running tests as dictated by the **master**.
The execution flow is: The execution flow is:
1. **master node** spawns one or more **worker nodes** at the beginning of 1. **master** spawns one or more **workers** at the beginning of
the test session. The communication between **master** and **worker** nodes makes use of the test session. The communication between **master** and **worker** nodes makes use of
[execnet](http://codespeak.net/execnet/) and its [gateways](http://codespeak.net/execnet/basics.html#gateways-bootstrapping-python-interpreters). [execnet](http://codespeak.net/execnet/) and its [gateways](http://codespeak.net/execnet/basics.html#gateways-bootstrapping-python-interpreters).
The actual interpreters executing the code for the **worker nodes** might The actual interpreters executing the code for the **workers** might
be remote or local. be remote or local.
1. Each **worker node** itself is a mini pytest runner. **workers** at this 1. Each **worker** itself is a mini pytest runner. **workers** at this
point perform a full test collection, sending back the collected point perform a full test collection, sending back the collected
test-ids back to the **master node** which does not test-ids back to the **master** which does not
perform any collection itself. perform any collection itself.
1. The **master node** receives the result of the collection from all nodes. 1. The **master** receives the result of the collection from all nodes.
At this point the **master node** performs some sanity check to ensure that At this point the **master** performs some sanity check to ensure that
all **worker nodes** collected the same tests (including order), bailing out otherwise. all **workers** collected the same tests (including order), bailing out otherwise.
If all is well, it converts the list of test-ids into a list of simple If all is well, it converts the list of test-ids into a list of simple
indexes, where each index corresponds to the position of that test in the indexes, where each index corresponds to the position of that test in the
original collection list. This works because all nodes have the same original collection list. This works because all nodes have the same
@@ -27,18 +27,18 @@ The execution flow is:
one of the workers to just *execute test index 3* index of passing the one of the workers to just *execute test index 3* index of passing the
full test id. full test id.
1. If **dist-mode** is **each**: the **master node** just sends the full list 1. If **dist-mode** is **each**: the **master** just sends the full list
of test indexes to each node at this moment. of test indexes to each node at this moment.
1. If **dist-mode** is **load**: the **master node** takes around 25% of the 1. If **dist-mode** is **load**: the **master** takes around 25% of the
tests and sends them one by one to each **worker node** in a round robin tests and sends them one by one to each **worker** in a round robin
fashion. The rest of the tests will be distributed later as **worker nodes** fashion. The rest of the tests will be distributed later as **workers**
finish tests (see below). finish tests (see below).
1. **worker nodes** re-implement `pytest_runtestloop`: pytest's default implementation 1. **workers** re-implement `pytest_runtestloop`: pytest's default implementation
basically loops over all collected items in the `session` object and executes basically loops over all collected items in the `session` object and executes
the `pytest_runtest_protocol` for each test item, but in xdist **workers** sit idly the `pytest_runtest_protocol` for each test item, but in xdist **workers** sit idly
waiting for **master node** to send tests for execution. As tests are waiting for **master** to send tests for execution. As tests are
received by **workers**, `pytest_runtest_protocol` is executed for each test. received by **workers**, `pytest_runtest_protocol` is executed for each test.
Here it worth noting an implementation detail: **workers** always must keep at Here it worth noting an implementation detail: **workers** always must keep at
least one test item on their queue due to how the `pytest_runtest_protocol(item, nextitem)` least one test item on their queue due to how the `pytest_runtest_protocol(item, nextitem)`
@@ -48,17 +48,17 @@ The execution flow is:
If it receives a "shutdown" signal, then it can execute the hook passing `nextitem` as `None`. If it receives a "shutdown" signal, then it can execute the hook passing `nextitem` as `None`.
1. As tests are started and completed at the **workers**, the results are sent 1. As tests are started and completed at the **workers**, the results are sent
back to the **master node**, which then just forwards the results to back to the **master**, which then just forwards the results to
the appropriate pytest hooks: `pytest_runtest_logstart` and the appropriate pytest hooks: `pytest_runtest_logstart` and
`pytest_runtest_logreport`. This way other plugins (for example `junitxml`) `pytest_runtest_logreport`. This way other plugins (for example `junitxml`)
can work normally. The **master node** (when in dist-mode **load**) can work normally. The **master** (when in dist-mode **load**)
decides to send more tests to a node when a test completes, using decides to send more tests to a node when a test completes, using
some heuristics such as test durations and how many tests each **worker node** some heuristics such as test durations and how many tests each **worker**
still has to run. still has to run.
1. When the **master node** has no more pending tests it will 1. When the **master** has no more pending tests it will
send a "shutdown" signal to all **workers**, which will then run their send a "shutdown" signal to all **workers**, which will then run their
remaining tests to completion and shut down. At this point the remaining tests to completion and shut down. At this point the
**master node** will sit waiting for **workers** to shut down, still **master** will sit waiting for **workers** to shut down, still
processing events such as `pytest_runtest_logreport`. processing events such as `pytest_runtest_logreport`.