c168706fbe
* Get `-nauto` default from `PYTEST_XDIST_AUTO_NUM_WORKERS` A few additional tests for existing functionality are added too. And the ``-n logical`` option is documented. Fixes: https://github.com/pytest-dev/pytest-xdist/issues/792
86 lines
3.2 KiB
ReStructuredText
86 lines
3.2 KiB
ReStructuredText
.. _parallelization:
|
|
|
|
Running tests across multiple CPUs
|
|
==================================
|
|
|
|
To send tests to multiple CPUs, use the ``-n`` (or ``--numprocesses``) option::
|
|
|
|
pytest -n auto
|
|
|
|
This can lead to considerable speed ups, especially if your test suite takes a
|
|
noticeable amount of time.
|
|
|
|
With ``-n auto``, pytest-xdist will use as many processes as your computer
|
|
has CPU cores.
|
|
|
|
Use ``-n logical`` to use the number of *logical* CPU cores rather than
|
|
physical ones. This currently requires the ``psutils`` package to be installed;
|
|
if it is not, pytest-xdist will fall back to ``-n auto`` behavior.
|
|
|
|
Pass a number, e.g. ``-n 8``, to specify the number of processes explicitly.
|
|
|
|
To specify a different meaning for ``-n auto`` and ``-n logical`` for your
|
|
tests, you can:
|
|
|
|
* Set the environment variable ``PYTEST_XDIST_AUTO_NUM_WORKERS`` to the
|
|
desired number of processes.
|
|
|
|
* Implement the ``pytest_xdist_auto_num_workers``
|
|
`pytest hook <https://docs.pytest.org/en/latest/how-to/writing_plugins.html>`__
|
|
(a ``pytest_xdist_auto_num_workers(config)`` function in e.g. ``conftest.py``)
|
|
that returns the number of processes to use.
|
|
The hook can use ``config.option.numprocesses`` to determine if the user
|
|
asked for ``"auto"`` or ``"logical"``, and it can return ``None`` to fall
|
|
back to the default.
|
|
|
|
If both the hook and environment variable are specified, the hook takes
|
|
priority.
|
|
|
|
|
|
Parallelization can be configured further with these options:
|
|
|
|
* ``--maxprocesses=maxprocesses``: limit the maximum number of workers to
|
|
process the tests.
|
|
|
|
* ``--max-worker-restart``: maximum number of workers that can be restarted
|
|
when crashed (set to zero to disable this feature).
|
|
|
|
The test distribution algorithm is configured with the ``--dist`` command-line option:
|
|
|
|
.. _distribution modes:
|
|
|
|
* ``--dist load`` **(default)**: Sends pending tests to any worker that is
|
|
available, without any guaranteed order.
|
|
|
|
* ``--dist loadscope``: Tests are grouped by **module** for *test functions*
|
|
and by **class** for *test methods*. Groups are distributed to available
|
|
workers as whole units. This guarantees that all tests in a group run in the
|
|
same process. This can be useful if you have expensive module-level or
|
|
class-level fixtures. Grouping by class takes priority over grouping by
|
|
module.
|
|
|
|
* ``--dist loadfile``: Tests are grouped by their containing file. Groups are
|
|
distributed to available workers as whole units. This guarantees that all
|
|
tests in a file run in the same worker.
|
|
|
|
* ``--dist loadgroup``: Tests are grouped by the ``xdist_group`` mark. Groups are
|
|
distributed to available workers as whole units. This guarantees that all
|
|
tests with same ``xdist_group`` name run in the same worker.
|
|
|
|
.. code-block:: python
|
|
|
|
@pytest.mark.xdist_group(name="group1")
|
|
def test1():
|
|
pass
|
|
|
|
|
|
class TestA:
|
|
@pytest.mark.xdist_group("group1")
|
|
def test2():
|
|
pass
|
|
|
|
This will make sure ``test1`` and ``TestA::test2`` will run in the same worker.
|
|
Tests without the ``xdist_group`` mark are distributed normally as in the ``--dist=load`` mode.
|
|
|
|
* ``--dist no``: The normal pytest execution mode, runs one test at a time (no distribution at all).
|