Commit Graph

170 Commits

Author SHA1 Message Date
Aleksandr Mezin
d1dfad3e92 Implement work-stealing scheduler (#862)
Closes #858
2023-01-11 08:38:57 -03:00
Aleksandr Mezin
9b0b5b1495 Add --maxschedchunk CLI option (#857)
Maximum number of tests scheduled in one step.

Setting it to 1 will force pytest to send tests to workers one by one -
might be useful for a small number of slow tests.

Larger numbers will allow the scheduler to submit consecutive chunks of tests
to workers - allows reusing fixtures.

Unlimited if not set.

Fixes #855
Fixes #255
2022-12-23 08:21:20 -03:00
Bruno Oliveira
92a76bb013 Release 3.1.0
Some checks failed
deploy / deploy (push) Failing after 1s
2022-12-01 17:37:59 -03:00
Bruno Oliveira
7a0bc4c1c9 Let users configure dist mode in the configuration file
Fix #789
2022-11-30 13:57:48 -03:00
Bruno Oliveira
0316baf3ec Add CHANGELOG for py36 drop and py311 support 2022-11-05 12:24:00 -03:00
Bruno Oliveira
38dcf52ae6 Release 3.0.1
Some checks failed
build / build (ubuntu-latest, 3.10-dev, py310-pytestlatest) (push) Failing after 1s
build / build (ubuntu-latest, 3.6, py36-pytestlatest) (push) Failing after 1s
build / build (ubuntu-latest, 3.7, py37-pytestlatest) (push) Failing after 1s
build / build (ubuntu-latest, 3.8, py38-psutil) (push) Failing after 1s
build / build (ubuntu-latest, 3.8, py38-pytestlatest) (push) Failing after 1s
build / build (ubuntu-latest, 3.8, py38-pytestmain) (push) Failing after 1s
build / build (ubuntu-latest, 3.8, py38-setproctitle) (push) Failing after 1s
build / build (ubuntu-latest, 3.9, py39-pytestlatest) (push) Failing after 2s
build / build (windows-latest, 3.10-dev, py310-pytestlatest) (push) Has been cancelled
build / build (windows-latest, 3.6, py36-pytestlatest) (push) Has been cancelled
build / build (windows-latest, 3.7, py37-pytestlatest) (push) Has been cancelled
build / build (windows-latest, 3.8, py38-psutil) (push) Has been cancelled
build / build (windows-latest, 3.8, py38-pytestlatest) (push) Has been cancelled
build / build (windows-latest, 3.8, py38-pytestmain) (push) Has been cancelled
build / build (windows-latest, 3.8, py38-setproctitle) (push) Has been cancelled
build / build (windows-latest, 3.9, py39-pytestlatest) (push) Has been cancelled
build / deploy (push) Has been cancelled
2022-10-25 11:10:07 -03:00
Bruno Oliveira
f2633f1b37 Fetch all tags, as required by setuptools-scm 2022-10-25 11:09:53 -03:00
Bruno Oliveira
192193197b Update CHANGELOG for 3.0
Some checks failed
build / build (ubuntu-latest, 3.10-dev, py310-pytestlatest) (push) Failing after 3s
build / build (ubuntu-latest, 3.6, py36-pytestlatest) (push) Failing after 2s
build / build (ubuntu-latest, 3.7, py37-pytestlatest) (push) Failing after 3s
build / build (ubuntu-latest, 3.8, py38-psutil) (push) Failing after 2s
build / build (ubuntu-latest, 3.8, py38-pytestlatest) (push) Failing after 2s
build / build (ubuntu-latest, 3.8, py38-pytestmain) (push) Failing after 2s
build / build (ubuntu-latest, 3.8, py38-setproctitle) (push) Failing after 3s
build / build (ubuntu-latest, 3.9, py39-pytestlatest) (push) Failing after 3s
build / build (windows-latest, 3.10-dev, py310-pytestlatest) (push) Has been cancelled
build / build (windows-latest, 3.6, py36-pytestlatest) (push) Has been cancelled
build / build (windows-latest, 3.7, py37-pytestlatest) (push) Has been cancelled
build / build (windows-latest, 3.8, py38-psutil) (push) Has been cancelled
build / build (windows-latest, 3.8, py38-pytestlatest) (push) Has been cancelled
build / build (windows-latest, 3.8, py38-pytestmain) (push) Has been cancelled
build / build (windows-latest, 3.8, py38-setproctitle) (push) Has been cancelled
build / build (windows-latest, 3.9, py39-pytestlatest) (push) Has been cancelled
build / deploy (push) Has been cancelled
2022-10-24 13:44:11 -03:00
Bruno Oliveira
bd23c2460f Fix changelog fragments 2022-10-24 13:43:03 -03:00
Petr Viktorin
c168706fbe Get -nauto default from PYTEST_XDIST_AUTO_NUM_WORKERS (#829)
* Get `-nauto` default from `PYTEST_XDIST_AUTO_NUM_WORKERS`

A few additional tests for existing functionality are added too.
And the ``-n logical`` option is documented.

Fixes: https://github.com/pytest-dev/pytest-xdist/issues/792
2022-10-24 09:33:19 -03:00
Bruno Oliveira
8b0856a986 Deprecate looponfail plugin
Fix #826
2022-10-22 10:43:02 -03:00
Bruno Oliveira
1fbf49909f Remove --boxed and pytest-forked dependency
Fixes #468
Fixes #543
2022-10-22 07:31:34 -03:00
Anthony Sottile
1471164197 Replace usages of py.log by custom implementation (#822)
Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>
2022-10-22 07:26:01 -03:00
Aleksandr Mezin
80130d59ea Restore old initial batch distribution logic in LoadScheduling (#812)
* Restore old initial batch distribution logic in LoadScheduling

pytest orders tests for optimal sequential execution - i. e. avoiding
unnecessary setup and teardown of fixtures. So executing tests in consecutive
chunks is important for optimal performance.

Commit 09d79ace35 optimized test distribution for
the corner case, when the number of tests is less than 2 * number of nodes.
At the same time, it made initial test distribution worse for all other cases.
If some tests use some fixture, and these tests fit into the initial batch,
the fixture will be created min(n_tests, n_workers) times, no matter how many
other tests there are. With the old algorithm (before
09d79ace35), if there are enough tests not using
the fixture, the fixture was created only once.

So restore the old behavior for typical cases where the number of tests is
much greater than the number of workers (or, strictly speaking, when there
are at least 2 tests for every node).

In my test suite, where fixtures create Docker containers, this change reduces
total run time by 10-15%.

This is a partial revert of commit 09d79ace35

Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>
2022-10-22 07:12:16 -03:00
Bruno Oliveira
25cc1a4119 Replace py.path.local usages by pathlib.Path 2022-10-21 22:54:17 -03:00
Anthony Sottile
42024f374a remove py._pydir as rsync candidate 2022-10-21 16:09:14 -07:00
Aleksandr Mezin
e407a3377e Cancel shutdown when crashed worker is restarted (#813)
Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>
2022-09-06 17:37:38 -03:00
Pepa
61132777f8 Added known limitation section to the docs (#796)
Co-authored-by: Josef <sorry.i-keep@it.private>
Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>
2022-07-22 10:44:17 -03:00
Petr Viktorin
1a5e680e91 Document the pytest_xdist_auto_num_workers hook (#791) 2022-07-13 15:54:26 -03:00
Ran Benita
2aeb5d51f5 CHANGELOG: add --boxed deprecation to 2.5.0 2022-01-24 16:14:24 +02:00
Bruno Oliveira
c8bbc03e49 Release 2.5.0 2021-12-10 08:18:22 -03:00
baekdohyeop
62e50d0097 Address review 2021-11-29 02:59:18 +09:00
baekdohyeop
1b5d6b6db7 Add changelog 2021-11-29 02:59:18 +09:00
Ran Benita
89a3df3e20 Deprecate --boxed
This will allow removing the pytest-forked dependency in the future.

Refs #468.
2021-11-02 09:38:56 +02:00
Ran Benita
ae58690f3e Avoid soft-deprecated pathlist ini type on pytest>=7
This allows the test suite to succeed with the legacypath plugin
blocked.

Fix #722.
2021-11-01 09:06:32 +02:00
Ran Benita
787b8d9c14 Add basic mypy linting
Based on pytest setup, though more lax. Currently only the tests are
annotated so only helpful for them.

Fix #721.
2021-11-01 09:06:32 +02:00
Ran Benita
5672d85809 Require pytest>=6.2
This makes things much easier in terms of typing.

pytest 6.2 has been out for a ~year so if someone updates pytest-xdist
they should also update pytest!

Fix #720.
2021-11-01 09:04:16 +02:00
Ran Benita
130dcdef76 Add build-system to pyproject.toml
So doesn't use legacy stuff.

Fix #719.
2021-11-01 09:04:14 +02:00
Bruno Oliveira
c7c7f9ef5b Use modern options configurations for hooks
Noticed while reviewing https://github.com/pytest-dev/pytest/pull/9118
2021-09-24 10:09:24 -03:00
Bruno Oliveira
ecf4d3be31 Release 2.4.0 2021-09-20 21:17:31 -03:00
Hugo van Kemenade
e0ce1b7418 Add news file to add support for Python 3.10 2021-09-15 17:28:44 +03:00
David Lamparter
766e67ce52 Use setproctitle if available to show state (#696)
Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>
2021-09-03 12:58:01 -03:00
Ronny Pfannschmidt
26e7d953f4 prepare release 2.3.0 2021-06-16 11:46:18 +02:00
Ronny Pfannschmidt
b02a6db771 Merge pull request #667 from graingert/fix-sys-path
fix sys.path for local workers  Fixes #421
2021-06-16 11:40:55 +02:00
Thomas Grainger
b0722675e0 add newsfile 2021-06-16 10:34:19 +01:00
Ronny Pfannschmidt
7f07d50808 Merge pull request #646 from kroeschl/numprocesses-logical
Add explicit 'logical' choice to --numprocesses flag
2021-06-08 08:44:29 +02:00
Kyle Roeschley
4d27110beb Add 'logical' CPU count choice for --numprocesses
This can significantly speed up runs when testing is not CPU-bound.
2021-04-27 08:47:21 -05:00
xoviat
f861f1667d add new pytest_handlecrashitem hook
allows handling and rescheduling crash tests

PR #651

Co-authored-by: xoviat <xoviat@users.noreply.github.com>
Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>
2021-04-27 09:07:49 -03:00
Bruno Oliveira
8b35945d54 Drop Python 3.5 support
pytest no longer supports it, and our CI is broken because of it.
2021-04-25 09:31:10 -03:00
Thomas Grainger
1dc019709c Use 'main' to refer to pytest default branch in tox env names. (#643)
Co-authored-by: Thomas Grainger <tagrain@gmail.com>
Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>
2021-03-20 11:43:57 -03:00
baekdohyeop
f5342962ab [HOTFIX] rename pytest's branch(master to main) 2021-03-15 13:14:08 +09:00
Ronny Pfannschmidt
166bdb4103 prepare #592: replace master with controller where we can 2021-02-07 22:08:22 +01:00
Bruno Oliveira
8f90fb73db Release 2.2.0
Some checks failed
build / build (ubuntu-latest, 3.5, py35-pytestlatest) (push) Failing after 1s
build / build (ubuntu-latest, 3.6, py36-pytestlatest) (push) Failing after 2s
build / build (ubuntu-latest, 3.7, linting) (push) Failing after 1s
build / build (ubuntu-latest, 3.7, py37-pytestlatest) (push) Failing after 1s
build / build (ubuntu-latest, 3.8, py38-psutil) (push) Failing after 1s
build / build (ubuntu-latest, 3.8, py38-pytestlatest) (push) Failing after 1s
build / build (ubuntu-latest, 3.8, py38-pytestmaster) (push) Failing after 1s
build / build (ubuntu-latest, 3.9, py39-pytestlatest) (push) Failing after 1s
build / linting (push) Failing after 1s
build / build (windows-latest, 3.5, py35-pytestlatest) (push) Has been cancelled
build / build (windows-latest, 3.6, py36-pytestlatest) (push) Has been cancelled
build / build (windows-latest, 3.7, linting) (push) Has been cancelled
build / build (windows-latest, 3.7, py37-pytestlatest) (push) Has been cancelled
build / build (windows-latest, 3.8, py38-psutil) (push) Has been cancelled
build / build (windows-latest, 3.8, py38-pytestlatest) (push) Has been cancelled
build / build (windows-latest, 3.8, py38-pytestmaster) (push) Has been cancelled
build / build (windows-latest, 3.9, py39-pytestlatest) (push) Has been cancelled
build / deploy (push) Has been cancelled
2020-12-14 09:07:49 -03:00
Bruno Oliveira
16d636391b Propagate internal errors to the master node
This should help users diagnose internal errors in workers like
exceptions from hooks or in pytest itself.
2020-12-13 20:05:31 -03:00
Bruno Oliveira
5b7eeab7dc Release 2.1.0 2020-08-25 09:01:37 -03:00
Bruno Oliveira
607d8288d4 Add psutil extra and introduce pytest_xdist_auto_num_workers hook
This makes using psutil optional and opens up the possibility of customization
through the pytest_xdist_auto_num_workers hook, making things like #477 possible.

Fix #585
2020-08-25 08:38:28 -03:00
Bruno Oliveira
43dad454c6 Prepare release 2.0.0 2020-08-12 17:28:27 -03:00
Christine M
c211d247a7 Noisy rsync (#473)
Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>
2020-08-12 17:23:03 -03:00
Sylvain Marié
ba8398478e New methods is_xdist_worker, is_xdist_master, get_xdist_worker_id. (#505)
Co-authored-by: Sylvain MARIE <sylvain.marie@se.com>
Co-authored-by: Zac Hatfield-Dodds <Zac-HD@users.noreply.github.com>
Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>
2020-08-05 20:41:54 -03:00
Bruno Oliveira
2f014c39df Minor changelog adjustment 2020-08-05 19:02:50 -03:00