Release 1.31.0

* No idea what I'm doing.

* Extend integration test to cover ac3f6a78c7401aba9b62c2c67f280a129ae84eaf.

* Remove useless output to stop failing other tests.

* Reformat.

* Upgrade virtualenv to see if it fixes py34 builds

* Add CHANGELOG entry

* Make foobarplugin part of the test only

* Fix language_version in pre-commit for rst

* Fix test in Python 2

Co-authored-by: Bruno Oliveira <nicoddemus@gmail.com>

.appveyor.yml

@@ -5,17 +5,18 @@ environment:
   - TOXENV: "py35-pytestlatest"
   - TOXENV: "py36-pytestlatest"
   - TOXENV: "py37-pytestlatest"
-  - TOXENV: "py37-pytestmaster"
+  - TOXENV: "py38-pytestlatest"
-  - TOXENV: "py37-pytestfeatures"
+  - TOXENV: "py38-pytestmaster"
+  - TOXENV: "py38-pytestfeatures"
 
 install:
-  - C:\Python37\python -m pip install -U pip setuptools
+  - C:\Python38\python -m pip install -U pip setuptools virtualenv
-  - C:\Python37\python -m pip install -U tox setuptools_scm
+  - C:\Python38\python -m pip install -U tox setuptools_scm
 
 build: false  # Not a C# project, build stuff at the test step instead.
 
 test_script:
-  - C:\Python37\python -m tox
+  - C:\Python38\python -m tox
 
 # We don't deploy anything on tags with AppVeyor, we use Travis instead, so we
 # might as well save resources

.pre-commit-config.yaml

@@ -21,4 +21,4 @@ repos:
         files: ^(CHANGELOG.rst|HOWTORELEASE.rst|README.rst|changelog/.*)$
         language: python
         additional_dependencies: [pygments, restructuredtext_lint]
-        python_version: python3.7
+        language_version: python3.7

.travis.yml

@@ -30,8 +30,8 @@ jobs:
       cache:
         directories:
           - $HOME/.cache/pre-commit
-    - python: '3.7'
+    - python: '3.8'
-      env: TOXENV=py37-pytestlatest
+      env: TOXENV=py38-pytestlatest
     - python: '2.7'
       env: TOXENV=py27-pytestlatest
 
@@ -43,12 +43,14 @@ jobs:
     - python: "3.6"
       env: TOXENV=py36-pytestlatest
     - python: "3.7"
-      env: TOXENV=py37-pytestmaster
+      env: TOXENV=py37-pytestlatest
-    - python: "3.7"
+    - python: "3.8"
-      env: TOXENV=py37-pytestfeatures
+      env: TOXENV=py38-pytestmaster
+    - python: "3.8"
+      env: TOXENV=py38-pytestfeatures
 
     - stage: deploy
-      python: '3.7'
+      python: '3.8'
       env:
       install: pip install -U setuptools setuptools_scm
       script: skip

CHANGELOG.rst

@@ -1,3 +1,42 @@
+pytest-xdist 1.31.0 (2019-12-19)
+================================
+
+Features
+--------
+
+- `#486 <https://github.com/pytest-dev/pytest-xdist/issues/486>`_: Add support for Python 3.8.
+
+
+Bug Fixes
+---------
+
+- `#491 <https://github.com/pytest-dev/pytest-xdist/issues/491>`_: Fix regression that caused custom plugin command-line arguments to be discarded when using ``--tx`` mode.
+
+
+
+pytest-xdist 1.30.0 (2019-10-01)
+================================
+
+Features
+--------
+
+- `#448 <https://github.com/pytest-dev/pytest-xdist/issues/448>`_: Initialization between workers and master nodes is now more consistent, which fixes a number of
+  long-standing issues related to startup with the ``-c`` option.
+
+  Issues:
+
+  * `#6 <https://github.com/pytest-dev/pytest-xdist/issues/6>`__: Poor interaction between ``-n#`` and ``-c X.cfg``
+  * `#445 <https://github.com/pytest-dev/pytest-xdist/issues/445>`__: pytest-xdist is not reporting the same nodeid as pytest does
+
+  This however only works with **pytest 5.1 or later**, as it required changes in pytest itself.
+
+
+Bug Fixes
+---------
+
+- `#467 <https://github.com/pytest-dev/pytest-xdist/issues/467>`_: Fix crash issues related to running xdist with the terminal plugin disabled.
+
+
 pytest-xdist 1.29.0 (2019-06-14)
 ================================
 
@@ -401,17 +440,6 @@ Improved Documentation
 - Added ``HOWTORELEASE`` documentation. (#155)
 
 
-..
-    You should *NOT* be adding new change log entries to this file, this
-    file is managed by towncrier. You *may* edit previous change logs to
-    fix problems like typo corrections or such.
-    To add a new change log entry, please see
-    https://pip.pypa.io/en/latest/development/#adding-a-news-entry
-    We named the news folder ``changelog``
-
-.. towncrier release notes start
-
-
 1.17.0
 ------
 

README.rst

@@ -95,6 +95,59 @@ any guaranteed order, but you can control this with these options:
   in version ``1.21``.
 
 
+Making session-scoped fixtures execute only once
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+``pytest-xdist`` is designed so that each worker process will perform its own collection and execute
+a subset of all tests. This means that tests in different processes requesting a high-level
+scoped fixture (for example ``session``) will execute the fixture code more than once, which
+breaks expectations and might be undesired in certain situations.
+
+While ``pytest-xdist`` does not have a builtin support for ensuring a session-scoped fixture is
+executed exactly once, this can be achieved by using a lock file for inter-process communication.
+
+The example below needs to execute the fixture ``session_data`` only once (because it is
+resource intensive, or needs to execute only once to define configuration options, etc), so it makes
+use of a `FileLock <https://pypi.org/project/filelock/>`_ to produce the fixture data only once
+when the first process requests the fixture, while the other processes will then read
+the data from a file.
+
+Here is the code:
+
+.. code-block:: python
+
+    import json
+
+    import pytest
+    from filelock import FileLock
+
+
+    @pytest.fixture(scope="session")
+    def session_data(tmp_path_factory, worker_id):
+        if not worker_id:
+            # not executing in with multiple workers, just produce the data and let
+            # pytest's fixture caching do its job
+            return produce_expensive_data()
+
+        # get the temp directory shared by all workers
+        root_tmp_dir = tmp_path_factory.getbasetemp().parent
+
+        fn = root_tmp_dir / "data.json"
+        with FileLock(str(fn) + ".lock"):
+            if fn.is_file():
+                data = json.loads(fn.read_text())
+            else:
+                data = produce_expensive_data()
+                fn.write_text(json.dumps(data))
+        return data
+
+
+The example above can also be use in cases a fixture needs to execute exactly once per test session, like
+initializing a database service and populating initial tables.
+
+This technique might not work for every case, but should be a starting point for many situations
+where executing a high-scope fixture exactly once is important.
+
 Running tests in a Python subprocess
 ------------------------------------
 

setup.py

@@ -46,5 +46,6 @@ setup(
         "Programming Language :: Python :: 3.5",
         "Programming Language :: Python :: 3.6",
         "Programming Language :: Python :: 3.7",
+        "Programming Language :: Python :: 3.8",
     ],
 )

src/xdist/dsession.py

@@ -49,11 +49,8 @@ class DSession(object):
         self._max_worker_restart = get_default_max_worker_restart(self.config)
         # summary message to print at the end of the session
         self._summary_report = None
-        try:
         self.terminal = config.pluginmanager.getplugin("terminalreporter")
-        except KeyError:
+        if self.terminal:
-            self.terminal = None
-        else:
             self.trdist = TerminalDistReporter(config)
             config.pluginmanager.register(self.trdist, "terminaldistreporter")
 

src/xdist/plugin.py

@@ -154,11 +154,7 @@ def pytest_addoption(parser):
 def pytest_addhooks(pluginmanager):
     from xdist import newhooks
 
-    # avoid warnings with pytest-2.8
+    pluginmanager.add_hookspecs(newhooks)
-    method = getattr(pluginmanager, "add_hookspecs", None)
-    if method is None:
-        method = pluginmanager.addhooks
-    method(newhooks)
 
 
 # -------------------------------------------------------------------------
@@ -174,6 +170,7 @@ def pytest_configure(config):
         session = DSession(config)
         config.pluginmanager.register(session, "dsession")
         tr = config.pluginmanager.getplugin("terminalreporter")
+        if tr:
             tr.showfspath = False
     if config.getoption("boxed"):
         config.option.forked = True

src/xdist/remote.py

@@ -15,6 +15,8 @@ import _pytest.hookspec
 import pytest
 from execnet.gateway_base import dumps, DumpError
 
+from _pytest.config import _prepareconfig, Config
+
 
 class WorkerInteractor(object):
     def __init__(self, config, channel):
@@ -211,18 +213,18 @@ def getinfodict():
 
 
 def remote_initconfig(option_dict, args):
-    from _pytest.config import Config
-
     option_dict["plugins"].append("no:terminal")
-    config = Config.fromdictargs(option_dict, args)
+    return Config.fromdictargs(option_dict, args)
+
+
+def setup_config(config, basetemp):
     config.option.looponfail = False
     config.option.usepdb = False
     config.option.dist = "no"
     config.option.distload = False
     config.option.numprocesses = None
     config.option.maxprocesses = None
-    config.args = args
+    config.option.basetemp = basetemp
-    return config
 
 
 if __name__ == "__channelexec__":
@@ -239,7 +241,13 @@ if __name__ == "__channelexec__":
     os.environ["PYTEST_XDIST_WORKER"] = workerinput["workerid"]
     os.environ["PYTEST_XDIST_WORKER_COUNT"] = str(workerinput["workercount"])
 
+    if hasattr(Config, "InvocationParams"):
+        config = _prepareconfig(args, None)
+    else:
         config = remote_initconfig(option_dict, args)
+        config.args = args
+
+    setup_config(config, option_dict.get("basetemp"))
     config._parser.prog = os.path.basename(workerinput["mainargv"][0])
     config.workerinput = workerinput
     config.workeroutput = {}

src/xdist/workermanage.py

@@ -112,7 +112,10 @@ class NodeManager(object):
         ignores += self.config.option.rsyncignore
         ignores += self.config.getini("rsyncignore")
 
-        return {"ignores": ignores, "verbose": self.config.option.verbose}
+        return {
+            "ignores": ignores,
+            "verbose": getattr(self.config.option, "verbose", False),
+        }
 
     def rsync(self, gateway, source, notify=None, verbose=False, ignores=None):
         """Perform rsync to remote hosts for node."""
@@ -186,6 +189,9 @@ def make_reltoroot(roots, args):
     for arg in args:
         parts = arg.split(splitcode)
         fspath = py.path.local(parts[0])
+        if not fspath.exists():
+            result.append(arg)
+            continue
         for root in roots:
             x = fspath.relto(root)
             if x or fspath == root:
@@ -236,10 +242,14 @@ class WorkerController(object):
     def setup(self):
         self.log("setting up worker session")
         spec = self.gateway.spec
+        if hasattr(self.config, "invocation_params"):
+            args = [str(x) for x in self.config.invocation_params.args or ()]
+            option_dict = {}
+        else:
             args = self.config.args
+            option_dict = vars(self.config.option)
         if not spec.popen or spec.chdir:
             args = make_reltoroot(self.nodemanager.roots, args)
-        option_dict = vars(self.config.option)
         if spec.popen:
             name = "popen-%s" % self.gateway.id
             if hasattr(self.config, "_tmpdirhandler"):

testing/acceptance_test.py

@@ -186,16 +186,46 @@ class TestDistribution:
         )
         assert result.ret == 1
 
-    def test_distribution_rsyncdirs_example(self, testdir):
+    def test_distribution_rsyncdirs_example(self, testdir, monkeypatch):
+        # use a custom plugin that has a custom command-line option to ensure
+        # this is propagated to workers (see #491)
+        testdir.makepyfile(
+            **{
+                "myplugin/src/foobarplugin.py": """
+            from __future__ import print_function
+
+            import os
+            import sys
+            import pytest
+
+            def pytest_addoption(parser):
+                parser.addoption("--foobar", action="store", dest="foobar_opt")
+
+            @pytest.mark.tryfirst
+            def pytest_load_initial_conftests(early_config):
+                opt = early_config.known_args_namespace.foobar_opt
+                print("--foobar=%s active! [%s]" % (opt, os.getpid()), file=sys.stderr)
+            """
+            }
+        )
+        assert (testdir.tmpdir / "myplugin/src/foobarplugin.py").check(file=1)
+        monkeypatch.setenv(
+            "PYTHONPATH", str(testdir.tmpdir / "myplugin/src"), prepend=os.pathsep
+        )
+
         source = testdir.mkdir("source")
         dest = testdir.mkdir("dest")
         subdir = source.mkdir("example_pkg")
         subdir.ensure("__init__.py")
         p = subdir.join("test_one.py")
         p.write("def test_5():\n  assert not __file__.startswith(%r)" % str(p))
-        result = testdir.runpytest(
+        result = testdir.runpytest_subprocess(
             "-v",
             "-d",
+            "-s",
+            "-pfoobarplugin",
+            "--foobar=123",
+            "--dist=load",
             "--rsyncdir=%(subdir)s" % locals(),
             "--tx=popen//chdir=%(dest)s" % locals(),
             p,
@@ -209,6 +239,7 @@ class TestDistribution:
                 "*1 passed*",
             ]
         )
+        result.stderr.fnmatch_lines(["--foobar=123 active! *"])
         assert dest.join(subdir.basename).check(dir=1)
 
     def test_backward_compatibility_worker_terminology(self, testdir):
@@ -559,6 +590,30 @@ def test_fixture_teardown_failure(testdir):
     assert result.ret
 
 
+def test_config_initialization(testdir, pytestconfig):
+    """Ensure workers and master are initialized consistently. Integration test for #445"""
+    if not hasattr(pytestconfig, "invocation_params"):
+        pytest.skip(
+            "requires pytest >=5.1 (config has no attribute 'invocation_params')"
+        )
+    testdir.makepyfile(
+        **{
+            "dir_a/test_foo.py": """
+                def test_1(): pass
+        """
+        }
+    )
+    testdir.makefile(
+        ".ini",
+        myconfig="""
+        [pytest]
+        testpaths=dir_a
+    """,
+    )
+    result = testdir.runpytest("-n2", "-c", "myconfig.ini", "-v")
+    result.stdout.fnmatch_lines(["dir_a/test_foo.py::test_1*"])
+
+
 @pytest.mark.parametrize("when", ["setup", "call", "teardown"])
 def test_crashing_item(testdir, when):
     """Ensure crashing item is correctly reported during all testing stages"""
@@ -760,7 +815,7 @@ class TestWarnings:
             )
         )
         result = testdir.runpytest(n)
-        result.stdout.fnmatch_lines(["*this is a warning*", "*1 passed, 1 warnings*"])
+        result.stdout.fnmatch_lines(["*this is a warning*", "*1 passed, 1 warning*"])
 
     @pytest.mark.parametrize("n", ["-n0", "-n1"])
     def test_custom_subclass(self, testdir, n):
@@ -784,7 +839,7 @@ class TestWarnings:
         )
         testdir.syspathinsert()
         result = testdir.runpytest(n)
-        result.stdout.fnmatch_lines(["*MyWarning*", "*1 passed, 1 warnings*"])
+        result.stdout.fnmatch_lines(["*MyWarning*", "*1 passed, 1 warning*"])
 
     @pytest.mark.parametrize("n", ["-n0", "-n1"])
     def test_unserializable_arguments(self, testdir, n):
@@ -801,7 +856,7 @@ class TestWarnings:
         )
         testdir.syspathinsert()
         result = testdir.runpytest(n)
-        result.stdout.fnmatch_lines(["*UserWarning*foo.txt*", "*1 passed, 1 warnings*"])
+        result.stdout.fnmatch_lines(["*UserWarning*foo.txt*", "*1 passed, 1 warning*"])
 
     @pytest.mark.parametrize("n", ["-n0", "-n1"])
     def test_unserializable_warning_details(self, testdir, n):
@@ -833,7 +888,7 @@ class TestWarnings:
         testdir.syspathinsert()
         result = testdir.runpytest(n)
         result.stdout.fnmatch_lines(
-            ["*ResourceWarning*unclosed*", "*1 passed, 1 warnings*"]
+            ["*ResourceWarning*unclosed*", "*1 passed, 1 warning*"]
         )
 
 
@@ -1050,6 +1105,22 @@ def test_color_yes_collection_on_non_atty(testdir, request):
     assert "gw0 C / gw1 C" not in result.stdout.str()
 
 
+def test_without_terminal_plugin(testdir, request):
+    """
+    No output when terminal plugin is disabled
+    """
+    testdir.makepyfile(
+        """
+        def test_1():
+            pass
+    """
+    )
+    result = testdir.runpytest("-p", "no:terminal", "-n2")
+    assert result.stdout.str() == ""
+    assert result.stderr.str() == ""
+    assert result.ret == 0
+
+
 def test_internal_error_with_maxfail(testdir):
     """
     Internal error when using --maxfail option (#62, #65).

testing/conftest.py

@@ -1,7 +1,11 @@
+import six
 import py
 import pytest
 import execnet
 
+pytest_plugins = "pytester"
+
+if six.PY2:
 
     @pytest.fixture(scope="session", autouse=True)
     def _ensure_imports():
@@ -12,11 +16,6 @@ def _ensure_imports():
         execnet.makegateway
 
 
-pytest_plugins = "pytester"
-
-# rsyncdirs = ['.', '../xdist', py.path.local(execnet.__file__).dirpath()]
-
-
 @pytest.fixture(autouse=True)
 def _divert_atexit(request, monkeypatch):
     import atexit
@@ -45,14 +44,6 @@ def specssh(request):
     return getspecssh(request.config)
 
 
-@pytest.fixture
-def testdir(testdir):
-    # pytest before 2.8 did not have a runpytest_subprocess
-    if not hasattr(testdir, "runpytest_subprocess"):
-        testdir.runpytest_subprocess = testdir.runpytest
-    return testdir
-
-
 # configuration information for tests
 def getgspecs(config):
     return [execnet.XSpec(spec) for spec in config.getvalueorskip("gspecs")]

tox.ini

@@ -1,8 +1,8 @@
 [tox]
 envlist=
   linting
-  py{27,34,35,36,37}-pytestlatest
+  py{27,34,35,36,37,38}-pytestlatest
-  py37-pytest{master,features}
+  py38-pytest{master,features}
 
 [testenv]
 passenv = USER USERNAME