https://pantsbuild.org/ logo
#general
Title
# general
b

brave-furniture-86963

09/27/2021, 7:11 PM
Hi all! What determines how many
requirements.pex
and
pytest_runner.pex
get built during
./pants test ::
? I have defined 9
python_tests
targets in all in our repository, yet I’m seeing over 26
pytest_runner.pex
and over 31
requirements.pex
built (also these numbers change depending on how I structure my requirements.txt files).
h

happy-kitchen-89482

09/27/2021, 7:57 PM
Hi @brave-furniture-86963! Pants runs each test file in its own process, with its own requirements. So if a
python_tests
target has multiple sources, each one will run separately.
This allows each test file to run with just its actual requirements, and not be invalidated when unrelated requirements change.
But (assuming there are no conflicting versions involved) Pants does one pip resolve, and then composes the the per-file requirements as a subset of that resolve
so you shouldn't be seeing 31 slow pip runs
you should see 1 slow pip run then 31 fast subset operations
b

brave-furniture-86963

10/04/2021, 2:52 AM
Thanks Benjy. Trying to understand this through an example: If: • the whole application uses 20 dependencies overall, and • a test file is testing 1 function of that application, and • that function only needs 3 dependencies, Then: • the test file will be run in a process that has the 3 dependencies only ?
h

happy-kitchen-89482

10/04/2021, 3:11 AM
Correct! (plus the transitive dependencies of those 3 direct dependencies)
b

brave-furniture-86963

10/04/2021, 3:35 AM
I see. So, the granularity is defined at the test file level, regardless of the how the test targets were set up ?
h

happy-kitchen-89482

10/04/2021, 12:58 PM
Exactly
pytest is run separately on each file
b

brave-furniture-86963

10/04/2021, 2:38 PM
I read that this behavior can be disabled by the user to use the repository.pex for all test files. Can you please point me to how to disable it? Also, do you envision making the granularity configurable by the user ? maybe make it at the test target level instead of the test file level? That way the user has control on the level of granularity.
How are data dependencies treated ? do they get copied over for each test process separately ?
h

happy-kitchen-89482

10/06/2021, 5:36 PM
Re changing the execution granularity, we have no plans to do that at the moment. What would be the advantage of supporting that, in your case?
And re data dependencies, do you mean like files () and resources () targets?
b

brave-furniture-86963

10/06/2021, 6:06 PM
Yes, our tests runtime went up 4x switching to pants. We have large data dependencies and we think they’re causing pants to take too long to run the tests, since it might be copying them over to each test process. If the granularity is user defined through test targets, then we can have 1 test target for all the tests using these heavy data dependencies.
h

happy-kitchen-89482

10/06/2021, 7:53 PM
Got it
@witty-crayon-22786 Any thoughts on this?
w

witty-crayon-22786

10/06/2021, 7:59 PM
we’ve discussed having a setting to control the test granularity by batching multiple test files together into a run, and that might be a possibility here.
but first: @brave-furniture-86963: do you have a sense of how much time is spent resolving dependencies (building requirements.pex and pytest_runner) vs actually running tests?
and are you using a
constraints.txt
file as described here: https://www.pantsbuild.org/docs/python-third-party-dependencies#user-lockfile ?
b

brave-furniture-86963

10/06/2021, 8:02 PM
We have pinned dependencies in our
requirements.txt
files already. Do we still need
constraints.txt
?
w

witty-crayon-22786

10/06/2021, 8:04 PM
yes, unfortunately: as it stands, having a full transitive lockfile as described on that page is a significant performance boost. as mentioned there, we’re planning to fully automate the process in the next few weeks.
@brave-furniture-86963: but my first question is relevant in this case: if you’re spending a lot of time resolving, that would be the first thing to fix most likely. and using a lockfile helps with that.
b

brave-furniture-86963

10/06/2021, 8:10 PM
Good to know. I’ll give the
constraints.txt
a try and report back. Thanks!
Re
do you have a sense of how much time is spent resolving dependencies (building requirements.pex and pytest_runner) vs actually running tests?
With no cache, building requirements.pex and pytest_runner.pex takes 36 minutes. Running tests takes 57 minutes. Using
constraints.txt
helped cut the time of building pytest_runner.pex by half (18min down from 36min). But when using
constraints.txt
, running tests fails (on a different test in both tries I did) with something like this
Copy code
Error expanding output globs: Failed to scan directory "/private/var/folders/ky/6q3_4s852_51v_3nm1f633b40000gq/T/process-executionfoFmYt/": No such file or directory (os error 2)
w

witty-crayon-22786

10/06/2021, 11:01 PM
hm… which version of Pants is this?
b

brave-furniture-86963

10/06/2021, 11:02 PM
2.7.0+git0f39c178
Note that the same branch passed in CI. The above failure happened on my laptop only.
w

witty-crayon-22786

10/06/2021, 11:04 PM
interesting: the tag indicates that you’ve built Pants from source? that shouldn’t be relevant here, but.
did you change any other settings at the same time that you changed the constraints file? it shouldn’t really impact our capturing of the sandbox…
b

brave-furniture-86963

10/06/2021, 11:05 PM
w

witty-crayon-22786

10/06/2021, 11:05 PM
also, some more context on which process failed would be good: can you pass
--print-stacktrace
on your laptop, and include a bit more from above the error?
ah. got it, sorry.
b

brave-furniture-86963

10/06/2021, 11:17 PM
Copy code
Engine traceback:
  in select
  in pants.core.goals.test.run_tests
  in pants.core.goals.test.enrich_test_result (hcm/optimize/test/test_partition_optimizer.py:../../tests)
  in pants.backend.python.goals.pytest_runner.run_python_test (hcm/optimize/test/test_partition_optimizer.py:../../tests)
  in pants.engine.process.remove_platform_information
  in multi_platform_process
Traceback (no traceback):
  <pants native internals>
Exception: Failed to execute: Process {
    argv: [
        "./pytest_runner.pex_pex_shim.sh",
        "--cov-report=",
        "--cov-config=.coveragerc",
        "--cov=.",
        "hcm/optimize/test/test_partition_optimizer.py",
    ],
    env: {
        "PEX_EXTRA_SYS_PATH": ".",
        "PYTEST_ADDOPTS": "--color=yes",
    },
    working_directory: None,
    input_files: Digest {
        hash: Fingerprint<a3b4161d915c63fccbb4e68f12c4974a6bf3419e18b116a07c563dcf2f5dd491>,
        size_bytes: 1155,
    },
    output_files: {
        RelativePath(
            ".coverage",
        ),
    },
    output_directories: {
        RelativePath(
            "extra-output",
        ),
    },
    timeout: None,
    execution_slot_variable: None,
    description: "Run Pytest for hcm/optimize/test/test_partition_optimizer.py:../../tests",
    level: Debug,
    append_only_caches: {
        CacheName(
            "pex_root",
        ): CacheDest(
            ".cache/pex_root",
        ),
    },
    jdk_home: None,
    platform_constraint: None,
    is_nailgunnable: false,
    cache_scope: Successful,
}

Error expanding output globs: Failed to scan directory "/private/var/folders/ky/6q3_4s852_51v_3nm1f633b40000gq/T/process-executiony9aBUA/": No such file or directory (os error 2)
w

witty-crayon-22786

10/06/2021, 11:49 PM
that is very strange indeed… and this happens consistently with a variety of tests? i’m wondering whether the test did something to delete the directory it was running in (unlikely, but…).
another flag that might be useful would be `--no-process-execution-local-cleanup`… would allow us to inspect that sandbox to see whether there was anything peculiar left inside of it.
b

brave-furniture-86963

10/07/2021, 12:13 AM
what would I look for in the sandbox ?
Copy code
cd /private/var/folders/ky/6q3_4s852_51v_3nm1f633b40000gq/T/process-executionC9iwWM/
➜  process-executionC9iwWM ll | wc -l
      14
Copy code
➜  process-executionC9iwWM ./pytest_runner.pex_pex_shim.sh my_test
.....
.....
=================================== 13 passed, 11 warnings in 2.36s =================================================================
w

witty-crayon-22786

10/07/2021, 12:36 AM
Very strange. I can't think of any reason why it wouldn't be able to scan that. You might try temporarily setting
Copy code
[GLOBAL]
local_execution_root_dir
... to a different temporary directory, and see whether tests will pass in that location?
6 Views