#general
Title
b
brave-furniture-86963
09/27/2021, 7:11 PMHi all!
What determines how many
requirements.pexpytest_runner.pex./pants test ::python_testspytest_runner.pexrequirements.pexh
happy-kitchen-89482
09/27/2021, 7:57 PMHi @brave-furniture-86963! Pants runs each test file in its own process, with its own requirements. So if a
python_testsThis allows each test file to run with just its actual requirements, and not be invalidated when unrelated requirements change.
But (assuming there are no conflicting versions involved) Pants does one pip resolve, and then composes the the per-file requirements as a subset of that resolve
so you shouldn't be seeing 31 slow pip runs
you should see 1 slow pip run then 31 fast subset operations
b
brave-furniture-86963
10/04/2021, 2:52 AMThanks Benjy. Trying to understand this through an example:
If:
• the whole application uses 20 dependencies overall, and
• a test file is testing 1 function of that application, and
• that function only needs 3 dependencies,
Then:
• the test file will be run in a process that has the 3 dependencies only ?
h
happy-kitchen-89482
10/04/2021, 3:11 AMCorrect! (plus the transitive dependencies of those 3 direct dependencies)
b
brave-furniture-86963
10/04/2021, 3:35 AMI see. So, the granularity is defined at the test file level, regardless of the how the test targets were set up ?
h
happy-kitchen-89482
10/04/2021, 12:58 PMExactly
pytest is run separately on each file
b
brave-furniture-86963
10/04/2021, 2:38 PMI read that this behavior can be disabled by the user to use the repository.pex for all test files. Can you please point me to how to disable it?
Also, do you envision making the granularity configurable by the user ? maybe make it at the test target level instead of the test file level? That way the user has control on the level of granularity.
How are data dependencies treated ? do they get copied over for each test process separately ?
h
happy-kitchen-89482
10/06/2021, 5:36 PMRe changing the execution granularity, we have no plans to do that at the moment. What would be the advantage of supporting that, in your case?
And re data dependencies, do you mean like files () and resources () targets?
b
brave-furniture-86963
10/06/2021, 6:06 PMYes, our tests runtime went up 4x switching to pants. We have large data dependencies and we think they’re causing pants to take too long to run the tests, since it might be copying them over to each test process.
If the granularity is user defined through test targets, then we can have 1 test target for all the tests using these heavy data dependencies.
h
happy-kitchen-89482
10/06/2021, 7:53 PMGot it
@witty-crayon-22786 Any thoughts on this?
w
witty-crayon-22786
10/06/2021, 7:59 PMwe’ve discussed having a setting to control the test granularity by batching multiple test files together into a run, and that might be a possibility here.
but first: @brave-furniture-86963: do you have a sense of how much time is spent resolving dependencies (building requirements.pex and pytest_runner) vs actually running tests?
and are you using a
constraints.txtb
brave-furniture-86963
10/06/2021, 8:02 PMWe have pinned dependencies in our
requirements.txtconstraints.txtw
witty-crayon-22786
10/06/2021, 8:04 PMyes, unfortunately: as it stands, having a full transitive lockfile as described on that page is a significant performance boost. as mentioned there, we’re planning to fully automate the process in the next few weeks.
@brave-furniture-86963: but my first question is relevant in this case: if you’re spending a lot of time resolving, that would be the first thing to fix most likely. and using a lockfile helps with that.
b
brave-furniture-86963
10/06/2021, 8:10 PMGood to know. I’ll give the
constraints.txtRe
do you have a sense of how much time is spent resolving dependencies (building requirements.pex and pytest_runner) vs actually running tests?With no cache, building requirements.pex and pytest_runner.pex takes 36 minutes. Running tests takes 57 minutes. Using
constraints.txtconstraints.txt Copy code
Error expanding output globs: Failed to scan directory "/private/var/folders/ky/6q3_4s852_51v_3nm1f633b40000gq/T/process-executionfoFmYt/": No such file or directory (os error 2)w
witty-crayon-22786
10/06/2021, 11:01 PMhm… which version of Pants is this?
b
brave-furniture-86963
10/06/2021, 11:02 PM2.7.0+git0f39c178Note that the same branch passed in CI. The above failure happened on my laptop only.
w
witty-crayon-22786
10/06/2021, 11:04 PMinteresting: the tag indicates that you’ve built Pants from source? that shouldn’t be relevant here, but.
did you change any other settings at the same time that you changed the constraints file? it shouldn’t really impact our capturing of the sandbox…
b
brave-furniture-86963
10/06/2021, 11:05 PMI downloaded the pex from https://github.com/pantsbuild/pants/releases/download/release_2.7.0/pants.2.7.0.pex
w
witty-crayon-22786
10/06/2021, 11:05 PMalso, some more context on which process failed would be good: can you pass
--print-stacktraceah. got it, sorry.
b
brave-furniture-86963
10/06/2021, 11:17 PM Copy code
Engine traceback:
in select
in pants.core.goals.test.run_tests
in pants.core.goals.test.enrich_test_result (hcm/optimize/test/test_partition_optimizer.py:../../tests)
in pants.backend.python.goals.pytest_runner.run_python_test (hcm/optimize/test/test_partition_optimizer.py:../../tests)
in pants.engine.process.remove_platform_information
in multi_platform_process
Traceback (no traceback):
<pants native internals>
Exception: Failed to execute: Process {
argv: [
"./pytest_runner.pex_pex_shim.sh",
"--cov-report=",
"--cov-config=.coveragerc",
"--cov=.",
"hcm/optimize/test/test_partition_optimizer.py",
],
env: {
"PEX_EXTRA_SYS_PATH": ".",
"PYTEST_ADDOPTS": "--color=yes",
},
working_directory: None,
input_files: Digest {
hash: Fingerprint<a3b4161d915c63fccbb4e68f12c4974a6bf3419e18b116a07c563dcf2f5dd491>,
size_bytes: 1155,
},
output_files: {
RelativePath(
".coverage",
),
},
output_directories: {
RelativePath(
"extra-output",
),
},
timeout: None,
execution_slot_variable: None,
description: "Run Pytest for hcm/optimize/test/test_partition_optimizer.py:../../tests",
level: Debug,
append_only_caches: {
CacheName(
"pex_root",
): CacheDest(
".cache/pex_root",
),
},
jdk_home: None,
platform_constraint: None,
is_nailgunnable: false,
cache_scope: Successful,
}
Error expanding output globs: Failed to scan directory "/private/var/folders/ky/6q3_4s852_51v_3nm1f633b40000gq/T/process-executiony9aBUA/": No such file or directory (os error 2)w
witty-crayon-22786
10/06/2021, 11:49 PMthat is very strange indeed… and this happens consistently with a variety of tests? i’m wondering whether the test did something to delete the directory it was running in (unlikely, but…).
another flag that might be useful would be `--no-process-execution-local-cleanup`… would allow us to inspect that sandbox to see whether there was anything peculiar left inside of it.
b
brave-furniture-86963
10/07/2021, 12:13 AMwhat would I look for in the sandbox ?
Copy code
cd /private/var/folders/ky/6q3_4s852_51v_3nm1f633b40000gq/T/process-executionC9iwWM/
➜ process-executionC9iwWM ll | wc -l
14 Copy code
➜ process-executionC9iwWM ./pytest_runner.pex_pex_shim.sh my_test
.....
.....
=================================== 13 passed, 11 warnings in 2.36s =================================================================w
witty-crayon-22786
10/07/2021, 12:36 AMVery strange. I can't think of any reason why it wouldn't be able to scan that. You might try temporarily setting
Copy code
[GLOBAL]
local_execution_root_dir... to a different temporary directory, and see whether tests will pass in that location?
6 Views