brave-furniture-86963
09/27/2021, 7:11 PMrequirements.pex
and pytest_runner.pex
get built during ./pants test ::
?
I have defined 9 python_tests
targets in all in our repository, yet I’m seeing over 26 pytest_runner.pex
and over 31 requirements.pex
built (also these numbers change depending on how I structure my requirements.txt files).happy-kitchen-89482
09/27/2021, 7:57 PMpython_tests
target has multiple sources, each one will run separately.brave-furniture-86963
10/04/2021, 2:52 AMhappy-kitchen-89482
10/04/2021, 3:11 AMbrave-furniture-86963
10/04/2021, 3:35 AMhappy-kitchen-89482
10/04/2021, 12:58 PMbrave-furniture-86963
10/04/2021, 2:38 PMhappy-kitchen-89482
10/06/2021, 5:36 PMbrave-furniture-86963
10/06/2021, 6:06 PMhappy-kitchen-89482
10/06/2021, 7:53 PMwitty-crayon-22786
10/06/2021, 7:59 PMconstraints.txt
file as described here: https://www.pantsbuild.org/docs/python-third-party-dependencies#user-lockfile ?brave-furniture-86963
10/06/2021, 8:02 PMrequirements.txt
files already. Do we still need constraints.txt
?witty-crayon-22786
10/06/2021, 8:04 PMbrave-furniture-86963
10/06/2021, 8:10 PMconstraints.txt
a try and report back.
Thanks!do you have a sense of how much time is spent resolving dependencies (building requirements.pex and pytest_runner) vs actually running tests?With no cache, building requirements.pex and pytest_runner.pex takes 36 minutes. Running tests takes 57 minutes. Using
constraints.txt
helped cut the time of building pytest_runner.pex by half (18min down from 36min).
But when using constraints.txt
, running tests fails (on a different test in both tries I did) with something like this
Error expanding output globs: Failed to scan directory "/private/var/folders/ky/6q3_4s852_51v_3nm1f633b40000gq/T/process-executionfoFmYt/": No such file or directory (os error 2)
witty-crayon-22786
10/06/2021, 11:01 PMbrave-furniture-86963
10/06/2021, 11:02 PM2.7.0+git0f39c178
witty-crayon-22786
10/06/2021, 11:04 PMbrave-furniture-86963
10/06/2021, 11:05 PMwitty-crayon-22786
10/06/2021, 11:05 PM--print-stacktrace
on your laptop, and include a bit more from above the error?brave-furniture-86963
10/06/2021, 11:17 PMEngine traceback:
in select
in pants.core.goals.test.run_tests
in pants.core.goals.test.enrich_test_result (hcm/optimize/test/test_partition_optimizer.py:../../tests)
in pants.backend.python.goals.pytest_runner.run_python_test (hcm/optimize/test/test_partition_optimizer.py:../../tests)
in pants.engine.process.remove_platform_information
in multi_platform_process
Traceback (no traceback):
<pants native internals>
Exception: Failed to execute: Process {
argv: [
"./pytest_runner.pex_pex_shim.sh",
"--cov-report=",
"--cov-config=.coveragerc",
"--cov=.",
"hcm/optimize/test/test_partition_optimizer.py",
],
env: {
"PEX_EXTRA_SYS_PATH": ".",
"PYTEST_ADDOPTS": "--color=yes",
},
working_directory: None,
input_files: Digest {
hash: Fingerprint<a3b4161d915c63fccbb4e68f12c4974a6bf3419e18b116a07c563dcf2f5dd491>,
size_bytes: 1155,
},
output_files: {
RelativePath(
".coverage",
),
},
output_directories: {
RelativePath(
"extra-output",
),
},
timeout: None,
execution_slot_variable: None,
description: "Run Pytest for hcm/optimize/test/test_partition_optimizer.py:../../tests",
level: Debug,
append_only_caches: {
CacheName(
"pex_root",
): CacheDest(
".cache/pex_root",
),
},
jdk_home: None,
platform_constraint: None,
is_nailgunnable: false,
cache_scope: Successful,
}
Error expanding output globs: Failed to scan directory "/private/var/folders/ky/6q3_4s852_51v_3nm1f633b40000gq/T/process-executiony9aBUA/": No such file or directory (os error 2)
witty-crayon-22786
10/06/2021, 11:49 PMbrave-furniture-86963
10/07/2021, 12:13 AMcd /private/var/folders/ky/6q3_4s852_51v_3nm1f633b40000gq/T/process-executionC9iwWM/
➜ process-executionC9iwWM ll | wc -l
14
➜ process-executionC9iwWM ./pytest_runner.pex_pex_shim.sh my_test
.....
.....
=================================== 13 passed, 11 warnings in 2.36s =================================================================
witty-crayon-22786
10/07/2021, 12:36 AM[GLOBAL]
local_execution_root_dir