alert-psychiatrist-14102
02/04/2023, 5:31 PMrequirements.pex
or pytest.pex
can take a few minutes and it happens relatively frequently.
Example -
19:24:15.74 [INFO] Preserving local process execution dir /private/var/folders/1r/vwvwzp312ngc1kfys_7g6x_c0000gp/T/pants-sandbox-uTOWh9 for Building requirements.pex with 15 requirements: av, aws_lambda_powertools, boto3, botocore, ffmpeg_python, frozendict, grpcio, mediapipe, numpy, opencv_python, protobuf, pyarrow, pyzipper, requests, spring_config_client
19:25:31.39 [INFO] Completed: Building requirements.pex with 15 requirements: av, aws_lambda_powertools, boto3, botocore, ffmpeg_python, frozendict, grpcio, mediapipe, numpy, opencv_python, protobuf, pyarrow, pyzipper, requests, s... (19 characters truncated)
Did anyone encounter such slowness? Ideas on how can be improved?happy-kitchen-89482
02/04/2023, 10:42 PMalert-psychiatrist-14102
02/05/2023, 6:29 AMinterpreter_constraints = ["==3.9.*"]
pytest_runner.pex
for every test. Aren't they the same?happy-kitchen-89482
02/05/2023, 6:40 AMpytest_runner.pex
is a "shell" pex that ties together pytest
itself and your tests' requirementsgenerate-lockfiles
doing so?generate-lockfiles
or on some subsequent, say, test
run?alert-psychiatrist-14102
02/05/2023, 6:45 AMpytest_runner.pex
can take more than 30s for me, so i wonder if i'm doing something wrong.
I mean updating through generate-lockdiles
, and the the cost is incuurred during the susequent test
runhappy-kitchen-89482
02/05/2023, 6:55 AMpytest_runner.pex
, so I'm not sure what's going on here.alert-psychiatrist-14102
02/05/2023, 6:57 AMhappy-kitchen-89482
02/05/2023, 7:01 AMalert-psychiatrist-14102
02/05/2023, 7:04 AMhappy-kitchen-89482
02/05/2023, 3:57 PMalert-psychiatrist-14102
02/05/2023, 5:13 PMhappy-kitchen-89482
02/05/2023, 5:44 PM./pants --no-pantsd --no-local-cache test tests/a_test.py
etc.) then that requirements.pex build is fast.--no-pantsd --no-local-cache
, so the work was still really being done)-rw-rw-r-- 1 benjy benjy 65M Feb 5 17:55 opencv_contrib_python-4.7.0.68-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-rw-r-- 1 benjy benjy 59M Feb 5 17:55 opencv_python-4.7.0.68-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-rw-r-- 1 benjy benjy 35M Feb 5 17:55 pyarrow-10.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-rw-r-- 1 benjy benjy 32M Feb 5 17:55 mediapipe-0.9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-rw-r-- 1 benjy benjy 30M Feb 5 17:55 av-10.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-rw-r-- 1 benjy benjy 17M Feb 5 17:55 numpy-1.24.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-rw-r-- 1 benjy benjy 12M Feb 5 17:55 matplotlib-3.6.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
[python]
run_against_entire_lockfile = true
alert-psychiatrist-14102
02/05/2023, 6:04 PMrun_against_entire_lockfile
just for tests?
I would not want all my deployments / dockers etc. to become huge bc of this flaghappy-kitchen-89482
02/05/2023, 6:11 PMThis option does not affect packaging deployable artifacts, such as
PEX files, wheels and cloud functions, which will still use just the exact
subset of requirements needed.
alert-psychiatrist-14102
02/06/2023, 6:54 AMadorable-nest-62558
02/14/2023, 5:53 PM./pants --loop package ::
And it takes on the order of 250 seconds for me.
Requirements I am using:
ansicolors==1.1.8
setuptools>=56.2.0,<57
types-setuptools>=56.2.0,<58
pytest==7.1.3
fire==0.5.0
requests==2.28.2
types-requests==2.28.11.8
coloredlogs==15.0.1
pandas
transformers==4.26.0
datasets==2.9.0
torch==1.13.1
torchvision==0.14.1
firebase-admin==6.1.0
python_requirements(name="reqs")
python_requirement(
name="fire",
requirements=["fire==0.5.0"],
)
python_requirement(
name="ansicolors",
requirements=["ansicolors==1.1.8"],
)
python_requirement(
name="requests",
requirements=["requests==2.28.2"],
)
python_requirement(
name="pandas",
requirements=["pandas"],
)
python_requirement(
name="transformers",
requirements=["transformers==4.26.0"],
)
python_requirement(
name="datasets",
requirements=["datasets==2.9.0"],
)
python_requirement(
name="torch",
requirements=["torch==1.13.1"],
)
python_requirement(
name="torchvision",
requirements=["torchvision==0.14.1"],
)
python_requirement(
name="firebase-admin",
requirements=["firebase-admin==6.1.0"],
)
happy-kitchen-89482
02/14/2023, 6:22 PM./pants --loop package ::
is an unusual idiom though--loop
with test
or fmt
adorable-nest-62558
02/14/2023, 6:24 PM./pants --loop package ::
to create main.pex
so that I avoid running ./pants run
and direct run via main.pex
since building is so slow.happy-kitchen-89482
02/14/2023, 6:24 PMadorable-nest-62558
02/14/2023, 6:25 PM./main.pex
happy-kitchen-89482
02/14/2023, 6:25 PMpex_binary
targets do you have?./pants --loop run path/to/main.py
?adorable-nest-62558
02/14/2023, 6:26 PMpex_binary(
name="main",
entry_point="main.py",
dependencies=[
"pipelines:a",
"pipelines:b",
"pipelines:c",
"pipelines:d",
"3rdparty/python:reqs#pandas",
"3rdparty/python:reqs#fire",
"3rdparty/python:reqs#ansicolors",
"3rdparty/python:reqs#coloredlogs",
],
)
happy-kitchen-89482
02/14/2023, 10:29 PM