https://pantsbuild.org/ logo
#general
Title
# general
a

alert-psychiatrist-14102

02/04/2023, 5:31 PM
For some reason building pex files on my machine (intel macos) is prohibitively slow. Building
requirements.pex
or
pytest.pex
can take a few minutes and it happens relatively frequently. Example -
Copy code
19:24:15.74 [INFO] Preserving local process execution dir /private/var/folders/1r/vwvwzp312ngc1kfys_7g6x_c0000gp/T/pants-sandbox-uTOWh9 for Building requirements.pex with 15 requirements: av, aws_lambda_powertools, boto3, botocore, ffmpeg_python, frozendict, grpcio, mediapipe, numpy, opencv_python, protobuf, pyarrow, pyzipper, requests, spring_config_client
19:25:31.39 [INFO] Completed: Building requirements.pex with 15 requirements: av, aws_lambda_powertools, boto3, botocore, ffmpeg_python, frozendict, grpcio, mediapipe, numpy, opencv_python, protobuf, pyarrow, pyzipper, requests, s... (19 characters truncated)
Did anyone encounter such slowness? Ideas on how can be improved?
h

happy-kitchen-89482

02/04/2023, 10:42 PM
I'm assuming you're using a lockfile? What are your interpreter constraints?
a

alert-psychiatrist-14102

02/05/2023, 6:29 AM
I am. The cost occurs after an update to the requirements.lock. So, even though less frequent, it still takes a lot of time.. Would be great to expedite. Is there a way of not running as pex while developing locally? interpreter -
Copy code
interpreter_constraints = ["==3.9.*"]
And I also wonder why it rebuilds
pytest_runner.pex
for every test. Aren't they the same?
h

happy-kitchen-89482

02/05/2023, 6:40 AM
The
pytest_runner.pex
is a "shell" pex that ties together
pytest
itself and your tests' requirements
So it changes across tests, because the requirements change
but note that it is just a thin wrapper venv, it doesn't actually contain anything, so it is trivial to build
When you say "an update to the requirements.lock", are you manually updating that file? Or is
generate-lockfiles
doing so?
And is the cost you're talking about incurred during that
generate-lockfiles
or on some subsequent, say,
test
run?
a

alert-psychiatrist-14102

02/05/2023, 6:45 AM
got it.. sometimes building the
pytest_runner.pex
can take more than 30s for me, so i wonder if i'm doing something wrong. I mean updating through
generate-lockdiles
, and the the cost is incuurred during the susequent
test
run
h

happy-kitchen-89482

02/05/2023, 6:55 AM
Hmm, so that cost is in the subsetting, which should be fast. And so should
pytest_runner.pex
, so I'm not sure what's going on here.
You mentioned "on my machine". Are you not seeing this on other machines?
a

alert-psychiatrist-14102

02/05/2023, 6:57 AM
Never tried it on another machine..:) What would be the best way to debug it? Any way i can generate a performance report?
h

happy-kitchen-89482

02/05/2023, 7:01 AM
Are you able to publish a repo to github that reproduces this? It would need your real requirements, and pants.toml, and BUILD files, but could have a dummy test that uses requirements but doesn't expose your real code.
a

alert-psychiatrist-14102

02/05/2023, 7:04 AM
sure, thanks!
I'll do it in the next few hours
the problem reproduces here when trying to run
Copy code
./pants test tests::
h

happy-kitchen-89482

02/05/2023, 3:57 PM
I get a 404 on that repo
Can you make it public?
a

alert-psychiatrist-14102

02/05/2023, 5:13 PM
h

happy-kitchen-89482

02/05/2023, 5:44 PM
OK, I can see the "Building X requirements for requirements.pex from the third_party/python/default.lock" processes taking a long time on a linux box too.
Oddly, if I run the tests one at a time (e.g.,
./pants --no-pantsd --no-local-cache test tests/a_test.py
etc.) then that requirements.pex build is fast.
Hmm, it's only the first run that was slow. Subsequent runs were much faster (with
--no-pantsd --no-local-cache
, so the work was still really being done)
OK, looks like your requirements.pex files are really huge. For test_a for example, it's ~250M. And we're creating separate pexes per test (or, more precisely, per unique subset of requirements).
These are your top offenders, for test_a:
Copy code
-rw-rw-r-- 1 benjy benjy  65M Feb  5 17:55 opencv_contrib_python-4.7.0.68-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-rw-r-- 1 benjy benjy  59M Feb  5 17:55 opencv_python-4.7.0.68-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-rw-r-- 1 benjy benjy  35M Feb  5 17:55 pyarrow-10.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-rw-r-- 1 benjy benjy  32M Feb  5 17:55 mediapipe-0.9.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-rw-r-- 1 benjy benjy  30M Feb  5 17:55 av-10.0.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-rw-r-- 1 benjy benjy  17M Feb  5 17:55 numpy-1.24.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
-rw-rw-r-- 1 benjy benjy  12M Feb  5 17:55 matplotlib-3.6.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
One thing you can do is set
Copy code
[python]
run_against_entire_lockfile = true
in pants.toml
This will create one big requirements.pex for your entire lockfile, and run all the tests against that, instead of subsetting
It means of course that all your tests will invalidate if any requirement changes, even if the test doesn't care about that requirement
That's the tradeoff
a

alert-psychiatrist-14102

02/05/2023, 6:04 PM
got it! thanks so much for looking into this. Is there an options to set
run_against_entire_lockfile
just for tests? I would not want all my deployments / dockers etc. to become huge bc of this flag
h

happy-kitchen-89482

02/05/2023, 6:11 PM
The documentation of that option states:
Copy code
This option does not affect packaging deployable artifacts, such as
            PEX files, wheels and cloud functions, which will still use just the exact
            subset of requirements needed.
So you're good!
a

alert-psychiatrist-14102

02/06/2023, 6:54 AM
cool, thx!
a

adorable-nest-62558

02/14/2023, 5:53 PM
With these heavy packages: https://pantsbuild.slack.com/archives/C046T6T9U/p1675620054923909?thread_ts=1675531860.346739&cid=C046T6T9U Is there no way to make sure run faster? Each file edit always try to rebuild pex:
Copy code
./pants --loop package ::
And it takes on the order of 250 seconds for me. Requirements I am using:
Copy code
ansicolors==1.1.8
setuptools>=56.2.0,<57
types-setuptools>=56.2.0,<58
pytest==7.1.3
fire==0.5.0
requests==2.28.2
types-requests==2.28.11.8
coloredlogs==15.0.1
pandas
transformers==4.26.0
datasets==2.9.0
torch==1.13.1
torchvision==0.14.1
firebase-admin==6.1.0
Copy code
python_requirements(name="reqs")

python_requirement(
    name="fire",
    requirements=["fire==0.5.0"],
)

python_requirement(
    name="ansicolors",
    requirements=["ansicolors==1.1.8"],
)

python_requirement(
    name="requests",
    requirements=["requests==2.28.2"],
)

python_requirement(
    name="pandas",
    requirements=["pandas"],
)

python_requirement(
    name="transformers",
    requirements=["transformers==4.26.0"],
)

python_requirement(
    name="datasets",
    requirements=["datasets==2.9.0"],
)

python_requirement(
    name="torch",
    requirements=["torch==1.13.1"],
)

python_requirement(
    name="torchvision",
    requirements=["torchvision==0.14.1"],
)

python_requirement(
    name="firebase-admin",
    requirements=["firebase-admin==6.1.0"],
)
h

happy-kitchen-89482

02/14/2023, 6:22 PM
I assume most of that time is due to re-zipping the pexes, since they must contain both your code (which you're editing) and all the large third-party deps
./pants --loop package ::
is an unusual idiom though
Usually people use
--loop
with
test
or
fmt
And then package once at the end when satisfied
a

adorable-nest-62558

02/14/2023, 6:24 PM
I was thinking of using
./pants --loop package ::
to create
main.pex
so that I avoid running
./pants run
and direct run via
main.pex
since building is so slow.
Is there anyway to cache the large third-party deps? Bazel seems to be able to cache, but pants cannot?
h

happy-kitchen-89482

02/14/2023, 6:24 PM
What are you doing with the resulting pex in that loop?
Pants is caching everything
but creating a new zipfile takes time, if you have large requirements
a

adorable-nest-62558

02/14/2023, 6:25 PM
after resulting pex is created, I try to run it
./main.pex
h

happy-kitchen-89482

02/14/2023, 6:25 PM
How many
pex_binary
targets do you have?
So why not do
./pants --loop run path/to/main.py
?
That should be much much faster, since it doesn't have to build a pex
1
a

adorable-nest-62558

02/14/2023, 6:26 PM
I just have two. pex_binary.
Copy code
pex_binary(
    name="main",
    entry_point="main.py",
    dependencies=[
        "pipelines:a",
        "pipelines:b",
        "pipelines:c",
        "pipelines:d",
        "3rdparty/python:reqs#pandas",
        "3rdparty/python:reqs#fire",
        "3rdparty/python:reqs#ansicolors",
        "3rdparty/python:reqs#coloredlogs",
    ],
)
ok yea, it looks like it is much faster. Thx!
1
h

happy-kitchen-89482

02/14/2023, 10:29 PM
Great!
6 Views