For some reason building pex files on my machine (intel macos) is prohibitively slow. Building

requirements.pex

or

pytest.pex

can take a few minutes and it happens relatively frequently. Example -

19:24:15.74 [INFO] Preserving local process execution dir /private/var/folders/1r/vwvwzp312ngc1kfys_7g6x_c0000gp/T/pants-sandbox-uTOWh9 for Building requirements.pex with 15 requirements: av, aws_lambda_powertools, boto3, botocore, ffmpeg_python, frozendict, grpcio, mediapipe, numpy, opencv_python, protobuf, pyarrow, pyzipper, requests, spring_config_client
19:25:31.39 [INFO] Completed: Building requirements.pex with 15 requirements: av, aws_lambda_powertools, boto3, botocore, ffmpeg_python, frozendict, grpcio, mediapipe, numpy, opencv_python, protobuf, pyarrow, pyzipper, requests, s... (19 characters truncated)

Did anyone encounter such slowness? Ideas on how can be improved?

I'm assuming you're using a lockfile? What are your interpreter constraints?

I am. The cost occurs after an update to the requirements.lock. So, even though less frequent, it still takes a lot of time.. Would be great to expedite. Is there a way of not running as pex while developing locally? interpreter -

interpreter_constraints = ["==3.9.*"]

The

pytest_runner.pex

is a "shell" pex that ties together

pytest

itself and your tests' requirements

got it.. sometimes building the

pytest_runner.pex

can take more than 30s for me, so i wonder if i'm doing something wrong. I mean updating through

generate-lockdiles

, and the the cost is incuurred during the susequent

test

run

Hmm, so that cost is in the subsetting, which should be fast. And so should

pytest_runner.pex

, so I'm not sure what's going on here.

Never tried it on another machine..:) What would be the best way to debug it? Any way i can generate a performance report?

Are you able to publish a repo to github that reproduces this? It would need your real requirements, and pants.toml, and BUILD files, but could have a dummy test that uses requirements but doesn't expose your real code.

sure, thanks!

I get a 404 on that repo

Sry, here - https://gitlab.com/neuralight_public/nl_pants

OK, I can see the "Building X requirements for requirements.pex from the third_party/python/default.lock" processes taking a long time on a linux box too.

got it! thanks so much for looking into this. Is there an options to set

run_against_entire_lockfile

just for tests? I would not want all my deployments / dockers etc. to become huge bc of this flag

The documentation of that option states:

This option does not affect packaging deployable artifacts, such as
            PEX files, wheels and cloud functions, which will still use just the exact
            subset of requirements needed.

cool, thx!

With these heavy packages: https://pantsbuild.slack.com/archives/C046T6T9U/p1675620054923909?thread_ts=1675531860.346739&cid=C046T6T9U Is there no way to make sure run faster? Each file edit always try to rebuild pex:

./pants --loop package ::

And it takes on the order of 250 seconds for me. Requirements I am using:

ansicolors==1.1.8
setuptools>=56.2.0,<57
types-setuptools>=56.2.0,<58
pytest==7.1.3
fire==0.5.0
requests==2.28.2
types-requests==2.28.11.8
coloredlogs==15.0.1
pandas
transformers==4.26.0
datasets==2.9.0
torch==1.13.1
torchvision==0.14.1
firebase-admin==6.1.0

python_requirements(name="reqs")

python_requirement(
    name="fire",
    requirements=["fire==0.5.0"],
)

python_requirement(
    name="ansicolors",
    requirements=["ansicolors==1.1.8"],
)

python_requirement(
    name="requests",
    requirements=["requests==2.28.2"],
)

python_requirement(
    name="pandas",
    requirements=["pandas"],
)

python_requirement(
    name="transformers",
    requirements=["transformers==4.26.0"],
)

python_requirement(
    name="datasets",
    requirements=["datasets==2.9.0"],
)

python_requirement(
    name="torch",
    requirements=["torch==1.13.1"],
)

python_requirement(
    name="torchvision",
    requirements=["torchvision==0.14.1"],
)

python_requirement(
    name="firebase-admin",
    requirements=["firebase-admin==6.1.0"],
)

I assume most of that time is due to re-zipping the pexes, since they must contain both your code (which you're editing) and all the large third-party deps

I was thinking of using

./pants --loop package ::

to create

main.pex

so that I avoid running

./pants run

and direct run via

main.pex

since building is so slow.

What are you doing with the resulting pex in that loop?

after resulting pex is created, I try to run it

./main.pex

How many

pex_binary

targets do you have?

I just have two. pex_binary.

pex_binary(
    name="main",
    entry_point="main.py",
    dependencies=[
        "pipelines:a",
        "pipelines:b",
        "pipelines:c",
        "pipelines:d",
        "3rdparty/python:reqs#pandas",
        "3rdparty/python:reqs#fire",
        "3rdparty/python:reqs#ansicolors",
        "3rdparty/python:reqs#coloredlogs",
    ],
)

Great!