hey, friends! I’m looking for any guide or discussion of how can pants package process be optimised from sometimes 10 minutes to something really faster 🙂 Can you share any links or something like this?

You'll have to be much more specific, I'm afraid 😅 What backends are you using? What are you packages? What settings are you applying? Etc...

i’m using pants only to work with python [GLOBAL] pants_version = “2.15.0” backend_packages = [ “pants.backend.shell”, “pants.backend.python”, “pants.backend.docker”, “pants.backend.awslambda.python” ] pants_ignore = [ “/backend/“, “/front/“, “/hasura/“, “/_scripts/“, “/nginx/“, “/li_profile_scraper” ] [python] interpreter_constraints = [“CPython>=3.10"] [anonymous-telemetry] enabled = false nothing special 🙂 Doing the build of pex on mac m1 to linjux sometimes, but that doesn’t seem to impact build time significantly

Are you using the default

layout

of

zipapp

? Try

loose

what triggers me is just adding any line in lib, for example import os can require the whole long build

See https://www.pantsbuild.org/docs/reference-pex_binary#codelayoutcode

thanks, yes, i was using default settings, will experiment with this

Ah well that's a different story, then. It shouldn't be that touching a firstparty source triggers rebuilding thirdparty requirements

hell yeah, do you know any ideas of how to deal with it?

So, to be clear, you're still seeing the long times with using

loose

layout?

unfortunately yes. doing my second build with: pex_binary( name=“db_triggers_pex”, entry_point=“__main__.py”, dependencies=[ “:db_triggers” ], output_path=“db_triggers.pex”, environment=“linux_docker”, layout=‘loose’ ) and still: ⠙ 89.39s Building db_triggers.pex with 13 requirements: confluent-kafka, countryinfo, dataclasses, fastapi, fastavro

Oh this is happening in a docker image? Yeah OK I'm not as surprised then

yeah, it’s a pex file, specifed for linux env, instead of M1

Yeah I think you're hitting an unfortunate side-effect of development having been made with assumptions. Mostly our Python stack relies heavily on the PEX cache, which is separate from Pants'. And that cache isn't shared with the docker environments (it really can't be). So when you build inside the docker environment, you'll be starting a bit from scratch 😕

oh, crap, Joshua! now i’m starting to get it. and if i get the nature of pex right (build everything in 1 file) - even moving requirements to Dockerfile will not help, correct?

Yeah there's nothing that'd help this today. Theoretically, Pants should be able to do this "the right way" but we just rely on Pex too much (and it's hard to not do that. Python's packaging ecosystem is VERY complex and Pex gets it right 😕)

A somewhat related question. When I run regular

python_source

targets, i hit some slowness (the question below). I suspect overriding the layout for

pex

in this setting would be helpful as well. Do you know if this is possible?

well, @bitter-ability-32190 what you are saying seems very sad. Ok, let’s say the right workaround is - to have multiple pants processes without docker. at least for testing purposes, and when it works locally - then I do long-packaging and deploy. is it the right strategy from your opinion?

I can't say for certain, but yeah I think doing thing locally will always be fastest.

Fwiw, each docker image will have its own PEX cache as a docker volume: so that cache can warm up too. It just can't be shared with the host.

Doing the build of pex on mac m1 to linjux sometimes, but that doesn’t seem to impact build time significantly

Does this mean that you're building the pex natively on your mac host, and it's still taking a long time? As in, unrelated to Docker?

thanks a lot, will research it