Hey y'all! Currently exploring Pants as a solution...
# general
b
Hey y'all! Currently exploring Pants as a solution to help implement a Python monorepo... but am definitely hitting a few roadblocks, most notably with DevEx. My immediate use-cases are numerous: 1. Extracting copy-pasted library code across multiple repos into a library. 2. Pulling all those repos (each with 1 or more AWS Lambdas) into the monorepo. 3. Creating better E2E test suites integrating workflows across microservices locally with mocked external dependencies, to circumvent lengthy E2E test cycles running AWS with stateful external dependencies. 4. Consolidating AWS CDK codebases to be able to dynamically import relevant CDK dependencies from one stack, into another stack. 5. Being able to run AWS CDK applications to synth and deploy AWS resources including Lambdas (which have a particularly strange requirement in terms of building source to a local directory before upload). • I'm curious if there's some CLI features that allows
tailor
to be run such that it actually writes the dependencies (for things like python libraries of a python source) to the BUILD file...? I'm struggling to see why it's a good pattern to have everything inferred in the CLI without any static output that can be searched, verified, etc. without having to run
peek
. • I'm confused how Pants works with Venvs or lockfiles... how does Pants know what versions of python or packages to use? How does it know which python executable to use? Does it make a Venv under the hood? I'm not familar with Pex, so maybe I'm missing something here... but I'm having a bit of trouble understanding how to transition from a Venv+Poetry+Make approach, to Pants. • What sort of development workflow would Pants ultimately allow me to have? For example, if I update a library version, would it have a command to say "run all the tests that transitively depend on [updated package]"? If I update a library used by multiple microservices, I can use the change detection to deploy those services? Please forgive my utter dismay and confusion, as I'm still just scratching the surface of this tool and have little to no support from my team so I'm shooting in the dark on this. I've never used a monorepo tool in JS/TS or any other language/framework, so I'm still trying to wrap my head around this approach (versus libraries). Thanks in advance for any and all information anyone can provide here!
👀 1
m
Ok, so I am not a Pants dev, but I can share my experience. I am an ML engineer and I used Blaze a lot in the past. So this is my personal view.
I'm curious if there's some CLI features that allows
tailor
to be run such that it actually writes the dependencies (for things like python libraries of a python source) to the BUILD file...? I'm struggling to see why it's a good pattern to have everything inferred in the CLI without any static output that can be searched, verified, etc. without having to run
peek
.
I understand where you are coming from. You would like to understand how things are inferred in case of debugging etc. In my experience, Pants import inference is very reliable and I am glad I don't need to explicitly declare everything like Blaze asks me to.
I'm confused how Pants works with Venvs or lockfiles... how does Pants know what versions of python or packages to use? How does it know which python executable to use? Does it make a Venv under the hood? I'm not familar with Pex, so maybe I'm missing something here... but I'm having a bit of trouble understanding how to transition from a Venv+Poetry+Make approach, to Pants.
From the dev point of view, Pants does create a venv (though implemented with pex). I lock my dependencies so I know exactly which dependencies Pants will use when it creates a venv for a target. Note Pants create venv per target not one large venv for everything. This makes things more portable.
What sort of development workflow would Pants ultimately allow me to have? For example, if I update a library version, would it have a command to say "run all the tests that transitively depend on [updated package]"? If I update a library used by multiple microservices, I can use the change detection to deploy those services?
Yes, Pants understands the dependency DAG and it will only run targets that are affected. This is why it is a powerful monorepo build tool. It will cache results that don't need to be re-run, and only do the work that needs to be done!
h
Hi! It’s my bedtime, but I will have a closer look tomorrow and respond.
h
From the dev point of view, Pants does create a venv (though implemented with pex). I lock my dependencies so I know exactly which dependencies Pants will use when it creates a venv for a target. Note Pants create venv per target not one large venv for everything. This makes things more portable.
Is it called sandbox? For example, when we do
pants test path/to/package:test
, does it create some sandbox and only install the necessary dependencies for the test? Not the whole universe of the dependencies of the whole monorepo, right?
m
Yes that is correct. My understanding is that it creates a sandbox and a venv in the form of pex and execute it. I learnt a lot how Pex works on the surface level (again I am not a Pants dev) using the following flags:
Copy code
pants --pex-verbosity=9 --print-stacktrace -ldebug --keep-sandboxes=always
h
Hi! Referring back to your questions:
• There is no way to write inferred deps out into the BUILD files. The disadvantages of doing so were deemed to outweigh the advantages. FWIW dep inference is written in Rust and is very fast (probably faster than parsing BUILD files…), and is cached of course.
• Pants selects a python interpreter version based on your configured interpreter constraints. It looks for a suitable interpreter on a defined search path, or it can install one.
• Pants selects the versions of 3rdparty deps to use by consulting the relevant lockfile.
• It uses Pex to build an immutable, cached venv that is then symlinked into the sandbox. So yes, it makes a venv under the hood, more or less.
You can have workflows like: • “run all tests that depend (transitively) on git changes” with
pants --changed-since=main (--changed-transitive) test
• “find all cloud functions that depend on file X and deploy them”
And that sort of thing
Some of these are achieved using
xargs
and piping the output of
pants list --filter...
to some other pants command