Hi all, I’m trying to create a

pex_binary

target with only the dependencies (including first-party dependencies) of a particular target (actually, a

python_sources

target generator). Is there a way I can tell pants to do this with the

dependencies

field of the

pex_binary

target? The use case is that I have some workflow code which is uploaded to my Docker image by a separate tool (Flyte), so I only want my pex that I’m unpacking into a venv to include 3rd party and 1st party dependency code. Thanks!

unless i’m missing something, you should be able to accomplish that by including the

python_sources

target in the

dependencies

list of the

pex_binary

?

That then includes the generated

python_source

targets in the pex. So if my directory structure is:

.
├── src
│   ├── cmi_db
│   └── cmi_orchestration
└── pants.toml

My

pex_binary

should have code from 3rd party and

cmi_db

in it, but not

cmi_orchestration

, where code in

cmi_orchestration

imports code from

cmi_db

(and not vice-versa), so pants can infer the first-party dependency

This sounds very close to the docker optimization described here where you split first-party/third-party code into two different stages: https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/

Yep, that’s the pattern I’m following, except if I list

src/cmi_orchestration

as the

pex_binary

dependency

, then that code is also included in the

srcs

. What I want is all the dependencies of

src/cmi_orchestration

, without having to list them manually

ah, i see what you’re saying. only the dependencies, but not the target itself.

Yes, precisely 🙂

i’m not aware of a way to do that via target declarations. but is it possible to refactor the code to split the portion which has the dependencies you’d like to package, from the other portion which you don’t want to package?

They are already,

src/cmi_db

is the first party code upon which the

src/cmi_orchestration

depends, so pants can infer the dependency, and

src/cmi_orchestration

is the code uploaded by Flyte into the container. I could list

src/cmi_db

in the

dependencies

of the

pex_binary

but of course there are many more first-party packages than

src/cmi_db

that are dependencies of

src/cmi_orchestration

🙂 So listing them all manually is feasible, but likely to go out-of-date as the code in

src/cmi_orchestration

changes, and since pants already knows about the dependencies, I was hoping there’d be a way to specify this

What I've done for some of our plugin workflows is create a "dummy" python file that is just used to ensure the correct extra deps get pulled in at build time.

Oh, that’s an interesting idea. Thanks @gorgeous-winter-99296 and @witty-crayon-22786 for the quick responses!

One note to be aware of that for 1st-party code, you're getting a transitive but explicit closure, not the package. So you'll need to ensure all files you need at runtime are referenced by

import plugin1

f.ex... If that is just an empty

__init__.py

you'll likely get less than you expect.

pants dependencies --dependencies-transitive cmd/app1.py

Will list that whole closure.