I think I asked before, but that was over 90 days ago. Sorry for asking again. • Is there a way to specify which python interpreter to use for a virtualenv export? https://www.pantsbuild.org/docs/setting-up-an-ide#python-third-party-dependencies-and-tools • Is there a way to specify which resolve I want exported from the cli? I want to allow people to export tool venvs, but I don't need it in CI so I don't want to disable it in

pants.toml

. • Can I export the same resolve multiple times, each with a different python interpreter? Background: For st2 CI, I need one virtualenv for python 3.8 and another for python 3.6 (3.6 is ancient I know. updating that is a separate effort). I will use the exported virtualenvs instead of a Makefile managed venv for processes that have not yet been configured in pants. Eventually, I'll be able to stop using the exported virtualenv and just use pants for everything, but this should help to bridge that gap.

Wondering about behaviors I should expect around removing dependencies and transitive dependent detection: Right now (Pants 2.13) if I remove

example/lib.py

I would expect files that import that like

example/app.py

to be an affected transitive dependent, but it doesn't seem like this is the case. Is there a way to enable this behavior if it exists? Or does that imply scanning deps on two different versions of the worktree?

https://www.pantsbuild.org/docs/project-introspection is great for general introspection of your project targets — are there any tools/tricks/commands to get a better understanding cache usage?

Is there a way to go back to the old way Pants generated the command at the top of the lockfile header? Some versions ago it was relative e.g.

./pants generate-lockfiles --resolve=my-resolve

. Recently it's become an absolute path which is quite annoying.

Is there a way to override the dependencies of a Python module in a target and discard the `import`s that take place in that particular file? 🧵

In PEX I can ship arbitrary directories using

-D

- these end up in the target

site-packages

. Is there a way to install data files outside

site-packages

as well, without using setup.py or Python build mechanisms? If not, I can repurpose

-D

to ship data files into, say

site-packages/some-data

. However is there an easy way to discover the path of the venv when running the pex-file? We build with pex tools but do not unpack the pex-file manually.

Hey folks, I have a use case in Python that I suspect isn't terribly special, and I wonder if folks could point me in the right direction. I have a codebase with a large library, and a handful of applications that use the library. The final distribution target is a container image, and I'm hoping to build the application itself as a Pex. When we ship the final build, the modules in the library are cythonized to shared library files. We do this today via setuptools

build_ext

, and implement our own cache so that we only rebuild modules that have changed. I can define a distribution target for the library in Pants that triggers the

build_ext

routine, but in this case our cache doesn't work. And when building the Pex, I'd like to bring along just those shared library files that are required and not the whole boatload. It seems like what I want is to be able to define a target for each eventual shared library file, with dependencies looking something like:

pex -> application code -> .so file -> .py source

Before I dive too deep into the plugin framework, any pointers or feedback? If I were to go the route of writing a plugin, any words of advice for how to think about this?

Is there a way to get

./pants list

to output only filenames and not the owning target name as well?

Where might Rust fall into the language prioritization for future releases? I would like to push our software development in the future from python to rust for a lot of our core tooling and I'm curious how that timeline may align with Pants development. I've tried out bazel's

rules_rust

and it seems to work fine, so we always have that as a fallback.

I am surprised to see what

./pants dependencies

does when dependency inference is disabled and enabled 🧵

A large (and knotted) portion of my monorepo is a django project. We would like start using pants to do mypy type checking on it and use mypy_django_plugin. The problem is mypy config plugin installation applies everywhere and installing this plugin requires that we add our entire django project as a mypy dependency, even when checking a file that is not part of the django code. What I need is to have different mypy configs (or a layer that modifies one) and

extra_requirements

for different segments of the codebase. Is this something pants might want to support natively? Any advise on what a plugin for it might look like?

👋 Hi everyone! I was following the following link in order to update our current pants version v.2.12.0 to v.2.14.0, but I was unsuccessful, could you please help me upgrade pants? link: https://www.pantsbuild.org/docs/upgrade-tips

One extra thing to add here. I needed to use the non-bundled versions of boto3/botocore to use more recent versions. I had to use the

PEX_INHERIT_PATH=fallback

- I couldn’t remove the

/var/runtime

from the python path 😞 https://pex.readthedocs.io/en/latest/api/vars.html#PEX_INHERIT_PATH

@billions-keyboard-33102 I want to make sure I track the disappointment. Is there something Pex isn't doing here you think it should or are you disappointed you can't remove /var/runtime from the

sys.path

? If its the latter, I think you can, you just don't use - I forget the terminology exactly - the standard AWS Lambda executor. They have an API to plug that.

Ok, it's called a "runtime": https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html

So, if it is the latter, just build the PEX with a

-D <dir>

where that dir includes an executable

bootstrap

file meeting their specs. If that proves difficult, perhaps a feature can be added to ... Pants I think, to do that for you. This feels 1 level higher than Pex, Pants knows AWS Lambda, Pex does not - currently.

Yes, I agree it’s at the pantsbuild level. I’m using this target: https://www.pantsbuild.org/docs/reference-python_awslambda

So, I'm getting lost, but, IIUC @billions-keyboard-33102 you chimed into this thread up above because you too are using a docker container for your lambda. If that's correct then I'd suggest not using `python_awslambda`and using

pex_binary

instead. That supports

PEX_INHERIT_PATH

config (https://www.pantsbuild.org/docs/reference-pex_binary#codeinherit_pathcode). You really only need

python_awslambda

for the lambda zip deployment scheme and even that is only historically true. You can now use a plain old PEX (

pex_binary

again) for that too; you just prefix your lambda entry point configuration value out in AWS with

__pex__.

to make PEX files as lambda zips work.

I’ve got a monorepo of Python Libraries and I’m struggling with some compiled modules. For example,

package-1

has a Pyrex file (

foo.pyx

) that gets compiled to a C Extension (

foo.c

) at build time. The issue that I’m having is with plain Python projects in the monorepo that depend on that module. So in

package-2

the following import raises an error,

from package_1.foo import bar

. My current workaround has been to create multiple resolves so that

package-1

is actually a third party dependency for

package-2

and the

from package_1.foo import bar

statement passes as expected. But I don’t really like that solution and now I’m stuck with an issue where in order to make a change to both

package-1

and

package-2

I have to go through the burdensome process of changing

package-1

and republishing it and then changing

package-2

. Is there any pants magic that will allow me to keep both of these projects using the same resolve (i.e.

package-2

depends on the compiled

package-1

instead of the uncompiled code)?

GoLang > Docker question... In Python, the dependency tree is automatically tracked when a Dockerfile references a pants artifacts such as:

FROM python:3.8
ENTRYPOINT ["/example_app"]
COPY apps.python.example_app/example_app_bin.pex /example_app

Are Go build artifacts tracked in the same way? Any examples or documentation to point me to?

Ah, sorry - no we’re using the zip deployment. We’ll try to migrate - thank you!

Ok, well, probaly best to migrate to containers, but zip works fine! Just switch to

pex_binary

like I said and ensure a few things like a

platforms

or

complete_platforms

that matches AWS (I can provide more info there is needed),

inherit_path

to solve your problem without having to set

PEX_*

env vars out in AWS and a handler spec like

__<http://pex__.my|pex__.my>_<http://package.my|package.my>_module:my_handler_func

.

great, thank you - I can use the lambda docker to get the complete_platforms - let me try now!

I will be killing the pantsbuild/lambdex project FWIW at some point in the new year since it is no longer needed. That's what currently powers the

python_awslambda

target / rules.

on installing tools — there seems to be a couple of options where pex seems to use some internals;

There is a predefined rule to go from PexProcess -> Process, so Get(ProcessResult, Process) will cause the engine to run PexProcess -> Process -> ProcessResult.

is it possible to create your own external tool installer which runs a specific command to get the environment correctly setup, similar to the pex machinery and

PexRequest

/

PexProcess

?

I want to make a macro that autogenerates a

pex_binary

and

docker_image

target for me for the majority of simple cases where we're just doing the standard base image and copying in a pex. Any recommendations on how to autopopulate the

COPY

line of the

Dockerfile

? I'd like to avoid requiring the user to specify the full path to the module that serves as the entrypoint for the

pex_binary

, but I'm not sure there's another way.

weird python type check issue… 🧵

I want to show my client a PoC how to test its python libraries with different python versions in the CI (GitHub actions), so I tried: 1. Make the CI a matrix on the desired python versions. But then I need to filter pants targets, to only test the targets for the only interpreter in scope. Clunky 😕 2. Have multiple python interpreters in scope in the CI, so that a single

./pants test ::

command tests all versions. Sounds cool on paper. But getting

pyenv

work fast in the CI is a job in itself. How do you people do that smartly on GitHub actions? Any pointer?

debugging some strange behavior with a virtualenv created using

./pants export --symlink-python-virtualenv

🧵

Is this a bug? I have

instructions

set to a list and

source=None

in a

docker_image

target, but I'm getting an error like "provides both a Dockerfile with the

source

field, and Dockerfile contents with the

instructions

field, which is not supported.". To fix, it's saying to set

source=None

or

instructions=None

. When I do

source=''

, things are happy.