Is there a way in pex land (rather, pants + pex land) to only package up pre-compiled python files? “Compiled” in the python bytecode sense. This is assuming the same interpreter is doing the compiling and running of said code

I’m getting an error when uploading a cache with pants that is being caused by pants not respecting the max Grpc message size:

[WARN] Failed to write to remote cache (1 occurrences so far): ResourceExhausted: "grpc: received message larger than max (5812170 vs. 4194304)"

If I increase the max receivable size on the server from the 4MB default, the cache starts working. Theoretically, AFAIK, client and server should automatically agree on a max message size and honor that throughout the message exchange. I tried looking up how pants is using

tonic

but couldn’t see anything exotic there in terms of message size. Has anyone seen anything like that happen?

Hey all, is there an explicit way I can tell the pants builder to only pull a specific requests/urllib3? As of May 4th, I noticed our builds are failing due to the OpenSSL mismatch between 1.1.1+ and 1.1.01. We require at the least

requests==2.28.2,<2.30

and

urllib3==1.26.1,<2

but the recent change to where requests and urllib3 are puled from are breaking our builds. Is there a

legacy

flag or something we can put onto our builds to ignore pulling

requests>=2.30

and

urllib3>=2

?

Hi all, I’m running into the following error with `./pants run Dockerfile`:

BuildKit is enabled but the buildx component is missing or broken.
Install the buildx component to build images with BuildKit:
<https://docs.docker.com/go/buildx/>

Notes: • I’m on an Intel-chip Macbook • I installed the most recent Docker Desktop • With

--keep-sandboxes=on_failure

, I can run the

__run.sh

and the docker build succeeds with

DOCKER_BUILDKIT=1

(and the image appears in my docker desktop) It seems like my Docker installation is able to build with BuildKit enabled, but when running through Pants it doesn’t detect buildx? I’m new to Pants, so any direction would very appreciated! Let me know what other information I can provide. Thanks!

Am I doing something wrong, or is it as designed that running "pants package ::" will rebuild all docker images, even if there has been no change in their dependecies?

Hi, just bringing up a golang question that i had a couple weeks ago again. I did find this conversation about this same issue, and was wondering if anything has changed since then? Specifically about consuming local versions of golang modules instead of doing tagged releases and consuming them as 3rd parties. If it hasn't is there an estimate to how much effort it would be to add this capability?

Maybe related to the above question from Sean re: urllib, I’m getting this error:

ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with LibreSSL 2.8.3. See: <https://github.com/urllib3/urllib3/issues/2168>

I was not previously getting this error, it started yesterday (Sean said May 4 for them). I’ve tried recreating my previous setup using the steps below. (FYI our pants_version is 2.13.0) Steps 1. Made a fresh clone of our repo 2. Made a fresh branch from an earlier commit hash (April 25) 3. Made a fresh virtual env (I don’t think this matters for Pants, but did it anyway) 4. Cleaned out the

~/.cache/pants

dir on my local system 5. Ran

./pants --version

, which fails with the above ImportError. What am I missing? I can only think of 2 reasons this could happen: 1. Some permanent change to my system was made that’s somehow influencing Pants (though I don’t believe it should, since it’s hermetic execution) 2. During Pants bootstrapping, live resources are pulled independent of local configuration (or there’s a config setting that could fix this, but I do not yet know of it) I’m new to Pants, so thank you for your help and patience!

Getting an error when I run `./pants generate-lockfiles`:

Engine traceback:
  in `generate-lockfiles` goal

ProcessExecutionFailure: Process 'Generate lockfile for python-default' failed with exit code 1.
stdout:

stderr:
not enough values to unpack (expected 2, got 1)

I've narrowed it down to a package that's in my

requirements.txt

as

package @ git+<https://github.com/owner/package@main>

. Is there something in that syntax I need to change, or is there something else going on?

Hi everyone. I've tried searching this channel to no avail. I have successfully built a Pex file and have it running in a Docker container with

pants run

. Everything works perfectly. I'm able to pass environment variables in with

--docker-run-args="-e MY_VAR=var123"

, and this also works. However, if I do this

--docker-run-args="--env-file .env"

, assuming that the .env file is in the current working directory, I get the following error:

docker: open .env: no such file or directory.

Is loading environment variables from a file supported by Pants in this case? I'm on version 2.16.0rc1.

Just had a very tedious incident caused by a missing file in a pex_binary. There was a warning, can't find an owner for an import, but the package built and deployed. Am I missing a way to make that fail the build? ignore_errors isn't set to True anywhere that I can see.

Hi all, does anyone know if it's possible to pass command line arguments into a script being run by

experimental_run_shell_command

? e.g for Python I can do:

./pants run path/to/file.py -- --arg1=foo

. Any equivalent for shell, other than hard-coding them into the

command

field of the rule?

Hi all, has anyone ever seen artificial and intermittent dependency cycles show up? e.g.

IntrinsicError: The dependency graph contained a cycle:
  @rule(pants.core.util_rules.source_files.determine_source_files()) <-
@rule(pants.engine.internals.graph.hydrate_sources(third_party/glog:glog))
@rule(pants.backend.shell.util_rules.shell_command.shell_command_in_sandbox())
@rule(pants.backend.shell.util_rules.shell_command.run_adhoc_result_from_target())
@rule(pants.core.util_rules.source_files.determine_source_files()) <-

As you can guess, we've thrown together C++ support with experiemental_shell_command rules. These aren't true dependency cycles, they only show up sometimes, more often for some than others. We suspect that the file-change detector gets a little confused. We have to get around them by using --no-local-cache, which is becoming less feasible as our repo grows. Any ideas?

Hello friends, @average-breakfast-91545 and I are doing some more macos + linux + docker + AWS lambda + pants shenanigans on the quest to get Carbon Re fully ✨ pantsy ✨. We have got mostly everything working (bar Pytorch — expect more head-scratching questions in the coming weeks for your amusement here), but just writing here to get some feedback in case we are doing silly things (or rather, whether I am). Basic context: we have the majority of devs running macos machines (although some are using a couple of different linux distros), and we need to ship code onto AWS either in the form of Lambdas or ECS. This means building pex and shipping them in docker images (+ some mega exciting Lambda layer stuff that Bob is working on). So, in practice we currently have something along the lines of:

docker_image(
    name="aws_py39_pex_builder",
    instructions=[
        "FROM public.ecr.aws/lambda/python:3.9",
        "RUN yum -y update && yum -y install some list of build packages",
        ...  # Some unfun ENV shenanigans
    ],
)

docker_environment(
    name="aws_lambda_py39",
    image="aws_py39_pex_builder",
)

pex_binary(
    name="entrypoint",
    dependencies=["./entrypoint.py"],  # this brings all sorts of nasty python packages
    environment="aws_lambda_py39",
)

docker_image(
    name="lambda",
    dependencies=[":entrypoint"],
    instructions=[
        "FROM public.ecr.aws/lambda/python:3.9",
        "RUN yum -y update && yum -y install some runtime packages",
        ...  # more fun stuff
        "COPY path.to.package/entrypoint.pex /build/entrypoint.pex",
        "RUN unzip /build/entrypoint.pex -d \"${LAMBDA_TASK_ROOT}\"",
        "CMD [\"__pex__.path.to.package.entrypoint.handle\"]",
    ],
    ...  # some metadata for tags and stuff here
)

My current big question / worry is that I can’t seem to find a way of making

aws_lambda_py39

depend on

aws_py39_pex_builder

. Thus, I’m exploring a couple of options: 1. Run

pants package :aws_py39_pex_builder

followed by

pants package :lambda

, manually, wherever we need to. 2. Run a simple “check_and_build_maybe” function in

.pants.bootstrap

as a remediative step that carries out either

pants package :aws_py30_pex_builder

or

docker build …

(likely the latter). Am I missing a simpler way of solving this problem? Have I missed a glaringly obvious bit in the docs that would fix this problem for me? If not, and if I were to hypothetically look into adding a

dependencies

field to

docker_environment

, how much trouble would I likely get myself into?

I’m getting an error trying to build a pex that I don’t understand. I’ve created a minimal repo here: https://github.com/bryanwweber/pants-pytorch. I’m trying to build a pex with a dependency from the Pytorch index via

pants package <target>

. The final error is:

Failed to digest inputs: "Error storing Digest { hash: Fingerprint<4626ecca9d97876686ac89adf4252f9
21617741f8fec8dcb16a8410375d240d8>, size_bytes: 2265668508 }: Invalid argument"

The full error message is on the README in the repo. Any help is appreciated! Thanks!

Why does

./pants fmt

log a warning when changes are made? Isn't that point of running formatters?

Important announcement about Toolchain in thread:

Is it recommended to run

./pants fmt ::

or to break up formatters? e.g.

isort

+

yapf

seem to cause a bunch of logs like this

12:02:16.98 [INFO] Filesystem changed during run: retrying `Fmt` in 500ms...
12:02:17.50 [INFO] Filesystem changed during run: retrying `Fmt` in 500ms...
12:02:18.01 [INFO] Filesystem changed during run: retrying `Fmt` in 500ms...
12:02:18.52 [INFO] Filesystem changed during run: retrying `Fmt` in 500ms...
12:02:19.04 [INFO] Filesystem changed during run: retrying `Fmt` in 500ms...

Hi all, when working with Python, why can't pants solve some of the newer versions of the requirements? E.g. Given the simplified requirements file

numpy==1.24.3 ; python_version >= "3.11" and python_version < "4.0"

pants

generate-lockfiles

fails with

ERROR: Could not find a version that satisfies the requirement numpy==1.24.3

but if I lower the numpy version to e.g.

1.15.0

it works ok

What's the best way to use pants w/ pytest-csv? I want to create a csv file of all test results, but it appears that executing under pants the parallelism by file results in only the last completed test having results (instead of appending). Alternative tools or methods welcome.

Has anyone had luck with running

isort

and

yapf

as part of

fmt

together?

isort

wants imports to look one way and

yapf

wants another and I can't seem to figure out the right config settings to make things happy.

Is there a timeline for stable Pants 2.16 and (more important for me) 2.17 releases? I'm eagerly awaiting some features but would rather not run unstable versions in production.

Has anybody been successful in getting Colima to work nicely with the

docker_environment

target? I’d asked in a previous thread but only found other unsuccessful attempts by users.

Hey guys! First time setting up pants on a work-repo, getting this:

I've set the source root in my pants.toml:

And

pants roots

finds it just fine:

Ah, seems my top-level

BUILD

file created by

pants tailor ::

included:

python_sources(  
    name="root",
)

But no python files were in the top level dir.

Hi folks, I’m looking for some recommendations on how to troubleshoot pants error like

ERROR: Could not find a version that satisfies the requirement PackageName==x.y.z

, where you can successfully run

pip install PackageName==x.y.z

. There is usually a lack of error message output. We get error message like

Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

, but because package is installed in a sealed environment, it’s unclear to me how to find the actual error logs (or is it already cleaned up by the time pants failed?)

Hey! We're in the process of upgrading our Python version, so we have tests for libs with a parametrized python interpreter constraint. We also tried to move away from a constraints file to a lockfile (resolve). And we're hitting a bug that's been reported (this). Are there any workarounds for this?

I've been debugging an issue for 2 days that I think I can describe, but don't have a minimal working example I've extracted from internal code to share. We use GitLab which get's cranky if you try to cache files that live outside the "CI_PROJECT_DIR" (the directory the git repo is checked out to. So in CI we set

named_caches_dir = '.cache/pants/named_caches'

. This worked fine, but recently while trying to merge two repositories we ended up with a

.gitignore

setting of

.cache

(that is matching the

named_caches_dir

). Pants then... mostly seems to work but does whacky things like

AttributeError: module 'pendulum' has no attribute 'tz'

(where pendulum totally has said attribute) when running some goals. part of the short term answer can be "dont' do that", but: • A cache seems like a reasonable thing to want to gitignore. • I'm not sure what other bugs might be latent

Hi all, has anyone ever seen this error while trying to generate lockfiles:

stderr:
'<' not supported between instances of 'VCSArtifact' and 'FileArtifact'