In the repo, we have some tasks where antlr is used to generate Python lexers, from the antlr G4. the process is simply to run the req'd antr JAR with with the jvm. eg.

java -Xmx500M -cp "/usr/local/lib/antlr-4.9.3-complete.jar:.:/usr/local/lib/antlr-4.9.3-complete.jar" org.antlr.v4.Tool

At a simple level this is

experimental_shell_command

, am I correct in thinking i can get Pants to "supply" the JVM, and the antlr dependency. adding the jvm backend, gives me the

jvm_artifact(...)

to pull antlr, How do i make the "jvm" a tool / dependency to the

experimental_shell_command

or is there a "java run" type target instead ?

Did you find a way to use .env files you were happy with?

Hey all! awesome to see the questions here, and wanted to throw one out there; happy to provide a minimal example with some more time. Basically, we'd like to be able to bake the dependencies of our repos into a base docker image so that the CI/CD pipeline will be as fast as possible. So for example, for the purpose of CI/CD pipelines: 1. we have an image that has all dependencies for our project-v1.0 installed into it, but not project-v1.0 a. Specifically,

poetry install --no-root

or whatever the equivalent command is (forgive me, going off the top of my head) 2. During merge request pipelines, the pipeline adds the current state of the repo to the container 3. (today) we run

poetry install --sync

in the container a. This allows any new dependencies to be added dynamically b. old dependencies to be removed c. Installs the library under test d. Is faster than a "clean" build because poetry knows that several of the dependencies are already installed in the virtual environment 4. We then run linting/security scanning/ tests / etc We'd really like to be able to replicate the above with pants, while keeping the benefits of how pants normally operates (with the hermetic execution, dep inference, etc) I tried building a pex and working with that, but i'm running into a few issues (with and without the pex; it's all a bit messy atm while i've been trying to debug / explore): •

pants repl ::

(just used for my interactive testing) in the container fails with a message saying manulinux versions of the dependent libraries are not available ◦ This is technically true; they aren't (it's numpy and such + a few custom deps we don't currently build manylinux versions of) ◦ What i'd expect / want to happen is for pants to automatically build the required wheels from the sdist (which is available on the private mirror, and pants is also able to reach pypi) ◦ Oddly, I think this might be working if I do something like

./pants test ::

in that it does exactly what i'd expect (build the whls from sdist) but I don't know that it's actually baking/caching the whl files into the image as opposed to re-downloading them •

./pants export ::

does indeed work, and if I

source ./path/to/venv/bin/activate

and

python -i; import my_library; import my_library_dep_that_won't_work_with_repl

that all works as expected. Additionally, my question at a high level is how can I bake the required whl files into the image so that they are available and will be used and also support ./pants downloading and using new deps as needed in the pipeline? And how can I prove / verify that they are actually there / being used? I thought I might need to run

./pants (re?)generate-lockfiles

but after opening them up they actually aren't platform specific (which was a duh! moment once I realized it) For what it's worth, I was seeing the above (no manylinux versions available) messages both with and without the pex_build. I kind of suspect this is something related to how i'm doing things (probably wrongly, although i've been pouring over the (well-written!) docs)? So happy to provide a minimal example once I have some time 🙂 I think in current state I arguably could just ignore the issues with

./pants repl ::

in the container because I believe

./pants test ::

and

./pants package

etc are working? but I don't want to just ignore it since it seems like this is something that should work. Also, with current state, i'm not certain that it isn't re-downloading the whl files on each new command; is there info about how/where pants caches the whl files it uses? Can someone confirm if pants is creating a separate .pex for each command / sub-portion of each command? • This actually would make a lot of sense for the hermetic execution reasons, and appears to be what's happening given the output messages

👋 Hi everyone! I'm having issues running pytest with alembic and pants. It seems the

alembic.ini

file is not being included in the pex runtime. I have attempted to explicitly include the ini file using the file and resource pattern. Still not managing to get this right. Any advice?

Does Pants expose module mappings in BUILD files in any way? I would like to be able to get the original requirement name given the mapped module name

module_mapping={
    "ansicolors": ["colors"]
} 
###  get ^ when given ^

I have the following requirement:

python_requirement(
    name="snowflake-connector",
    requirements=["snowflake-connector-python[pandas]==2.7.11"],
    modules=["snowflake"],
)

until recently I was able to build my package with pants, but now i’m getting

No matching distribution found for snowflake-connector-python[pandas]==2.7.11

. i don’t know what’s changed

Hey I’m trying to set up Pants with an example project but I’m my project structure is wrong (probably). What I got going is under the root of the git repo there are three folders where each folder is a poetry project. Each poetry project has got it’s own name, dependencies, etc. I tried to follow the instructions in the docs but when I reached the end pants is not able to understand the depenencies. How can I make this structure work with pants?

I'm trying to setup pants on a pre-existing python repo and am unsure how to debug

generate-lockfiles

failing with

ERROR: Could not find a version that satisfies the requirement awswrangler==2.16.1
ERROR: No matching distribution found for awswrangler==2.16.1

Since that version clearly exists https://pypi.org/project/awswrangler/2.16.1/#history

When would we expect

experimental_shell_command

to rerun? I'm seeing it run essentially every daemon restart, which isn't ideal. 🧵

Hey Pants Team, been seeing OOM errors in CI recently with warnings such as

[2022-11-10T18:49:23Z] 18:49:23.75 [WARN] Error storing process execution result to local cache: Error storing fingerprints ["000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20", "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20", "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20", "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20", "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f20"]: MDB_CURSOR_FULL: Internal error - cursor stack limit reached - ignoring and continuing

the only major change I can think of is we pull the cache in our lint step. Not sure if that somehow impacted.

I’m trying to migrate an existing monorepo to use pants and I’m a bit lost when trying to tie first-party dependencies between the different packages within the repo - while also dealing with requirements.txt files that declare the dependencies. More details in 🧵

Hi, I am trying to create a wheel from a

python_distribution

target but the

package

goal is failing. Following is the trail of the debug output:

14:26:04.48 [DEBUG] Completed: Resolve transitive targets

14:26:14.43 [DEBUG] Completed: Downloading: <https://github.com/protocolbuffers/protobuf/releases/download/v3.20.1/protoc-3.20.1-osx-aarch_64.zip> (2.58 MB)

14:26:14.43 [DEBUG] Completed: pants.core.util_rules.external_tool.download_external_tool

14:26:14.43 [DEBUG] Completed: Generate Python from Protobuf

14:26:14.43 [DEBUG] Completed: pants.backend.python.util_rules.python_sources.prepare_python_sources

14:26:14.43 [DEBUG] Completed: pants.backend.python.util_rules.python_sources.strip_python_sources

14:26:14.43 [DEBUG] Completed: pants.backend.python.util_rules.python_sources.prepare_python_sources

14:26:14.43 [DEBUG] Canceled: pants.backend.python.util_rules.python_sources.strip_python_sources

`142614.43 [DEBUG] Completed:

package

goal`

14:26:14.43 [DEBUG] computed 1 nodes in 10.502206 seconds. there are 262 total nodes.

14:26:14.43 [ERROR] 1 Exception encountered:

Exception: Error downloading file: error sending request for url (<https://github.com/protocolbuffers/protobuf/releases/download/v3.20.1/protoc-3.20.1-osx-aarch_64.zip>): error trying to connect: proxy authentication required

I have tried adding the subsystem

subprocess-environment

with proxy variables in

pants.toml

but the error remains the same. I am using python 3.9 on an M1 Mac. My question is if the debug output says that package goal completed, why and what is it failing for?

The change to have mypy's cache work is great (makes our

./pants check ::

runs 10x faster when there's a small change to python code), thank you! However, I noticed that there seems to be a single

cache.db

file per python verison in

~/.cache/pants/named_caches/mypy_cache/

. Does that mean a single cache is used for all invocations of mypy via pants? E.g. I've got multiple check-outs of our internal company repo (sharing a cache seems to make sense between them...), but also the pants repo, and sharing a cache between the pants and our internal one seems a bit unexpected?

Hey folks. I have a monorepo with ~40 pex executables being built by pants 2.10 on github actions in separate, concurrent workflows. We’ve been seeing intermittent but consistent “Could not find a version that satisfies the requirement” errors when PEX tries to locate and download python packages. These seem to occur at random, affecting a variety of common python packages. We’re using artifactory to mirror PyPi, and I’ve ensured that I’m locally caching the following paths. I’m also ensuring that the cache is populated with common libraries before before the 40 builds start running in parallel.

~/.cache/pants/lmdb_store
~/.cache/pants/named_caches
~/.cache/pip

yet, I still see these errors.

ProcessExecutionFailure: Process 'Building src.stages.fragment_counts/exe.pex with 9 requirements: boto3==1.21.45, confluent-kafka[avro]==1.8.2, cryptography==3.4.8, overrides==6.1.0, redis[hiredis]==4.3.4, toml==0.10.2, typeguard==2.13.3, types-redis==4.3.20, types-toml==0.10.1' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement boto3==1.21.45
ERROR: No matching distribution found for boto3==1.21.45
pid 1910 -> /home/runner/.cache/pants/named_caches/pex_root/venvs/97c49a45855c8337ed6c7437a8f9bce7e5f12e07/15ba0a31f835e32f40e40a07c519d52e41260cdc/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /home/runner/.cache/pants/named_caches/pex_root --log /tmp/process-execution74ifxC/.tmp/tmpa2vp83vp/pip.log download --dest /tmp/process-execution74ifxC/.tmp/tmpddxwdbv9/opt.hostedtoolcache.Python.3.9.15.x64.bin.python3.9 boto3==1.21.45 confluent-kafka[avro]==1.8.2 cryptography==3.4.8 overrides==6.1.0 redis[hiredis]==4.3.4 toml==0.10.2 typeguard==2.13.3 types-redis==4.3.20 types-toml==0.10.1 --index-url ***((our-artifactory-redacted))/artifactory/api/pypi/sw-pypi/simple --retries 5 --timeout 15 exited with 1 and STDERR:
None

Any advice on how I could go about fixing this?

How am I supposed to add the

py.typed

?

How do I remove the

python2

compatibility on the wheel? 🤔

What's the state of art on the Build Server Protocol thingy for pants?

Why pants can't infer it? 😢

22:47:26.30 [WARN] Pants cannot infer owners for the following imports in the target tests/test_http.py:run:

  * tests.response.Response (line: 11)

If you do not expect an import to be inferrable, add `# pants: no-infer-dep` to the import line. Otherwise, see <https://www.pantsbuild.org/v2.14/docs/troubleshooting#import-errors-and-missing-dependencies> for common problems.

Is there some way I can get pants with

[python-infer].assets = true

to detect a dependency on assets that are loaded from a folder like so:

root_dir = "path/to/directory/with/data/files/"
for filename in os.listdir(root_dir):
  with open(root_dir + filename) as f:
    ...

Or would I have to open each file explicitly?

Is this something known?

❯ ./pants test tests
23:31:26.61 [INFO] Completed: Building pytest_runner.pex
23:31:26.61 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building pytest_runner.pex' failed with exit code 1.
stdout:

stderr:
Failed to resolve requirements from PEX environment @ /home/marcelo/.cache/pants/named_caches/pex_root/unzipped_pexes/c7d8fc40ba3f3ee9dda9538e11bcfa4f1046db26.
Needed cp37-cp37m-manylinux_2_31_x86_64 compatible dependencies for:
 1: httptools>=0.5.0
    Required by:
      uvicorn-trailers 0.0.0
    But this pex had no ProjectName(raw='httptools', normalized='httptools') distributions.
 2: uvicorn>=0.19.0
    Required by:
      uvicorn-trailers 0.0.0
    But this pex had no ProjectName(raw='uvicorn', normalized='uvicorn') distributions.

🤔

What is the story around updating dependencies? Like lets say I’m using a requirements.txt for the whole monorepo and I want to update all the dependencies (and not because I am dependent on new features, I just want bugfixes, security patches, etc.). Do I just re-lock and that updates? Is this something that I have to deal with out of band with pip, poetry, or pipenv?

Are there known issues with the pytest capture logs (caplog) fixture on pants?

Besides the fact that I'm duplicating a lot of information on

pyproject.toml

and

python_distribution

, any feedback on this https://github.com/Kludex/uvicorn-extensions?

What's the state of art on the Build Server Protocol thingy for pants? Is there update on this?

It's a bit troublesome that if I forget to add the requirements on the

pyproject.toml

, the tests works anyway ... 🤔 Can't pants make sure that I've included them in the

METADATA

or something like that?

Is there a way to run the tests with multiple versions of a dependencies? 🤔

I'm using Pants for a Python project and have written some stubs for MyPy that I put into a

stubs

directory. I have a line like

mypy_path = $MYPY_CONFIG_FILE_DIR/stubs

in my config file and when I run MyPy by itself, then it uses them. However, when I run

pants check

then it does not. Pants is definitely using the same MyPy config file, though. Does anyone happen to have any ideas on how to track down the cause?

I verified that Pants is using the same MyPy config by inserting same garbage into my mypy.ini and watching it crash.

I want to use

pyproject.toml

to set the configuration for

flake8

. I'm unable. Even changing the

config = "pyproject.toml

, and adding

extra_requirements.add = ["Flake8-pyproject"]

doesn't let me do that. 😢

Hm... In Linux arm64/aarch64 setups, using

pants_requirements

from the

pants.backend.plugin_development

backend causes failure on

./pants-local export ::

(where pants itself is built from source) because there is no binary wheel for Linux aarch64. Is it possible to make it to look at the source directory that it is built from?

18:53:32.67 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building 2 requirements for requirements.pex from the tools/pants-plugins.lock resolve: pantsbuild.pants.testutil<2.15,>=2.14.0a0, pantsbuild.pants<2.15,>=2.14.0a0' failed with exit code 1.
stdout:

stderr:
Failed to resolve compatible artifacts from lock tools/pants-plugins.lock for 1 target:
1. /home/joongi/.pyenv/versions/3.9.10/bin/python3.9:
    Failed to resolve all requirements for cp39-cp39-manylinux_2_31_aarch64 interpreter at /home/joongi/.pyenv/versions/3.9.10/bin/python3.9 from tools/pants-plugins.lock:

Configured with:
    build: True
    use_wheel: True

Dependency on pantsbuild-pants not satisfied, 1 incompatible candidate found:
1.) pantsbuild-pants 2.14 does not have any compatible artifacts:
    <https://files.pythonhosted.org/packages/71/bc/6662a9cc8cff3e2d4a116d33859c98c3295f45e93189096ca84b6f554a45/pantsbuild.pants-2.14.0-cp39-cp39-manylinux2014_x86_64.whl>
    <https://files.pythonhosted.org/packages/4e/fe/f46438d974b4d19b32728f3823f63ae05397ee7da064d9c2284272f351bc/pantsbuild.pants-2.14.0-cp38-cp38-manylinux2014_x86_64.whl>
    <https://files.pythonhosted.org/packages/65/4b/959230b5a72b177c1ef2c9dc6bdb7214fb429b39d694c7c85363bde4814f/pantsbuild.pants-2.14.0-cp37-cp37m-manylinux2014_x86_64.whl>
    <https://files.pythonhosted.org/packages/68/48/b8df6c443d06187663b27d4e88c8ef8b1ba2916e225c78ab1b4b032c7072/pantsbuild.pants-2.14.0-cp39-cp39-macosx_10_16_x86_64.whl>
    <https://files.pythonhosted.org/packages/68/71/50d1567ce077b3dc8acf0d9a11e8878ece1d031f3bd432412da558381cd7/pantsbuild.pants-2.14.0-cp38-cp38-macosx_11_0_x86_64.whl>
    <https://files.pythonhosted.org/packages/80/f9/f485c7a71d2adc937620c0d61fc4818da5ac92b968f6ea5dbffe280f29b7/pantsbuild.pants-2.14.0-cp39-cp39-macosx_11_0_arm64.whl>
    <https://files.pythonhosted.org/packages/92/6c/bd8b17ff1df4931ea0358c2475307f22259461df8f96a92db85ab8809c9c/pantsbuild.pants-2.14.0-cp37-cp37m-macosx_10_16_x86_64.whl>
    <https://files.pythonhosted.org/packages/9b/e9/c00b331432f789bfb8c10b5489f19245edaf8ba9c5140cc5584530fded5c/pantsbuild.pants-2.14.0-cp39-cp39-macosx_10_15_x86_64.whl>
    <https://files.pythonhosted.org/packages/b6/f0/c3bb663441bf2dc30846690396a858c021a1fb02a4fa4c85af0f66cc041e/pantsbuild.pants-2.14.0-cp37-cp37m-macosx_10_15_x86_64.whl>
    <https://files.pythonhosted.org/packages/db/2d/3319dc26b7d372bbc3ddd096222613b36ba7fc428c4ba14ee012c0e700b9/pantsbuild.pants-2.14.0-cp38-cp38-macosx_10_15_x86_64.whl>
    requirers:
    via: pantsbuild.pants<2.15,>=2.14.0a0
    via: pantsbuild.pants.testutil<2.15,>=2.14.0a0 -> pantsbuild.pants==2.14.0