I suspect they are sub-batches of the

CPython==3.9.13

partition

and the ever-growing `_`s are an easy way to prevent collisions between the outputs of the batches

@rhythmic-glass-66959 I think Pants doesn't generate reports by default, so presumably you're setting cli args for bandit and pylint (and flake8 for that matter) to make the emit reports? Can you share your config?

Hello, good people! I'm trying to mark a dependency on a package that is merely installed but doesn't need to be imported directly. Specifically, I refer to snowflake-sqlalchemy. I'm trying to figure out what the

3rdparty/python

part of the documentation is referring to. In my case, the lockfile and the reqs file are in the root of the repo, what does the path need to be in that case?

I have a 3rd party dependency issue that I can’t quite work out how to resolve from the docs. I’m trying to add

authzed

to a project and can’t quite get pants to properly resolve its dependencies: When doing a

./pants run src/python/my_project/script.py

which imports from

authzed

, I get an error with the protobuf module:

from google.protobuf.internal import builder as _builder
ModuleNotFoundError: No module named 'google.protobuf'

Now, I can see in my lockfile that

authzed

properly declares this dep, so I’m not sure why pants isn’t including it. The module name/import mapping are in the defaults here, and my lockfile contains the following:

"project_name": "authzed",
          "requires_dists": [
            "async_generator<2.0,>=1.10",
            "google-api-core<3.0.0,>=2.4.0",
            "google_api<0.2.0,>=0.1.12",
            "grpcio==1.50.0",
            "mock<5.0.0,>=4.0.3",
            "protobuf==3.20.2",
            "protoc-gen-validate<0.5.0,>=0.4.1",
            "typing-extensions<5,>=3.7.4"
          ],
          "requires_python": "<4.0,>=3.7",
          "version": "0.7"

This is purely a transitive dependency, do I have to explicitly specify it via a

python_requirement

target or something?

Just had a really cool use case for the advanced target selection and wanted to share in case it's helpful for others. We wanted to know if changes someone introduced into our monorepo affected code our deployed services depended on so we could coordinate with them about merging it into official releases. We already had our containerized services tagged, so the query ended up looking like

./pants --changed-since=master --changed-dependees=transitive --filter-tag-regex=<our tag here> list

As has been the case for a while, awesome features from the Pants team continue to make our lives easier!

Anyway, thanks for the help! Much appreciated

Does anyone have a recommendation for in-repo scaffolding? I want users to be able to scaffold out a new project in the monorepo easily. In the node world, I use hygen for this. I took a look at the docs and I don't see anything first-class built into pants. I saw some other folks in this Slack talking about cookiecutter, which seems promising.

I don’t know if it’s the way coverage.py works or it’s the pants integration but the coverage doesn’t include those modules which aren’t imported by the tests. I thought it would look at all the source files and then compare it with the tests inside the test files.

Hi, I saw this today:

./pants test ::                                                       
Another pants invocation is running. Will wait up to 60.0 seconds for it to finish before giving up.
If you don't want to wait for the first run to finish, please press Ctrl-C and run this command with PANTS_CONCURRENT=True in the environment.

I was running

./pants repl ::

in another terminal. Anyways, I was able to get it to run by

PANTS_CONCURRENT=True ./pants test ::

. Was wondering why it isn’t default? Are there any issues which I should be aware of? Typically, have a repl in another shell is a common strategy

Hi all. We're trying to setup Pants in our companies extremely convoluted Python repo. So the repo currently consists of different micro-services each with its own PyProject.toml file (we are using Poetry) and then there is a main PyProject.toml file outside of the services folder which we use in our CI pipeline (running tests etc). For testing, instead of each service containing its own tests, all tests are located in a tests folder and are supposed to use the main PyProject.toml file. The issue is that when I try running ./pants test <path-to-test-file> it is unable to correctly identify a single root and gives an error like:

The target lib/modules/serviceA/controller.py imports `sqlalchemy.text`, but Pants cannot safely infer a dependency because more than one target owns this module, so it is ambiguous which to use: ['services/serviceA:poetry#SQLAlchemy', 'services/serviceB:poetry#SQLAlchemy', 'services/serviceC:poetry#SQLAlchemy'].

This is actually just a warning but ultimately it leads to an error like:

ModuleNotFoundError: No module named 'flask_sqlalchemy'

I think somehow I need to specify that all tests need to use only the main PyProject.toml file but I'm not sure how I do that. Currently I'm setting my source as:

[source]
marker_filenames = ["pyproject.toml"]

Please advise on how I can go about solving this problem. Thanks in advance.

Have a look here: https://www.pantsbuild.org/docs/python-third-party-dependencies#multiple-lockfiles The keyword you're missing in your search is "resolve" and Pants supports multiple resolves. It sounds like you have 1 resolve per project plus 1 extra one for tests. You need to explicitly configure Pants to know about each resolve I think. I don't know much more about resolves than that, but folks who do should be along on Monday with more details if you're still stuck then.

Hi all. I am facing some issues testing out pyspark with pants. Running a pyspark job via

./pants run

gives

ModuleNotFoundError: No module named 'pandas'

. here is a repo with the issue: https://github.com/adityav/pants-python-tryouts

Thanks a lot. I think this is exactly what I need.

Hi, I'm trying to generate a new lock, and it just takes an infinite amount of time (it used to be 25-35 minutes).

> pants generate-lockfiles --resolve=python-default
⠂ 4795.88s Generate lockfile for python-default

Is there anything I can do to debug it? There is probably a bad constraint that wrack havoc, but I don't know which and there are 370 dependencies in the repo...

the documentation talks about allowing 3rd party libraries on disk in

python_requirements

but only ever mentions files external to the repository. Is there a way to include .zip/.whl files in the repository?

As an alternative to multiple resolves, we handled a similar migration from many-poetries to pants by having one resolve, and only importing the 'main' pyproject.toml that had all the required dependencies (via path dependencies to each subproject, and/or copying their requirements) and have progressively moved to not use/remove the project pyproject.tomls at all (as we switched all required functionality into pants). We don't have pants load dependencies from each of the subproject's

pyproject.toml

: there is only one `poetry_requirements`/`python_requirements` target in our whole repo. This does force pants to be using everything the same versions of all dependencies across all projects, though.

My coworker had some thoughts on using pants at his previous job

Pants had scalability issues, and both pants and buck are not as easily extensible as Bazel with skylark

Just so I can provide accurate information on whether we should move forward with pants to use bazel, it seems like there are two versions of pants, and he is possibly referring to pants v1? Also, my understanding is pants v2 is known to be used on repos with thousands of python files but pants v1 was proven to work on monorepos a magnitude larger? And pants v2 is more extensible than bazel's skylark, from my understanding?

Hi! Within our project, we used to just copy pex files into Docker images. Now I tried to implement the "Simple Multi-Stage Build" approach from https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/. The thing is that it works for some of my images and not for others 🙂 I got quite confusing messages from Pants - first, I thought that there was some problem with dependency inference (see screen), but it turned out that it might be related to a problem with 3rd party dependency (fonttools?) - I tried to run

RUN PEX_TOOLS=1 /usr/local/bin/python3.10 /binary.pex venv --scope=deps --compile /bin/app

outside of Pants/Docker and got this (selected the interesting part):

$ PEX_VERBOSE=9 PEX_TOOLS=1 python dist/src.python.arcor2_calibration.scripts/calibration.pex venv --scope=deps --compile appTest
pex: Laying out Spread PEX directory /home/zdenal/arcor2_repos/arcor2/dist/src.python.arcor2_calibration.scripts/calibration.pex: 0.1ms
pex: Executing installed PEX for /home/zdenal/arcor2_repos/arcor2/dist/src.python.arcor2_calibration.scripts/calibration.pex at /home/zdenal/.pex/unzipped_pexes/b1de9cb1dc137d74acdc0779598b9da266d9abcc
...
pex: Re-writing /home/zdenal/arcor2_repos/arcor2/appTest/bin/ttx
pex: Executing: /usr/bin/python3.10 -s -E -m compileall appTest
Traceback (most recent call last):
  File "/home/zdenal/.pex/unzipped_pexes/b1de9cb1dc137d74acdc0779598b9da266d9abcc/.bootstrap/pex/result.py", line 105, in catch
    return func(*args, **kwargs)
  File "/home/zdenal/.pex/unzipped_pexes/b1de9cb1dc137d74acdc0779598b9da266d9abcc/.bootstrap/pex/tools/commands/venv.py", line 209, in run
    pex.interpreter.execute(["-m", "compileall", venv_dir])
  File "/home/zdenal/.pex/unzipped_pexes/b1de9cb1dc137d74acdc0779598b9da266d9abcc/.bootstrap/pex/interpreter.py", line 1239, in execute
    stdout, stderr = Executor.execute(cmd, stdin_payload=stdin_payload, env=env, **kwargs)
  File "/home/zdenal/.pex/unzipped_pexes/b1de9cb1dc137d74acdc0779598b9da266d9abcc/.bootstrap/pex/executor.py", line 99, in execute
    raise cls.NonZeroExit(cmd, process.returncode, stdout, stderr)
pex.executor.Executor.NonZeroExit: received exit code 1 during execution of `['/usr/bin/python3.10', '-s', '-E', '-m', 'compileall', 'appTest']` while trying to execute `['/usr/bin/python3.10', '-s', '-E', '-m', 'compileall', 'appTest']`
received exit code 1 during execution of `['/usr/bin/python3.10', '-s', '-E', '-m', 'compileall', 'appTest']` while trying to execute `['/usr/bin/python3.10', '-s', '-E', '-m', 'compileall', 'appTest']`
...

Any idea what might be wrong? Many thanks in advance!

Hi I am building a docker image from my project and hosting it on GitHub packages. Then I am using microk8s to run my Kubernetes cluster locally using this image. But I am getting the following error on starting pods Can anyone please suggest how to resolve this?

exec /bin/main.pex: exec format error

Hi. Thanks to @enough-analyst-54434, I was able to resolve my pyspark issue. However, adding a constraints file caused another. Using a constraints file causes

ModuleNotFoundError: No module named 'pyarrow'

when running tests. Repo reproducing the issue: https://github.com/adityav/pants-python-tryouts

Hi all. I'm setting up pants on an existing Python project but am facing an issue when running tests. The issue is the following code:

import json

def get_module_json(module, file_name):

"""

Helped method to load a file

"""

file_name = "tests/" + module + "/jsons/" + file_name

with open(file_name) as file:

return json.load(file)

We're using this to load json payloads from various locations but since we're loading them like this pants is unable to detect these files and hence does not include them in the sandbox. So I need to know how I can include all files in my tests folder in pants sandbox (at least all json files). Thanks in advance.

i suffered a little bit today because the pytest subsystem doesn’t support the filterwarnings ini option. (this was only partially because of pants-- problem was that filerwarnings is more powerful than and different from

-W

, but i had to use

-W

via pytest

args

in pants). is there any reason not to support all the built-in ini options in pants? what about supporting just

filterwarnings

for now?

Running into an interesting lockfile generation issue with

torch

package. 🧵

Is there any proper step by step guide to make pex file platform independent? Mainly I want to run same pex file in macOS and linux?

How can I pass

PYTHONOPTIMIZE

to a pex file? Some of our code depends on

__debug__

- when I run

PYTHONOPTIMIZE=1 python

, then

__debug__

becomes false. But it seems that when I run

PYTHONOPTIMIZE=1 file.pex

,

__debug__

remains true. The same result with

python -O file.pex

🙂 Not sure how this is supposed to work... But I would like to turn on optimizations for production images based on pex files. Thanks for hints.

My project uses a docker-compose.yml file and I’m working on wrapping docker with pants. I don’t see docker-compose referenced anywhere in the pants docs though - does pants support this feature? Or do I need to convert the docker-compose.yml to a BUILD file?

Sorry in advance for raising a potentially thorny topic, but we are at a crossroads in continuing our monorepo journey as a substantial portion of apps we want to build will have a React front end. Is there any guidance from the Pants developers on interim tooling for node/js while awaiting the feature in Pants? After a brief survey of the tooling targeting js, it looks like NX with the integrated approach is semantically most similar to Pants: A project.json to define targets/dependencies of a package (analogous to BUILD files), a “factored out” root package.json of third party requirements (analogous to a global resolve), and task executors running over a resolved targets DAG. Bazel with rules_js seems even closer, but not sure it’s worth the learning investment, not to mention it’s not a core feature of Bazel itself.

Hello folks, I am reaching out to check if any of you had issues running pants targets behind proxies or in offline environments. We currenlty use pants in one of our main services to package multiple libraries, however in some situations we have to develop in environemnts behind proxies. pants package target seems to run for along period of time then timing out. Here’s what we tried with no success: 1. Set the env variables for HTTP_PROXY and HTTPS_PROXY with no success (fyi, pip works well after setting those up) 2. Following this recipe from pex official documentation: https://pex.readthedocs.io/en/v2.1.121/recipes.html#pex-and-proxy-settings seems that pex no longer package requests as extra dependency

$ pip install pex[requests]
Collecting pex[requests]
  Downloading pex-2.1.121-py2.py3-none-any.whl (2.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.9/2.9 MB 20.2 MB/s eta 0:00:00
WARNING: pex 2.1.121 does not provide the extra 'requests'
Installing collected packages: pex
Successfully installed pex-2.1.121

I would appreciate any help or guidance to run pants behind proxies and any recipes for running in offline mode as well.

https://www.pantsbuild.org/v2.15/docs/using-pants-in-ci this works fine on desktop/local dev, not sure what to do about CI, specifically, how to make this work in CI. Specifically, caching the pants script and a particular version of pants. https://www.pantsbuild.org/v2.15/docs/using-pants-in-ci Based on my testing, when using the launcher, pants is no longer cached under

~/.cache/pants/setup