hoping to get some insight here: running a script with 21 dependencies and pants is 20+ minutes and counting to build the pex file. i’ve deleted pants and pip cache, and also asked someone else to try on their machine, and they seem to be facing the same problem.

⠓ 1462.31s Building main.pex with 21 requirements: arn<0.2.0,>=0.1.5, backoff<2.0.0,>=1.10.0, boltons<21.0.0,>=20.2.1, boto3<2.0.0,>=1.15, botocore<2.0.0,>=1.21.60, click<9.0.0,>=8.1.3, fsspec<2022.0.0,>=2021.4.0, orjson<3.7.0,>=3.6, pandas<2.0.0,>=1.5.3, protobuf<4,>=3.19

here’s the list of 3rd party dependencies

./pants dependencies --transitive src/my_script:main
//:poetry#arn
//:poetry#backoff
//:poetry#boltons
//:poetry#boto3
//:poetry#botocore
//:poetry#click
//:poetry#fsspec
//:poetry#orjson
//:poetry#pandas
//:poetry#protobuf
//:poetry#psycopg2-binary
//:poetry#pulumi
//:poetry#pulumi-aws
//:poetry#pydantic
//:poetry#ratelimit
//:poetry#redis-py-cluster
//:poetry#requests
//:poetry#s3fs
//:poetry#sentry-sdk
//:poetry#setuptools
//:poetry#tomli
//pyproject.toml:poetry
...

is there something i can/should do differently for pants to run this in a more reasonable timeframe?

🎉 slack pro 🎉 Thanks all the time!

Hey Pants Team, running into the following errors running mypy for a multi project repository We have the following pants.toml configuration

[mypy]
skip = true
config = "tools/python/mypy.ini"
extra_requirements.add = ['SQLAlchemy[mypy]==2.0.0']
lockfile = "tools/python/mypy.lock"

and the following mypy.ini

[mypy]
plugins = sqlalchemy.ext.mypy.plugin
no_implicit_optional = True
local_partial_types = True
follow_imports = normal
disallow_untyped_defs = True
disable_error_code = misc
exclude = alembic

We added a new project that does not use sqlalchemy and now running mypy with pants we see the following errors:

/Users/njgrisafi/.cache/pants/named_caches/pex_root/venvs/5bcbf65366e68d9647087f9935ca1efcc8fff5d1/b3aef238742d4e1a2d2b93cbfe8e2aa5b6944556/lib/python3.9/site-packages/mypy/typeshed/stdlib/fnmatch.pyi: error: Cannot find implementation or library stub for module named "sqlalchemy.orm"
/Users/njgrisafi/.cache/pants/named_caches/pex_root/venvs/5bcbf65366e68d9647087f9935ca1efcc8fff5d1/b3aef238742d4e1a2d2b93cbfe8e2aa5b6944556/lib/python3.9/site-packages/mypy/typeshed/stdlib/fnmatch.pyi: error: Cannot find implementation or library stub for module named "sqlalchemy.orm.attributes"
/Users/njgrisafi/.cache/pants/named_caches/pex_root/venvs/5bcbf65366e68d9647087f9935ca1efcc8fff5d1/b3aef238742d4e1a2d2b93cbfe8e2aa5b6944556/lib/python3.9/site-packages/mypy/typeshed/stdlib/fnmatch.pyi: error: Cannot find implementation or library stub for module named "sqlalchemy.orm.decl_api"
tools/python/service-catalog/tests/test_service_catalog.py: error: Cannot find implementation or library stub for module named "sqlalchemy.orm"
tools/python/service-catalog/tests/test_service_catalog.py: error: Cannot find implementation or library stub for module named "sqlalchemy.orm.attributes"
tools/python/service-catalog/tests/test_service_catalog.py: error: Cannot find implementation or library stub for module named "sqlalchemy.orm.decl_api"
tools/python/service-catalog/service_catalog/cli/__init__.py: error: Cannot find implementation or library stub for module named "sqlalchemy.orm"
tools/python/service-catalog/service_catalog/cli/__init__.py: error: Cannot find implementation or library stub for module named "sqlalchemy.orm.attributes"
tools/python/service-catalog/service_catalog/cli/__init__.py: error: Cannot find implementation or library stub for module named "sqlalchemy.orm.decl_api"

Hi community! When working with pants to build standalone pex files. Is there an opinionated way to change the dist path in the relevant BUILD filed? It seems the output path option only changes the name of the object being generated not the dist path itself. My use case is keeping modules in version control via lfs, where each pex file represents a snapshot of a given release. To this end, does pants keep track of the build/package state of pex files? i.e. when triggering

./pants package example

is this operation repeated regardless of changes?

Hello, community! I happen to do some work on a Pants-built python app from a colleague that recently left the company and I have an issue, which I prefer to address here instead of stackoverflow. So, the issue is that we need to run this pants app on RHEL 9, 8 and 7 and we build it on CentOS 7 (which works on rhel7 / 8 as expected), but on RHEL 9, I get a glibc error : "return code=1, stdout=, stderr=awk: /tmp/_MEIu4qVxf/libm.so.6: version `GLIBC_2.29' not found (required by awk)". I've also tried to build it on rhel 8, but throws the same error as above. Also attaching an output of glibc backwards compat. Building the pants bin directly on RHEL 9 works correctly. Do you have any idea how to make it run from rhel 7 to all the upcoming distros? Thanks alot!

Hi guys. Sorry to bother, but I'm trying to migrate some stuff in my company to a monorepo. Is there any open source repository that encompasses most of the functionalities of pants that I could look into? Thanks a lot and keep up the good work 😊

This is the one I know about https://github.com/lablup/backend.ai

Hi everybody! 🙂 Thank you for creating pants, it looks very promising! I am trying to get the experimental terraform backend up and running, but it keeps complaining that init has not been run to download the required providers. Running init manually did not fix it for me, and I did not find anything helpful in docs or issues except for an old issue that claims to have solved that (or something similar). How should I proceed? Thanks a lot and cheers for all your efforts! 🙂

It seems the output path option only changes the name of the object being generated not the dist path itself.

I assume you're using https://www.pantsbuild.org/docs/reference-pex_binary#codeoutput_pathcode - which is the right idea. If you configure:

[GLOBAL]
pants_distdir = "%(buildroot)s"

Then you should find:

pex_binary(
    ...
    output_path="foo/bar/baz"
)

Nets you a PEX file

baz

at

foo/bar/baz

in your repo after running

./pamts package ...

on the `pex_binary`target.

i.e. when triggering

./pants package example

is this operation repeated regardless of changes?

Yes. In fact all Pants operations that produce visible output files act this way. They basically can't assume the file already on the filesystem is up to date. IOW Pants does not 1st look at the output path, see if there is a file, checksum it and proceed from there. It does do something like that for all internal - invisible to you in normal circumstances - sandboxed operations, just not the final operations that materialize output to disk in your workspace.

Makes sense. Thanks for clarifying John. I guess in the end git should pick up changes and regardless of the rebuild.

@rhythmic-morning-87313 has given a talk about how Lablup use Pants in that repo, by the way. Here it is in English at Pycon Japan and in Korean at Pycon Korea

The English link seems to be deleted. At least that’s what I see when I click on the English one.

Actually it says “This video is private”

Note that you can use

./pants --changed-since=main --changed-dependents=transitive

to rebuild all packages that are affected by changes in git.

Otherwise,

/pants package example

will repeat just the final step of writing to your workspace, but the work leading up to that will be retrieved from cache, if possible.

There's https://github.com/pantsbuild/example-python for a very simple example, and https://github.com/StackStorm/st2 for a much more intricate one

https://github.com/lablup/backend.ai

That repo makes mine feel like a cheap knockoff 😳

@happy-kitchen-89482 https://github.com/pantsbuild/example-python advocates for

pants

executable in repo. Whats the plan to move to global install ?

The

./pants

script does install Pants globally per user (see

~/.cache/pants/setup

). That said we've started moving towards a

pants

binary: https://www.pantsbuild.org/v2.15/docs/installation

Hi everyone! I'm trying to figure out how to exclude a transitive dependency. I'm depending on a python library called talon and it then depends on a library called scipy. First thing is when i call dependencies with --dependencies-transitive flag, it only shows one layer of transitive dependencies, which is talon. It doesn't the entire dependency tree.

./pants dependencies sailebot/referral:referral --dependencies-transitive

So i'm able to exclude talon fine, but i want to exclude a dependency that talon depends on. This works, to exclude talon

python_awslambda(
    name="referral",
    runtime="python3.7",
    handler="referral_lambda.py:lambda_handler",
    dependencies=["//:reqs#psycopg2-binary","!!//:reqs#talon"],
)

But this doesn't work

python_awslambda(
    name="referral",
    runtime="python3.7",
    handler="referral_lambda.py:lambda_handler",
    dependencies=["//:reqs#psycopg2-binary","!!//:reqs#scipy"],
)

I'm running an M1 laptop and am having trouble installing*

grpcio

when using pants. I'm trying to run some tests but I get a very long error

ERROR: Failed building wheel for grpcio
ERROR: Failed to build one or more wheels

One of the line I see before getting a whole bunch of irrelevant

c

noise is

Building requirements.pex with 4 requirements: google-cloud-pubsub~=2.14.1, google-cloud-secret-manager~=2.12.6, pydantic~=1.10.2, pytest~=7.1.3

I've set

[python]
interpreter_constraints = ["CPython==3.10"]

I'm confused because if I create a

venv

via

python3.10 -m venv venv

I can install all of the above dependencies. I've tried adding

--keep-sandboxes=on_failure

but I don't know how to inspect the sandboxes (they just seem to be bash files?)

👋 Hi everyone! Before I start asking specific questions, are there some good example repos that use pants for a python monorepo? I know about the one in the docs (https://github.com/pantsbuild/example-python), but that is a bit to simple for me to really wrap my head around the BUILD files, test locations and such.

I'm sorry if my question seems foolish. I have been using namespace packages (https://packaging.python.org/en/latest/guides/packaging-namespace-packages/) in my Python repository to ensure a consistent namespace while allowing me to choose from several smaller packages when building Docker images, thus avoiding the inclusion of a large and unnecessary set of libraries. I use Poetry to manage the dependencies among my libraries and third-party packages. The folder structure is organized as follows:

.
├── libs
│   ├── lib1
│   │   ├── company_name
│   │   │   └── lib1
│   │   │       ├── __init__.py
│   │   │       └── something.py
│   │   │   
│   │   └── tests
│   │       └── test1.py
│   └── lib2
│       ├── company_name
│       │   └── lib2
│       │       ├── __init__.py
│       │       └── something.py
│       └── tests
│           └── test2.py
│           
└── executable_apps
    ├── app1
    │   └── app1.py
    └── app2
        └── app2.py

If I was to adopt pants, would there be any need for using namespace packages? Would Pants be able to handle only pulling in the necessary libraries or files?

Any suggestions on how to generate a

Pipfile.lock

for a given target? The pex lockfile seems similar, but not exactly the same.

Hm, PyCon JP channel seems to have reorganized the 2022 talks by splitting per-track videos into per-talk videos: https://www.youtube.com/playlist?list=PLMkWB0UjwFGm6_NIPfJiBvYqgjqb6pt4W, but there seems to be missing some talks including mine 😞

You could refer my slides directly: https://speakerdeck.com/achimnol/pycon-jp-modernizing-development-workflows-for-a-7-year-old-74k-loc-python-project-using-pantsbuild

Hi there! does anyone have an open repo that I can take a look using pants + poetry ?

Hi all, me again seeking some guidance. I currently have a repository with multiple libraries that each require versioned individually. I use poetry for definition/configuration of those libraries (using a pyproject.toml file per project) My current thinking is to add a command that developers can run to increase the version number running

poetry version $(type)

been the type major, minor or patch. I have tried adding an

experimental_run_shell_command

and sort of works, but I can’t find a way to run it in all packages. Currently I have to do

./pants run src/python/library1:bumpversion

but ideally I would like to do

./pants run --changed-sinse=main ::bumpversion

. Is this possible? also, it is possible to pass arguments to that run command? here the command:

experimental_run_shell_command(
        name="bump_version",
        tags=["bump_version"],
        workdir=f"src/python/{name}/",
        command=f"poetry version patch",
    )

Maybe I should write a goal…?? if so, do you have any examples? Some extra option in the python_distribution target/goal?? Thanks in advance!!

Hey! I am having some headaches using Pants on an M1 mac. Running

pants test ...

results in Pex errors with the following:

Failed to resolve requirements from PEX environment @ /Users/peder/.cache/pants/named_caches/pex_root/unzipped_pexes/34b6a8b7483f933e686d61c8b7737afee5e7dd24.
Needed cp39-cp39-macosx_12_0_x86_64 compatible dependencies for:

I managed to get this working for building a Pex binary and specifying the target platform, but that isn't possible for

python_tests

afaik?