Ok, that make couple of hours I am trying to solve this one, when I try to run a python project, I have a problem with

psycopg2

that is not installed correctly. I have

Service A

that have

Library A

installed with

poetry add -e ../../libraries/a

and this lib have the

psycopg2-binary

in the

pyproject.toml

, if I try to run

poetry run python run.py

which is the entrypoint of my

Service A

everything run perfectly, but when I do

./pants run src/services/a/run.py

I get everytime... (same when I build a Docker container with

./pants package

the container image is done ok but when I run it same error)

File "/home/cerberus/.cache/pants/named_caches/pex_root/venvs/s/71f9e365/venv/lib/python3.10/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 811, in dbapi
    import psycopg2
ModuleNotFoundError: No module named 'psycopg2'

in my

BUILD

file in

Library A

I have tested every possible combination of

module_mapping

and

overrides

to try to make it works.

im having an issue publishing to a private repo from a github runner, im able to

docker login

, but for some reason pants is not inheriting the auth context. I have

[docker].env_vars

set to

DOCKER_CONFIG=%(homedir)s/.docker

but for some reason its not getting it

In my open source project (using Python and Scala) using Pants: https://github.com/liga-ai/liga I want to migrate the Scala part from sbt to pants, but last time I tried

deploy_jar

https://github.com/pantsbuild/pants/issues/17790 , I failed to exclude Apache Spark. Am I doing something wrong or it is not supported in Pants？What is the

deploy_jar

equiv of

sbt package

and

sbt assembly

?

Hi question please. I have question please, if I have an existing python project using poetry with following where i have this in my

pyproject.toml

[tool.poetry.scripts]


[tool.black]
line-length = 120


[tool.isort]
profile = "black"
multi_line_output = 3 # Vertical Hanging Indent
line_length = 120
from_first = true

and a

tox.ini

file containing something like this

[tox]
envlist = lint, pytest-ci, mypy, vulture
skipdist = true

[testenv:lint]
basepython = python3.9
allowlist_externals=*
list_dependencies_command=echo
commands =
    flake8 lambdas
    pylint lambdas --rcfile=.pylintrc

will pants pick up these configuration or there something I can do so it respects any of these configurations

Hey all, I am trying to figure out why the same django app running in docker is almost using double the memory when running using pants compared to running my old manual docker file. I am using the template example-django in my pants project. and I have gunicorn configured with same params in both cases.

workers = 1
threads = 8
timeout = 0

HI have a question (more of seeking opinion on something) is there any recommended convention one can use with

confetest.py

files. As am adopting pants from a monlith repo to a mon-repo. I wondered anyone either • copying the conftest.file into each root_pattern where its required ◦ this would cause heavy duplication, which might be ok if the conftest file stays consistent • having a shared folder that is also known by pants containing a conftest file and using symlink for the other packages to point to it. • does pants have some magic that makes that easier ◦ with the potential option of overriding or enhancing package specific conftets. sorry if this is open ended

Gave a talk at my local Python Meetup last night about Pants. They loved it! Thanks for all the hard work Pants team!

Hey, I am working on integrating

pants

into our CI/CD (GitHub Actions) and I would like to use

changed-since=origin/main

for example, but within a module within the repo - e.g.

products/productA

but it doesn't let me 😞

Error: 5.99 [ERROR] You used `--changed-since` at the same time as using target arguments. Please use only one.

My intention was to set up separate workflows for each of the products. I see the value in doing it without limiting it, but the problem I currently have is that the IaC deployments need to go to different accounts for each product and it's currently configured at the github workflow level. Do I have an option other than to use approach #2 from the pants documentation of removing the

--changed-since

related flags? Medium term, I plan on integrating the iac credentials into the build system rather than relying on the CI/CD pipeline which seems to be the superior solution.

I’m so close to handing over my service to pants but have hit the last hurdle pretty hard - I hit the issue described here: https://pantsbuild.slack.com/archives/C046T6T9U/p1670366698746769 running

./pants package ::

in an AWS codebuild container. Following that thread, I’ve managed to use complete_platforms to build and test inside a local dev container, and added

pex3 interpreter inspect --markers --tags

for both the codebuild container (where it builds in CI) and for my dev container (for local dev). I still get the same error in Lambda though - am I doing something wrong, or do I need to build for the lambda’s runtime container too in

complete_platforms?

Is there an option that controls whether Pants resolves the path of bootstrap script in suggestions? Sometimes it seems to be putting this in stuff I commit in things it generates (like lockfiles), and I don't really want my local repo path being committed to code... example:

// This lockfile was autogenerated by Pants. To regenerate, run:
//
//    /home/jreed/devel/aiven/aiven-core/pants generate-lockfiles --resolve=pylint

I'd rather it just say

./pants

tbh. Is that possible?

Hey experts, I’m trying to get mypy running in my pants workspace, but for some reason running

./pants check path/to/dir

just gives me

error: Cannot find implementation or library stub for module named

errors for all my imports. Running the actual

./pants run

seems to work just fine. Do I need to setup mypy somehow to correctly understand the structure of my pants workspace? For example explicitly putting the root or something in the options?

Thanks to everyone here who helped with my dive into Pants. I've learned a lot, and I like the direction that Pants is going. However, for my needs (easy onboarding of readers into the book's code repo) I've decided that there are currently too many barriers to entry for the average reader and thus a high probability of discouraging them. I'm planning to use Hatch for the time being, but I will periodically check back in with Pants to see how it's progressed towards removing the barriers. I'm telling you this because the usual experience is that a customer just stops using a product and disappears, so the provider is left without this kind of essential feedback (e.g.: why customers stop using your product).

I noticed that

./pants run ...

and

[subprocess-environment]

interact in inconsistent ways, is this intentional/expected? https://gist.github.com/huonw/3e6ebe4c9a4037a31655fce34eea357d 1.

./pants run ./main.py

(running a python source directly) sets the env vars from

[subprocess-environment]

2.

./pants run //:pex

(running a

pex_binary

) does not set the env vars

Hi all, I am evaluating the use of pants for a mono repo that contains all of our internal libraries. The idea is to move from many single python poetry libraries to one mono repo that publishes the same libraries. Some of the libraries depend on libraries that are in the same project. As example, Got a logging library that every library depend. Then a stream library that some depend. No circle dependencies, just level dependencies that makes it really annoying/hard to keep the dependencies up to date. Is there any example out there of this use case? Thanks in advance!

Hi all, I'm encountering an issue whilst doing

./pants package ...

which either ends in one of two scenarios: 1. Running with

./pants package

results in a nasty traceback stating that the pantsd process was killed during the run

Traceback (most recent call last):
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/bin/pants", line 8, in <module>
    sys.exit(main())
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 112, in main
    PantsLoader.main()
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 108, in main
    cls.run_default_entrypoint()
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/pants_loader.py", line 90, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/pants_runner.py", line 89, in run
    return remote_runner.run(start_time)
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/remote_pants_runner.py", line 117, in run
    return self._connect_and_execute(pantsd_handle, start_time)
  File "/home/liam/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/lib/python3.9/site-packages/pants/bin/remote_pants_runner.py", line 151, in _connect_and_execute
    return PyNailgunClient(port, executor).execute(command, args, modified_env)
native_engine.PantsdClientException: The pantsd process was killed during the run.

If this was not intentionally done by you, Pants may have been killed by the operating system due to memory overconsumption (i.e. OOM-killed). You can set the global option `--pantsd-max-memory-usage` to reduce Pantsd's memory consumption by retaining less in its in-memory cache (run `./pants help-advanced global`). You can also disable pantsd with the global option `--no-pantsd` to avoid persisting memory across Pants runs, although you will miss out on additional caching.

If neither of those help, please consider filing a GitHub issue or reaching out on Slack so that we can investigate the possible memory overconsumption (<https://www.pantsbuild.org/docs/getting-help>).

Re-running with

./pants --pantsd-max-memory-usage=8GiB package ...

had no affect on the outcome however and running with

/usr/bin/time -v

prefixing the aforementioned command gives an output which suggests a maximum of ~48MB was used (

free -h

suggests there is

10Gi

memory available on my WSL Ubuntu system). 2. Running with

./pants --no-pantsd package ...

(as suggested by the error message from point 1) results in different message:

/github/home/.cargo/git/checkouts/lmdb-rs-369bfd26153a2575/6ae7a55/lmdb-sys/lmdb/libraries/liblmdb/m:2432: Assertion 'mp->mp_pgno != pgno' failed in mdb_page_touch()
Aborted

Does anyone have experience with the above type of problem? How would I go about debugging the root cause behind the issue? Any help would be most appreciated 🙏 😃

Good monday everyone! Small question maybe not related directly to pants but this is a first for me to use Pex and I am having an hard time to make it works with Docker, Poetry and Gunicorn for a Flask application. My Services are Flask App but I need

gunicorn gevent psycogreen

as requirements from poetry only when building my Docker Image. Currently I have added them in another poetry group which appear to include the dependencies when building the Docker image but I can't make it work with the

pex_binary

target, tested to specify the

entry_point

or the

script

in

pex_binary

but I keep getting error message. Currently my build look like this

python_sources()

poetry_requirements(
    name="poetry"
)

pex_binary(
    name = "binary",
    entry_point = "run.py",
    layout = "packed",
    execution_mode = "venv",
)

docker_image(
    name = "img",
    registries = ["<http://docker.io|docker.io>"],
    repository = "repo/service",
    image_tags = ["{build_args.RELEASE_VERSION}"]
)

My Dockerfile looks like this

FROM python:3.10-slim
ARG RELEASE_VERSION=latest

RUN apt-get update \
  && apt-get install -y --no-install-recommends \
    python3-dev \
    libpq-dev \
  && apt-get clean autoclean \
  && apt-get autoremove -y \
  && rm -rf /var/lib/apt/lists/* \
  && rm -f /var/cache/apt/archives/*.deb \

COPY src.services.service/binary.pex /bin/app

ENV CONSUL [h=consul-svc,t=None]
EXPOSE 5001

CMD ["python", "/bin/app"]

This is working for development purpose, but we need

gunicorn

in production I was reading this https://pex.readthedocs.io/en/v2.1.43/recipes.html#gunicorn-and-pex but I don't see how I can reproduce this through Pants! Thanks 🙂

Go error when including

confluent-kafka-go

I'm having trouble building a go module that requires

<http://github.com/confluentinc/confluent-kafka-go|github.com/confluentinc/confluent-kafka-go>

I'm using

pants_version = "2.15.0rc1"

I have a single

go.mod

file defined at the repos root. When I attempt to run the go app

./pants run apps/go/greeter_en:greeter_en

I get the following error:

Engine traceback:
  in `run` goal
  in Package Go binary - apps/go/greeter_en:greeter_en
  in Compile with Go - apps/go/greeter_en:go-package
  in Compile with Go - <http://optios.com/pantsbuild/example-golang/libraries/go/greeter|optios.com/pantsbuild/example-golang/libraries/go/greeter>
  in Compile with Go - <http://github.com/confluentinc/confluent-kafka-go/kafka|github.com/confluentinc/confluent-kafka-go/kafka>

ProcessExecutionFailure: Process 'Generate Go and C files from CGo files.' failed with exit code 1.
stdout:

stderr:
In file included from /private/var/folders/_d/zt8y2x055597l63bhdztp82c0000gn/T/pants-sandbox-Pjctus/gopath/pkg/mod/github.com/confluentinc/confluent-kafka-go@v1.9.2/kafka/00version.go:24:
gopath/pkg/mod/github.com/confluentinc/confluent-kafka-go@v1.9.2/kafka/select_rdkafka.h:26:10: fatal error: 'librdkafka_vendor/rdkafka.h' file not found
#include "librdkafka_vendor/rdkafka.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I can build and run the app using go tools. from the repos root:

go mod tidy
cd apps/go/greeter_en
go build .
./greeter_en

The attached project demonstrates the above behavior.

We have a Python repo that uses a

conda

environment. This means we have to add

python_requirement

entries for all third-party dependencies used by goals like

test

and

package

to our root

BUILD

file or

python_infer

throws

UnownedDependencyError

. I’d like to run

./pants dependencies

on our tests to check that Pants can infer all the third-party dependencies successfully. What is the best way to do this?

Hi, I'm relatively new to Pants, I'm now evaluating it for use in our team, and I really enjoyed using the tool for about a couple of weeks or so! I have a few questions, I'll split them up so it's easier to answer:

1. I want to build a dockerized app that depends on

mysqlclient

that requires building a C library, but as Pants kinda pushes you to use pex, I'm ending up with building it not on the system that will actually run the app, and of course I'm getting a 'missing .so' error at run time. How do you solve this problem in a Pants setup? Maybe there's a way to add a dependency on top of existing pex? Or how do I prebulid a proper variant of

mysqlclient

and make sure Pants includes it specifically (or platform dependent)? I tried building a

pex_binary

with

complete_platforms

that include my system's and container system's output of

pex3 interpreter inspect --markers --tags

, but it fails with 'No matching distribution' for both.

2. I don't want to build 3rd party dependencies every time I make a change in our code. Is this the way to do it properly? I'm a little bit confused because the docker plugin may seem to be advertised like a thin wrapper around pex both in docker reference and in Benjy's post (I'm not trying to offend, just describing my experience)

3. Is there a way to specify persistent dirname for the unzipped pex? I need it for instance for code mapping for remote debugging. I know it's

$PEX_ROOT/unzipped_pexes/<some hash>

, and I can technically search the archive for the exact name, but it will be easier to just have it as a constant.

I’m trying to add golang code to a polyglot repo, and I’m finding that pants downloads Go modules into it’s sandbox much more frequently than I would expect. Even when doing a

pants run some.py

which has no relationship with any go code results in a lengthy Go module download. Am I doing something obviously wrong, or is this the expected behaviour?

Hello! I added new microservice to my repo and it uses updated dependency (cats-effects 3). And I'm trying to

./pants generate-lockfiles

And it generates a lot of errors, like

org.typelevel:cats-effect_2.13:2.0.0 or 2.1.4 or 2.3.0-M1 or 2.3.1 or 2.5.0 or 2.5.1 or 2.5.4 or 2.5.5 or 3.3.14 wanted by

  co.fs2:fs2-core_2.13:2.5.6 wants 2.3.0-M1
  ├─ co.fs2:fs2-io_2.13:3.3.0 wants co.fs2:fs2-core_2.13:3.3.0
  │  └─ org.http4s:http4s-core_2.13:0.23.16
  │     ├─ org.http4s:http4s-ember-core_2.13:0.23.16
  │     │  └─ org.http4s:http4s-ember-server_2.13:0.23.16
  │     └─ org.http4s:http4s-server_2.13:0.23.16
  │        └─ org.http4s:http4s-ember-server_2.13:0.23.16

Scala compiler is ok, as it different projects. Issues only with pants. Maybe I missed something in docs, but Are there any variants to use different versions of one dependency in one repo?

Hello all! If I were to setup Buf

breaking

command (more info here -> https://docs.buf.build/breaking/overview) within Buf backend, should I: 1. add a second command in lint goal (https://github.com/pantsbuild/pants/tree/main/src/python/pants/backend/codegen/protobuf/lint/buf)? or 2. add it to check goal? or 3. it should be a separate plugin? So far I don’t know how I could achieve solution 1 since there’s already a command buf lint behind the lint goal. Solution 2 seems more convenient and makes sense as it deals with type compatibility (see usage here https://docs.buf.build/breaking/usage). Any opinion? Thanks!

Hi Team, we are using Pants for pyspark apps. Recently we faced an issue where while executing the test-suite , it started failing because it cannot find the enough core. We realised it was happening because it started executing the all pyspark test cases in parallel. As a workaround for we are using

--debug

flag to run the pyspark tests in sequence. Is it possible to control the number of tests that can run in parallel ?

Hello again! Just want to make sure, Is it possible to set multiple lockfiles for scala to define different versions of one dependency, as it mentioned in python docs? As scala docs doesn't include such article, should it be anything, like

[jvm.resolves]
non-default-resolve = "3rdparty/jvm/defaultV2.lock"

? Thank you in advance!

👋 Is there a way to reference

pytest

in the

dependencies

parameter of a

python_tests

target?

python_tests(
    dependencies = [
        "tests/my/special/test:lib",
        "3rdparty/python:pytest", # <-- I have test sources that `import pytest` so I think the type checker wants to see an explicit dependency stated here…
    ],
)

if you have dependency inference turned on you shouldn't need to do that. try remove the explicit dependency and see if pants can see that dependency by running:

./pants dependencies <path to python test file>

(dep inference should be on by default) https://www.pantsbuild.org/docs/reference-python-infer#imports