I am using python_awslambda according to the documentation by copying the zip to docker_image. As recommended, I set [python].enable_resolves = true and get a resolve error that I cannot resolve. Details of the error are in the thread.

Hi, I am trying to use common resolve for two poetry generated projects inside a mono-repo and I get error about missing 3rd party dependency when running test using pants I have created a toy repo to reproduce it. I think I might be doing something wrong. https://github.com/ShantanuKumar/pants-multi-poetry In this case pandas is the only 3rd party dependency.

After 2.13.0rc0 getting error “not enough parameters to specify X” when packaging a python_distribution where X is the address to a python_source w/o specifying a specific parametrization of its resolve field. Things work fine for packaging pex_binary. Temporarily able to fix by picking any one of the resolves, but I expect this is unintended for python_distribution.

hey guys, I’m setting up airflow and was wondering if there are any best practices or recommendations in getting airflow set up in our monorepo. specifically: 1. we’re using poetry to track all our common dependencies, but since airflow pins its dependencies and doesn’t support poetry, we’re not able to simply do a

poetry add apache-airflow

since it leads to an unsatisfiable list of dependency constraints 2. because of this, airflow requires you provide a

requirements.txt

with a list of 3rd-party libraries used in your DAGs. does pants have a way to generate a requirements file from a

python_sources

target? am i thinking about this the wrong way? 3. haven’t started working on adding support for 1st-party code that’s imported into DAGs but I have a feeling this is not going to be very straightforward. any recommendations here?

Hey, not that I have answer for all these questions but how were you using airflow without pants in terms of 1st party dependencies? In the past I have used docker image to publish all the dependencies needed to run a DAG.

i haven’t gotten to supporting 1st party code yet 😅

Where are you running airflow? Self managed?

MWAA

deployed through pulumi

Possible bug: Add resolve=X override to a requirement R in python_requirements generator parametrized by resolve=X,Y. Then pants still generates two @resolve=X,Y targets, but the resolve for both is X. Remove the override and you have the same two targets but each has their respective resolve X or Y. I expected in the override case the @resplve=X target would not be generated at all. This is causing ambiguous import warnings in some cases.

I’m unsure how to handle the case where a python source depends explicitly (I.e., listed in its dependencies list) on a third party requirement conditional on which of the source's parametrizations is being used (in particular, of which resolve is in play). If only one is listed, pants complains when another resolve is in play. If it's parametrized, lots of off-diagonal targets get generated and leads to specification problems elsewhere. Am I missing something here?

It seems general consensus for semantic fixers has landed on a new

fix

goal. However this brings up a follow-up issue, which goal should run these to validate code?

lint

or

check

. Which then, IMO, turns into a question of, have we really nailed down the difference between

lint

and

check

w.r.t. which belongs where? I know at least for Python the delineation is fuzzy and even has a few maintainers unsure. I'd like to open a discussion for really nailing down exactly how someone can determine which tool belongs where, and why (with a slight lens on languages where it isn't obvious, like Python or JS). (Note this isnt a discussion about where fixers go, we can have that discussion after this one)

Hey, is anyone familiar with ChaosConf? Its central topic is reliability, with target audience being "SREs, developers, engineers, and technology leaders". I'm considering whether to do significant Pants outreach there. If you have experience attending or speaking at that conference, I'd love to hear your thoughts on what it was like. Thank you!

How do you copy a Python file when building a Docker image? I get this message:

The following files were not found in the Docker build context:
  * src/python/some_package/some_file.py

I've got some odd behavior with a library we're trying to use,

pyudev

. Some details in the thread, but TLDR is there's different behavior between using this library in a test and source code.

Hello, I want to use pants test to do some testing in python including database tests. I got >400 test cases and I am really happy that pants runs tests in parallel. But there are a lot of problems, when multiple database tests run at the same time, because they are all changing the database so that many tests are failing. Is there any solution to this problem except using [test].debug = true which disables parallelims?

Hey, Im curious, what would be the best way to run profilers like py-spy on a pants python target? So far I have issues like

Permission Denied: Try running again with elevated permissions by going 'sudo env "PATH=$PATH" !!

or py-spy not being able to find the pid/python code at all..

Is Pants able to infer dependancies between python test files? I have the structure like this:

src

|_ python

|_ package

tests

|_python

|_package

- test_1.py

- test_2.py

- BUILD

With a import in test_2.py like

from test_1 import function

However the test fails this import unless I explicitly provide python_tests the source for

tests/python/package

.

python_tests(name=“tests”, sources=[“tests/python/package”])

Strangely while the above works, it gives a Unmatched glob warning ([WARN] Unmatched glob from tests/python/package:tests’s

sources

field: “tests/python/package/tests/python/package”

I had wanted some confirmation about a interaction between lockfiles and

PANTS_PYTHON_REPOS_INDEXES

or the

python-repos.index

setting. I assume that custom indexes have to be set during the

pants generate-lockfiles

step to be used. If a custom index is set by

PANTS_PYTHON_REPOS_INDEXES

env var or a

pants.ci.toml

only during the

pants package

step, it will not be used for retrieving the locked dependencies. Is that a correct understanding?

Hello, I am currently building an

archive

which should contain multiple build artefacts. Those artefacts are of type

pex_binary

and

python_distribution

(whl file for python). These artefacts are built when they are included in the

packages

field. Is there a way to control their location within the archive? For files, there is

relocated_files

.

Hi, what is the syntax for copying a

python_distribution

(wheel) into a Dockerfile built by pants?

I think I'm missing something, but couldn't grok this from the docs. How can I get env vars propagated to

./pants run <docker_img>

? I've tried `--docker-env-vars="MYVAR"`:

MYVAR=1 ./pants --docker-env-vars="MYVAR" run <docker>

, but the

pex_binary

which is run is no dice. I suspect maybe

docker.run_args

?

QQ: when we run pants with

--changed-since

, will the changed files dependEEs be checked, or only dependents will be checked?

When contributing to Pants, how do I set the Pants version to use the Pants source code in my branch? For example, I want to add a new argument to the

test

goal, and want to be able to test it out manually via

./pants test [my_argument_here]

. When I run

./pants version

, I see

2.14.0.dev3

.

QQ: what does the

+-

signs mean in

--filter-tag-regex="[[+-]regex1,regex2,..., [+-]regex1,regex2,..., ...]"

(link)?

Sometimes I imagine a feature that I could enable which would have Pants warn/error on manually-specified dependencies which it also inferred (which as pants gets smarter, so does inference). This helps ensure we don't refactor our code and leave a stale dependency. Thoughts?

has anyone had success adding

pdb

with

./pants check

? there are some mypy errors that i’d like to drop into pdb to see. Things I’ve tried that didn’t work: 1. adding

pdb = True

in

mypy.ini

2. adding this in

pants.toml

[mypy]
args = ["--pdb"]

My generated

sdist

doesn't contain any sources. Any ideas of what could be the problem? I have the

python_sources

target defined...

@refined-addition-53644 do you have any links to docs on how you were using docker to publish your dependencies? currently i’m looking at bundling our code into a wheel to create a

plugins.zip

file and then generating a

requirements.txt

file semi-manually, but i can’t imagine this is the right way to do things https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-dag-import-plugins.html#configuring-dag-plugins-airflow-ex

Sorry i didn't add the disclaimer that I haven't used MWAA. I had used airflow (cloud composer) on GCP which was running inside a k8s cluster and we used to push docker image to our private docker repo during CI/CD. Then you can use kubernetes pod operator. I googled around for docker support for MWAA and I found this https://medium.com/@sohflp/how-to-work-with-airflow-docker-operator-in-amazon-mwaa-5c6b7ad36976