Hi Folks! I’m getting alot of errors running pex targets … pants is complaining about not being able to find matching distributions. For example :

ERROR: No matching distribution found for SQLAlchemy==1.3.22

Any ideas?

https://twitter.com/peter_szilagyi/status/1437646118700175360 Pants Build retweeted: Between the 3 Sept and 10 Sept, secure env vars of all public @travisci repositories were injected into PR builds… https://twitter.com/i/web/status/1437646118700175360 Twitter

TIL: This works

kwargs.get("tags", []) + ["my_tag"]

This doesn't work

kwargs["tags"] = kwargs.get("tags", []).append("my_tag")

Any reason why the later doesn't work? maybe all lists are read only and

+

creates a new list that is the combination of the two?

Bike shed opportunity - feedback welcomed! We're adding proper "target generation", which allows for us to fix a lot of issues with the old "macros" we have (

python_requirements

,

poetry_requirements

), and unlocks new boilerplate reduction. We need a syntax for how to address generated targets, e.g. in the

dependencies

field. It needs to include the generating target's address so we know how to generate it. The current proposal is

dir:generator!generated

, e.g.

//:reqs!Django

or

src/py/project:lib!foo.py

. But,

!

requires quotes to work in the terminal 🤷 The other alternatives to

!

are

=

and `@`:

Hi! A multi-part message coming up. First of all, huge thanks to the team for the support these last few weeks. I’ve managed to perform the first migration phase our monorepo to pants (linting and testing) and it works perfectly. So thanks again! Now, the complicated part comes: migrating our old custom tooling. I’ll spread this in two messages so that we can have separate threads.

First is doc building: currently we use

nox

to build our docs (a single docset for the whole monorepo), which basically implies starting a new virtualenv, installing sphinx and a couple of things and running

sphinx-build

. This command needs to be able to load the libraries, as we have all sorts of autodocs features. I have been thinking on how to achieve that with pants and I guess I should create a new target that would be

sphinx_docs

and it would have all the libraries as dependencies. However, it seems strange that this has never come up (I have searched slack to no avail) so I thought I’d ask first.

The second part is a bit more complicated. Each of the applications of the monorepo is run using Airflow, and for this each application dynamically generates a set of tasks that are then put together by the tooling. The process currently is as follows: 1. We call the

build-dags

command in our custom tool 2. For each of the libraries inside the

app

folder, the tool loads

library.dag

and executes a

generate_tasks

function, which returns a list of airflow tasks and their dependencies. 3. Once all tasks have been collected, the tool figures out the DAGs (you can have cross-app DAGs) and writes an output YAML file that is deployed from the CI in airflow. How to achieve that with

pants

? My idea is to break this into two steps: 1. On

./pants package

, not only the

pex

are generated but also YAML files with the tasks for each app/library. This would mean creating a new target

airflow_tasks

that would load the

library.dag

, run the

generate_tasks

function and dump the result in a per-app YAML. 2. Create a new goal

./pants build-dags

(or maybe it could be done with

package

if

pants

supports “ordered” builds) that has the

airflow_tasks

as inputs and then puts together all the YAMLs and builds the DAGs. I guess this is how I would do it, but looking at the plugins docs it all seems a bit daunting. Any example plugins I can look at to understand how to do this myself?

Hi Pants! Is there a quick way to specify "all targets in and under this directory", like how we can

./pants test ::

at the top level?

./pants goal path/to::

does not seem to work.

Separately, is there a way to load pex into the environment like how we can load it to a jupyter notebook session?

https://twitter.com/jacobian/status/1438135448476401668 Pants Build retweeted: This is intensely cool: you can now access AWS resources from GitHub Actions without needing to store AWS credentia… https://twitter.com/i/web/status/1438135448476401668 Twitter

Hey any idea when pants

2.6.1

will be marked stable? hoping to pick up some of those changes.

Is there an alternative name to the BUILD file recommended on the documentation besides

BUILD.pants

that you guys recall?

Hi Pants! If my pex binary depends on a file target, in the code I put in this pex binary where should I look for the file? (e.g. the code needs to load a yaml file)

Hi all! Is there a way to override the default

pytest

naming conventions? I’ve tried adding a

pytest.ini

overriding the global test file naming convention to

*_test.py

but pants is still picking up a file named

test_case.py

. I may be missing something obvious 🤔

OK, I feel silly for asking because I'm sure it's been solved, but I searched and didn't see... how do you do a python library with eg C extensions? Or worse, cuda? (got some internal libraries which do this, and while we could package them and then use them as third-party extensions, it's not elegant). Do I need to write a plugin of some sort?

Hey yall! Is there a bash command I can run to kill the pants daemon in a general way? Just creating a

make

command to nuke my repo for a clean restart.

Does running

pants run

happen inside of a pex? I'm getting an error when running the pex, but not via run. Hitting

NotADirectoryError: MultiplexedPath only supports directories

from

importlib_resources.files("my.dumb.package")

JVM v2 plugin bikeshed: 

jvm_dependency

 , 

jvm_requirement

, 

jar_requirement

, 

java_jar_requirement

, 

jvm_jar_requirement

, or some other name?

Hi! Editor question: has somebody managed to get a good vim config to work with pants? I run all the ide functions (pylint, etc) on vim so I need to start the editor in an environment that knows about third party deps, the libraries, etc. If I create a virtualenv, then I can’t really run any pants command from there, so I was wondering if I was missing something

https://twitter.com/pantsbuild/status/1438612080010186760 Did you know Pants 2 supports Python, Shell, Protobuf, and gRPC? It gets better: adding support for Go, Java, and Scala is in progress. Stay tuned! https://t.co/tLL7h1xBwV (P.S. Come say hi at https://t.co/93cNl1ay12 if you'd be interested in helping to add JS support next.) Twitter

https://twitter.com/stuhood/status/1438626100515463173 Pants Build retweeted: Very excited about this! We have some neat tricks up our sleeves (more on that soon), and generic language support… https://twitter.com/i/web/status/1438626100515463173 Twitter Did you know Pants 2 supports Python, Shell, Protobuf, and gRPC? It gets better: adding support for Go, Java, and S… https://twitter.com/i/web/status/1438612080010186760 Twitter

Hi Pants! I was wondering if you can add a feature to error out specifically for circular dependencies. This is something we typically run into, especially for people new to build systems; it's ambiguous when we're using automatic dependencies and on the error line it just says

E   ImportError: cannot import name 'xxx' from 'xxxmodule' (path/to/xxx.py)

It will be much more helpful if the error actually says "circular dependency detected: file name 1, file name 2"

Hi all! I wanted to understand how caching and dependencies work, for example when running in a CI. I find myself a lot of times modifying a config file in the repo (declared as a

resources

) and I’d like to avoid rerunning unnecessary tests when this happens (but also I want to run a few relevant ones). I was thinking of setting up a special set of “config tests” but I am not sure what happens if the only changed files are for example yaml included in resources. Can somebody shed light on this?

Hi Pants! I'm following this guide to set up my IDE with pants, but failed at this step:

Pants sets up its development virtualenv at 

~/.cache/pants/pants_dev_deps/<arch>.<version>.venv/

. Point your editor to the 

bin/python

 file in this folder, e.g. 

~/.cache/pants/pants_dev_deps/Darwin.py37.venv/bin/python

.

I currently have this in my environment but no

pants_dev_deps

:

❯ ls ~/.cache/pants
lmdb_store   named_caches setup

Any help?

Hello we are using our own

PyPi

mirror hosted on AWS CodeArtifact that requires a username and token to access. Is there any configuration in

Pants

that will allows us to inject the PyPi token for

Pants

to use, or make it use

pip.conf

which is setup with the username and token for the PyPi mirror.

https://twitter.com/pantsbuild/status/1440004063555375113 Pants 2.6.1 is now available! 🏰https://pypi.org/project/pantsbuild.pants/2.6.1/ Twitter

Hey pants, some of our tests inconsistently fail in CI and I was wondering if I could get some guidance debugging this. We are running

./pants\ --print-stacktrace\ -ldebug\ --pex-verbosity=9\ run\ ci/src/python/pynest_ci:runner\ --\ regular_unit_test

, but it doesn’t seem to give enough information to figure out the issue. It looks like

requirements.pex

builds fine, but when the test tries to build

pytest_runner.pex

it can fail.

17:25:57.58 [INFO] Starting: Building requirements.pex with 31 requirements: GPUtil==1.4.0, boto3-stubs[s3]==1.18.35, boto3==1.17.112, botocore==1.20.112, catboost==0.26.1, daemonize==2.5.0, dask-gateway==0.9.0, dask[dataframe]==... (434 characters truncated)
17:26:25.49 [INFO] Completed: Building requirements.pex with 31 requirements: GPUtil==1.4.0, boto3-stubs[s3]==1.18.35, boto3==1.17.112, botocore==1.20.112, catboost==0.26.1, daemonize==2.5.0, dask-gateway==0.9.0, dask[dataframe]==... (434 characters truncated)
17:26:25.49 [INFO] Starting: Building pytest_runner.pex
17:26:25.52 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
...
pid 2787 -> /root/.cache/pants/named_caches/pex_root/venvs/8427a8787e07d8e0828aa91a7c0695bba322863d/f28b3dbba3c9dae1b4357adde5b079b8b3ca9fac/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /root/.cache/pants/named_caches/pex_root --log /tmp/process-executionHBj8Tl/.tmp/tmph9shut22/pip.log download --dest /tmp/process-executionHBj8Tl/.tmp/tmplh07wwwf/usr.local.bin.python3.7 GPUtil==1.4.0 boto3-stubs[s3]==1.18.35 boto3==1.17.112 botocore==1.20.112 catboost==0.26.1 daemonize==2.5.0 dask-gateway==0.9.0 dask[dataframe]==2021.7.2 distributed==2021.7.2 future==0.18.2 hypothesis==6.17.4 jellyfish==0.8.8 moto==1.3.14 numpy<1.21 pandas==1.2.5 parmap==1.5.2 protobuf==3.17.3 pyarrow==5.0.0 pydantic==1.7.4 python-json-logger==2.0.2 pyyaml==5.4.1 requests==2.22.0 scikit-learn==0.24.0 shap==0.31.0 simplejson==3.17.5 snowflake-sqlalchemy==1.2.3 torch==1.9.0 tqdm==4.62.2 types-PyYAML==5.4.6 types-protobuf==3.17.4 types-requests==2.25.6 --index-url <https://pypi.org/simple/> --extra-index-url <https://pypi.cbhq.net/> --find-links ~/wheelhouse/ --retries 5 --timeout 15 exited with -9 and STDERR:
None

I attached the output for the failing tests as well.

When using pants tailor - it always generates

BUILD

files, even if the config specifies

build_patterns = ["BUILD.v2"]

is there a way to change what gets generated

Also the error message is:

Exception message: 1 Exception encountered:

  ResolveError: Directory 'build-tools/bins' does not contain any BUILD files.

Which would be better if the error message would take the pattern from the config variable