(I don't have Twitter so this is my tech community) How do people handle reviewing changes to a file-generator? Our engs have a file checked into git because it means they can review the changes made to the generator, but other than that there's no point.

Hi folks, quick question. I'm trying to use a

requirements.txt

with a

GIT_TOKEN

env variable that usually get expanded when using pip. For example:

my-lib@$GIT_TOKEN@github.com/my-org/my-lib.git@v1.0.0

but when trying to use the

python_sources

I get a

ValueError

saying it's an invalid URL. What's the best way to tackle this?

Is there an introspection that will show only the targets declared in build files, and not all the generated targets like in

./pants list ::

?

Fun little Sunday project results. Apparently creating a VSCode extension is wayyyy easier than I had previously assumed 🙂 Monitoring my source roots and transitive targets in the VS Code sidebar

Hi all, Is there any way I can make the pex file work on windows? And if not then what can be the ideal solution to distribute a python project as executable from the pants project? Has anyone tried it before?

Hi all. I am evaluation if pants can be used to manage our data pipelines codebase. Its mostly Airflow / python codebase. I have this test which is giving me issues

import pytest
from airflow.models import DagBag

def test_no_import_errors():
    dag_bag = DagBag(
        dag_folder="src/airflow-dags/tunein/dags",
        include_examples=False,
        read_dags_from_db=False
    )
    assert dag_bag.size() > 0, "Loaded dags"
    assert len(dag_bag.import_errors) == 0, "No Import Failures"

Generally, unit testing airflow code requires creating

DagBag()

which basically scans the

dags

folder to load the DAG. Any ideas on how to get tests like this working with Pants?

pants-adjacent: Does anybody know if you can make pytest autouse a fixture based on a mark?

Hi all 🙂 I am trying to use remote caching (not execution) using https://www.buildbuddy.io/. I know that is not listed here, but it is here why I was assuming that it might work with pants as well. It has a good free plan and good docs for configuring a deployment to the cloud. The caching sections looks as follows:

local_cache = false
remote_cache_read = true
remote_cache_write = true
remote_store_address = "<grpcs://remote.buildbuddy.io>"
remote_store_headers = "{'user-agent': 'pants', 'x-buildbuddy-api-key': 'token'}"
remote_instance_name = "buildbuddy-io/buildbuddy/local"
remote_store_chunk_upload_timeout_seconds="3600"

I could not confirm that it worked to be honest. Can I somehow test the remote connection or enable verbose logs?

vscode ➜ $ ./pants test libs/lib_common/tests/grid_test.py 
10:33:33.20 [INFO] Initializing scheduler...
10:33:33.48 [INFO] Scheduler initialized.
10:33:35.11 [INFO] Canceled: Building pytest.pex from lockfiles/python/tools/pytest
10:33:35.41 [INFO] Canceled: Building pytest.pex from lockfiles/python/tools/pytest
10:33:36.68 [INFO] Completed: Building 3 requirements for requirements.pex from the lockfiles/python/default resolve: Shapely==1.7.1, numpy==1.21.0, pyproj==3.1.0
10:33:37.35 [INFO] Completed: Building local_dists.pex
10:33:37.81 [INFO] Canceled: Building pytest_runner.pex
10:33:38.28 [INFO] Completed: Run Pytest - libs/lib_common/tests/grid_test.py succeeded.

✓ libs/lib_common/tests/grid_test.py succeeded in 0.90s (cached remotely).
vscode ➜ $ ./pants test libs/lib_common/tests/grid_test.py 
10:34:06.47 [INFO] Completed: Run Pytest - libs/lib_common/tests/grid_test.py succeeded.

✓ libs/lib_common/tests/grid_test.py succeeded in 0.90s (memoized).

The pants logs seem okay, but I cannot see any artifacts in the buildbuddy UI. Did I miss something?

Hi I have been trying to build a package which has dependency on another package. Although I don’t see any files from the dependency. I have this toy repo where I am playing around with it. There are two packages:

package_a

and

package_b

,

package_a

has dependency on

package_b

I do see the

package_b

as dependency in generated

setup.py

. Is this how it’s supposed to work? The python_distribution for

package_a

is defined here and the repo

Folks using

[python-repos]

for private indexes: would you find it useful to let you scope this to specific resolves/lockfiles? For example, your user lockfile needs to use your custom index, but the Black tool lockfile is fine w/ just PyPI

Hats off to @bitter-ability-32190 for making

./pants --changed-since=HEAD fmt lint check

substantially faster the past 3 months 🚀 1.

fmt

and

lint

reuse the same process results 2. MyPy's cache now works in 2.14! (

check

)

Hey, check it out! Pants team members @hundreds-father-404 @bitter-ability-32190 @rapid-bird-79300 got together with InfoQ podcast to discuss the team's commitment to maintaining a welcoming and supportive Pants community, and some of the many things we do to cultivate and maintain that culture. https://www.infoq.com/podcasts/welcoming-open-source-community/

One of the things we do, by the way, is meet monthly. All members of Pants team, maintainers and contributors alike, are invited to the join us on the 3rd Monday of every month at 9am Pacific. The next meeting is this Monday, August 15th. Ping me no later than this Wednesday if you have a topic to put on the agenda.

Hello. I wonder if I'm doing something wrong with my tests or at least how to debug that issue. My pytest tests take a long time to execute. Each test file takes between 4 and 8 seconds to execute and there are ~200 files. On a dual core, it means it takes between 7 and 14 minutes to execute the first time. (Then most of the results are cached). That's a long time so I wonder if something isn't right with my setup.

What would be the cause for pants 1.30 suddenly failing with this stacktrace? We suddenly encountered this last week and after clearing out the cache, we still run into this issue on our Team City hosts, but can't replicate locally. Mainly trying to understand why pants would fail this way.

Hi I am getting import errors for a very basic project. Feels like I am making a silly mistake. here are the logs

➜ ./pants roots                                                     
src/python
test/python

➜ ./pants filedeps --transitive test/python/tunein/test_samplelib.py 
src/python/tunein/BUILD
src/python/tunein/samplelib.py
test/python/tunein/BUILD
test/python/tunein/test_samplelib.py

➜ ./pants test test/python/tunein/test_samplelib.py
00:07:33.40 [INFO] Initializing scheduler...
00:07:33.63 [INFO] Scheduler initialized.
00:07:35.62 [ERROR] Completed: Run Pytest - test/python/tunein/test_samplelib.py:tests failed (exit code 2).
============================= test session starts ==============================
platform darwin -- Python 3.7.10, pytest-7.0.1, pluggy-1.0.0
rootdir: /private/var/folders/0t/dmh8ynt13pbc2y2stvb0by6c0000gn/T/process-execution34eEWG
plugins: cov-3.0.0
collected 0 items / 1 error

==================================== ERRORS ====================================
____________ ERROR collecting test/python/tunein/test_samplelib.py _____________
ImportError while importing test module '/private/var/folders/0t/dmh8ynt13pbc2y2stvb0by6c0000gn/T/process-execution34eEWG/test/python/tunein/test_samplelib.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/vishwakarma/.pyenv/versions/3.7.10/lib/python3.7/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test/python/tunein/test_samplelib.py:1: in <module>
    from tunein.samplelib import sample_func
E   ModuleNotFoundError: No module named 'tunein.samplelib'
- generated xml file: /private/var/folders/0t/dmh8ynt13pbc2y2stvb0by6c0000gn/T/process-execution34eEWG/test.python.tunein.test_samplelib.py.tests.xml -
=========================== short test summary info ============================
ERROR test/python/tunein/test_samplelib.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.11s ===============================

any suggestions on how to solve this

Is there a way to run shell commands easily with pants while using a virtual environment defined by a

poetry_requirements

target? Normally I would use

poetry run <shell command>

, but I would like to start using pants for this of course.

Hi, I'm trying to solve a problem that occurs when I run

./pants export ::

in a package that has Horovod as a dependency. During installation, Horovod looks for TensorFlow and if it finds it, it compiles C++ code that is linked against the TensorFlow library. If not, the support for TensorFlow is disabled. Although I'm installing both Horovod and TensorFlow, the respective Horovod submodule is not being built. I've reduced my problem to the call to

pex

, for which a minimal example would be the following:

# First try, as pants calls pex:
$ python -m pex --disable-cache --output-file hvd.pex horovod[tensorflow]
...
$ ./hvd.pex
>>> import tensorflow
>>> import horovod.tensorflow
ImportError: Extension horovod.tensorflow has not been built

# Retry, forcing Horovod to build its Tensorflow submodule:
$ HOROVOD_WITH_TENSORFLOW=1 python -m pex --disable-cache --output-file hvd.pex horovod[tensorflow]
  ERROR: Command errored out with exit status 1:
  [...lots of output...]
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
  ModuleNotFoundError: No module named 'tensorflow'
  CMake Error at /tmp/horovod-cmake-tmpmdzj4w8z/cmake/data/share/cmake-3.13/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
    Could NOT find Tensorflow (missing: Tensorflow_LIBRARIES) (Required is at
    least version "1.15.0")
  [...more output...]

# Just for comparison, pip works:
$ python -m venv tmpenv
$ source tmpenv/bin/activate
(tmpenv) $ HOROVOD_WITH_TENSORFLOW=1 pip install --no-cache-dir horovod[tensorflow]
[...]
(tmpenv) $ python
>>> import tensorflow
>>> import horovod.tensorflow
>>> ^D
(tmpenv) $

I realize that it's mainly a problem of Horovod as a package, because it needs TensorFlow to already be installed in order to build its submodule, while it does not list TensorFlow as a build dependency. However, pip follows the order of the dependencies for installation, installing TensorFlow first and then Horovod. Is there a way to make it work with pants/pex? E.g. forcing the order of package installation as in pip, specifying (somehow) an additional build dependency for horovod, or some other way I might be missing? Thanks! PS. If this is not the right place to post this since it is about pex and not pants directly, please let me know where I should post it instead.

For buildable targets like

pex_binary

, are there any options now or plans in the future to support pre/post-packaging script injection? Example: a project contains some source code that an npm packaged tool (e.g.,

gulp

) uses to generate html/css/js assets for a Python webserver that's packaged into a

pex_binary

with the built assets bundled as a

resource

. It would be nice to still be able to do

./pants package webserver

instead of A)

./build-support/prepackage_prepare_script.sh && ./pants package webserver && ./build-support/postpackage_cleanup_script.sh

or B) build assets continuously and check into source control. Presumably the cleanup step can be eliminated if Pants does all the packaging in a tmp folder.

I noticed Javascript is on the languages roadmap -- is there a blog post or doc about what that's going to look like? Is it planned to be as powerful as Python in terms of packaging and dependency inference (e.g., could a webserver written in nodejs using npm dependencies be packaged/tested/ran)?

Between Pant 2.9 (I think) and now 2.11, running

./pants generate-lockfiles --resolve=pytest

gives us pretty different outputs. It seems the later version gets rid of almost all of the hashes. Is that expected? We hadn't switched to a pex lockfile yet, but I was surprised to see the different behavior. The pex form seems like it preserves hashes.

What could cause

experimental_shell_command

to report a

/usr/bin/bash: my_script.sh: command not found

error regarding the

command="./my_script.sh"

script field? When I change it to something like

ls -la ./ && echo $(pwd)

I see the expected

.sh

file bundled from the

shell_sources

dependency. I also see it and can run it directly from the preserved workspace. The error just doesn't make sense and I've trippled-checked the spelling. Using 2.13.0.dev5.

Another solution could be to allow injecting a “prefix” to any process invocation. For the immediate issue, if you are willing to edit a copy of the pants sources, you could just hard code what you want and invoke Pants using the checked out copy of Pants.

Hello, I have an issue related to lambdex. I create a lambda via the target python_awslambda with the runtime python3.9. I need a recent boto version and I can see in the zip file that the latest version is there (1.24.47). But if I run the lambda and ask for the version with

boto3.__version__

I get 1.20.32, which I think is the boto3 version included in the runtime. Is there a way to force the usage of the packaged boto3 over the one on the runtime?

I commented on a closed issue about a problem with trying to use tensorflow that doesn’t work. https://github.com/pantsbuild/pants/issues/16055#issuecomment-1210154370

Hello, I just wanted to let you know that I successfully deployed Pants on prod for my project at work. Everything works as expected. There are some edges to polish but overall, it's great. Thanks to everybody who helped me reach this point! 🙂

The link to community overview on the contributor-overview page seems to be broken

Is there some recommendations about managing pants cache? I have my

~/.cache/pants

getting 40GB right now and I'll run out of space soon. I could just

rm -rf .cache/pants

I guess, but I was wondering if there is a command or something to trim old stuff or anything.

I have a team member trying to access some environment variables within pytest code but we're unable to find the environment variables inside of the sandbox that 1.30 operates in. Is there a way to do something like

export PANTS_PYTEST_VAR1=test

and have the python code see that env var?