Is there any way to make

pants

aware of my aliases? E.g. I alias docker to podman, but pants doesn’t pick that up natively

In bazel I can tag targets as

manual

and they never run unless explicitly invoked directly.. or indirectly through a dependee. Can I do something similar in pants? In particular, I have a sphinx-build target that I never want to run in export-codegen.

Is there a way I can use something I extracted with a shell_command as a runnable in an adhoc_tool? As far as I can tell this is not allowed In particular, I and grabbing a http_source and untaring it. I'd like to use the adhoc tool to create a resource to be used by a pythonsource.

file(
    name="downloaded-sass",
    source=per_platform(
        macos_x86_64=http_source(
            url="<https://github.com/sass/dart-sass/releases/download/1.69.5/dart-sass-1.69.5-macos-x64.tar.gz>",
            len=3457742,
            sha256="75e29a5bd76069abf2532c9a4b36b164f1c91a461452f0fbdc4167fd1685550c",
            filename="sass.tar.gz",
        ),
        linux_x86_64=http_source(
            url="<https://github.com/sass/dart-sass/releases/download/1.69.5/dart-sass-1.69.5-linux-x64.tar.gz>",
            len=3697303,
            sha256="42b3d7b82098432f80d057da5e7464e96e6cd8d90827a9e3a47974f39c930218",
            filename="sass.tar.gz",
        ),
    ),
)

shell_command(
    name="extracted-sass",
    command="tar -zxvf sass.tar.gz",
    tools=["tar", "gzip"],
    execution_dependencies=[":downloaded-sass"],
    output_directories=["dart-sass"],
    root_output_directory="./dart-sass"
)

run_shell_command(
    name="sass",
    command="XDG_CONFIG_DIRS=${CHROOT}:$XDG_CONFIG_DIRS {chroot}/sass $@",
    workdir="/",
    execution_dependencies=[":extracted-sass"],
)

❓*pants rebuild everything every time*❓Hi community, I understand pants supports caching and should not rebuild things if unchanged. However, for my pex_binary, even if I just changed one file (for example, the main.py file),

pants run my_pex_binary_target

will rebuild from scratch. Could someone suggest how to troubleshoot?

Has anyone seen errors like this when running `pants check ::`:

Completed: Typecheck using MyPy - mypy - mypy failed (exit code 2).
mypy: Invalid python executable './requirements_venv.pex_bin_python_shim.sh': Exec format error

Hey! Question re caches and ec2 AMI's. We've created an AMI with a "warmed up" pants project - specifically for some target. i.e. we ran

pants run my_target

, and then created the AMI with the relevant caches. However, when we create a new machine and run the same

pants run my_target

it still spends a lot of time on building the requirements. Anyone knows how to improve this? (we have huge requirements, and this stage can take up to 20 mins)

Hi all! We're in the process of replacing flake8,isort,black etc. with ruff and have now come to the point where we want to replace

isort

but are experiencing issues when doing so. As many before me have reported there seems to be confusion with what packages are 1st and 3rd party. I've tried many of the proposed solutions here and in the issue tracker by e.g. setting

src = ["src/*"]

but that seems to not be able to identify all 1st party packages. The only solution that seems to do exactly what is expected is to list all packages manually under:

[tool.ruff.lint.isort]
known-first-party = [
  "my_package1",
  "my_package2",
  "my_packageN",
]

But surely that can't be the solution can it? Is there anything else I've missed that I can try out or is this still an open issue?

So I'm attemping to set up remote caching for pants with Buildbarn and it works great locally, but I'm seeing a ton of tcp timeouts when running from our self-hosted github runners. I'm pretty far out of my depth attempting to debug this, does anyone have any ideas on what could cause these? I've also proposed this question on the Buildbarn slack and they seem confident that it's a networking config somewhere on my end but I don't even really know where to start trying to figure out the issue.

Another

module_mapping

issue. I'm trying to use the

pytest-json-report

package which has a nonstandard import format, but even

module_mapping

seems insufficient to infer deps. I have to explicitly include the dep as a dependency.

Question on PEX binaries. How do arbitrary file dependencies get included into the Pex binary or sandbox if at all? Context: We have a bit of a nonstandard setup for creating a Pex test runner which involves invoking

pytest.main('./tests')

where it's located under

dir/

from the repo root. I tried to get this to work via

pants run dir:runner-target

but it doesn't work because pytest can't find that directory.

cd dir && pants run :runner-target

also doesn't work. So I included

files(sources=["**"])

as a dependency to the Pex target.

pants dependencies dir/tests_runner.py

shows that all files targets are includes as deps. But running the Pex still doesn't work in any case. So a ran with

--preserve-sandboxes=always

and went inside the sandbox. I see

dir/test_runner.py

instead, but nothing else in the directory.

How should I be handling package versions? In the process of migrating some code to Pants as a proof of concept. What I've done up till now -- before ever running

pants package

-- is to add a text

resource

that has a version string inside it. (It's not something that I want to do long term, but it has gotten me this far...). Today I'm looking at the

pants package

goal and have added this to the build file:

python_sources(
    dependencies=[":version"]
)

resource(
    # Adding a version file as a package resource
    name="version",
    source="VERSION"
)

python_distribution(
    name="dist",
    ...
    provides=python_artifact(
        ...
        # Adding a version here too, as required by `provides=`?
        version="1.2.3+what",
    ),
)

So now I have two ways of declaring a package version. And two ways of inspecting the version at runtime. In the first version, I am able do

version = pkgutil.get_data(__name__, "VERSION").decode().strip()

from the python runtime to get the package version using the text file. On the other hand, this is not guaranteed to coincide with the package metadata's notion of a version, which is given by the

provides

argument of

python_distribution

Which versioning mechanism is correct? Also, what alternatives exist? As for why we want to inspect the package version at runtime: Some of our CLI tools provide a --version option, and to accomplish this we inspect the package version and echo that back to the user in the terminal.

Hello there. I'm getting an unexpected shebang for generated Pex binaries. My global interpreter constraints specify 3.11.x, and for debugging purposes I added it to the target as well. But actual pex when run (as well as shebang at top) suggests Python 3.9

❯ dist/dir/tests_runner.pex
env: python3.9: No such file or directory

Can try to come up with a repro if needed. Docs and intuition suggest this should respect interpreter constraints so I'm fairly surprised.

Hey, I'm currently grappling with a challenge involving Terraform modules. My objective is to effectively utilize these modules across various deployments located in different places. However, I'm uncertain about the placement of the 'tvars' file. Should it be located within a subpath relative to the Terraform files? I'm considering altering the folder structure to accommodate this, but I'm also open to other potential solutions. What would you recommend in this scenario? Are there any practical workarounds or best practices I should consider?

I think I have a problem with my python lockfile and CI due to the fact I am building some wheels locally do deal with different platforms. In my

pants.toml

file I have the following:

find_links = [
    "file://%(buildroot)s/wheels"
]

This then results in urls in the lockfile with

"url": "file:///Users/scottmelhop/...../wheels/...

Obviously when I push to CI I get errors. Any ideas how to get around this without having to generate lockfiles each time I run CI?

Hi, I'm trying to run Streamlit in Docker. To run the Streamlit app locally, I use this pex binary configuration:

pex_binary(
    name="bin",
    script="streamlit",
    args=["run", "projects/test/main.py", "--server.port", "8501", "--global.developmentMode", "false"],
    dependencies=[":source"],
    execution_mode="venv",
    restartable=True,
)

However, because the path in the run command points to the local filesystem this does not work when I use the pex in a Docker container. To make it work in the container, I have to change the args to point to the main file in the python env site packages like so:

args=["run", "lib/python3.11/site-packages/test/main.py", "--server.port", "8501", "--global.developmentMode", "false", "--server.address", "0.0.0.0"]

Is there a better way to accomplish this? I imagine people using gunicorn or something similar where a python file needs to be passed as an argument to the actual program would run into the same issue?

Hello 👋 I have question regarding publishing python packages to a private pypi repository. In CI/CD, the URL for this pypi repository and the basic auth credentials to access it are stored in environment variables. Conventionally, without

pants

, we publish packges using

twine

(which I believe

pants publish

uses as well), which uses the environment variables

TWINE_REPOSITORY_URL

,

TWINE_USERNAME

,

TWINE_PASSWORD

. When using

pants

, it’s not possible to use environment variables in the repositories parameter of python_distribution, so instead we dynamically generate a

~/.pypirc

file with values pulled from environment variables. In the

.pypirc

file, we defined an index alias

[distutils]
index-servers = artifacts

[artifactory]
username = xxxx
password = xxxx
repository = <https://artifacts.our-host.com/some/path/pypi/pypi-local>

Therefore, in the

python_distribution

target,

repositories=["@artifacts"]

. With this setup

pants publish

publishes packages to our private pypi repository as expected. Recently a bug in CI stopped generating this

.pypirc

file. Now there is no

.pypirc

file, but the

TWINE_*

environment variables are still defined, but somehow

pants publish

still works? If there is no

.pypirc

file, then how does

pants

know what

@artifacts

is? Is

twine

defaulting to the environment variables? If that is the case, then could any dummy alias be set for

repositories

- e.g.

repositories=["@null"]

(

pants publish

will skip a package entirely if

repositories

is not set at all)

What kind of caching can pants do for docker images? I have a docker image that takes a long time to build but doesn't change that often. I had expected pants could just pull it out of the remote cache once I had set that up but that does not appear to be the case?

I was migrating from 2.17 to 2.18 and stumbled upon export --style=universal incompatibility issue with

torch

like documented and discussed many times before. Is there any reason not to allow --style configurable for generating lockfiles here https://github.com/pantsbuild/pants/blob/6c90b26f62c108d5a8724cd92db3b69887f8f431/src/python/pants/backend/python/goals/lockfile.py#L120 ? Maybe we can drop target-system advisory if the style is strict, assuming the developer knows what they are doing generating machine-specific lock file?

👖 Just FYI I’d like to advertise our #announce channel, it is low frequency high value to follow that to get notified about new releases (mostly) and other notable information. 💡

Okay there's something I'm not understanding about

pants repl

. Given a repo that has multiple projects, what addresses are valid to use with

pants repl

?

src
└── python
    ├── BUILD
    ├── project1
    ├── project2
    ├── project3
    .
    .
    .

pants repl src/python/project1

works as expected. However if

project1

and

project2

both have a common third party dependency, then I see Pex `ResolveError`'s. So, apparently it's not okay to do this. Which is a bummer because I wanted to

pants repl ::

. What should I be doing instead? This is sort of a hypothetical... Not sure who would actually want to open a REPL with the entire universe in it, but someone on my team might ask to do this and I want to know what the story is 😄

Another question: How are people using Pants with an IDE for Python projects? In particular, how do you tell VSCode where to find the right Python interpreter? What I've done:

pants export --resolve=default

and then in VSCode I find that venv with "> python: Select interpreter". However, this doesn't install first-party code into the environment. Without first-party code installed in "editiable" mode, I can't have VSCode launch and debug a Python module, as none of my first-party code is in the venv.

remote cache question - I've setup buildbarn remote cache and connected pants to it. It works mostly fine, however I've noticed that some of the bigger pants artifacts (like pexes with multiple requirements) fails to be cached. Did someone face the same issue?

I've been toying with ways to take a pex lockfile and visualize it. I don't have a super concrete usecase here, but like "trying to get a sense of things" or "see if something looks fishey". The graph is big enough that I think I'm going to end up wanting some amount of interactivity (ex: click a node and highlight it's connections, search by id). Has anyone walked down this road before?

Hi! I had the impression that the adhoc_tool target is runnable, but it looks like it is not. How would I test it in isolation if "pants run" is not applicable?

Hi! I'm having issues for testing (running

pants test

). When I run pants test , the files in the folder I'm adding the tests are being imported automatically even if my test file is empty. Is this common behavior? Is there a way to tweak it? My BUILD file in the folder:

python_tests(name="tests")
__defaults__({"python_tests": {"sources": ["**/test_*.py"]}})

What is a scenario where [GLOBAL].build_ignore would be used instead of [GLOBAL].pants_ignore?

Hi! Every time someone calls

pants run

, I want do add flag:

--filter-address-regex='.*@resolve=XXX'

. I imagine there's some way to do this in

pants.toml

? In particular, can

XXX

be an environment variable? Essentially, I want to specify that when I call

pants run path/to/my:file

it should call

pants run path/to/my:file@resolve=XXX

, were

XXX

is set according to some environment variable (or some other mechanism)

is Pants cache shared between different Pants versions? I.e. on Pants upgrade, will all / any parts of the cache get invalidated?

Hi, I need some advice/best practices on how to work with resolves for shared code. In my repo, i have 40-50 files which form shared library, which till yet were using default resolve. Now i have 2 main files(2 pex_binary) which use 2 different resolve(say

ai-cpu

and

ai-gpu

). my resolves looks like:

python_requirements(
    name="reqs",
    resolve=parametrize("python-default", "ai-gpu", "ai-cpu"),
)
python_requirements(
    name="ai-gpu",
    source="ai-gpu-requirements.txt",
    resolve="ai-gpu",
)

python_requirements(
    name="ai-cpu",
    source="ai-cpu-requirements.txt",
    resolve="ai-cpu",
)

Based on docs,

If a first-party target is compatible with multiple resolves, e.g., shared utility code, you can use the parametrize mechanism with the resolve= field.

i should use `python_sources(resolve=parametrize('ai-gpu', 'ai-cpu', 'python-default'). Given my shared code is spread across multiple directories, it will get unmaintainable to always set these resolves for all the targets. So i am thinking to set the defaults at root of python source code, like:

__defaults__(
    {(python_sources, python_source): dict(resolve=parametrize(python-default", "ai-gpu", "ai-cpu) )}
)

and for the 2 main files(pex_binary targets) that i am interested it, i will set the resolve

ai-cpu

,

ai-gpu

. Do you think this is a good idea? Should

__defaults__

be used/exploited like this, does it form a good practice, or is there a better way of doing this?

Hi I've added

celery[sqs]

and

celery[redis]

package in poetry (

pyproject.toml

) config. I can able to generate

python_default.lock

file. However I'm not sure how to add this extras as dependancy in BUILD file. Can someone please guide me how to configure this?

# pyproject.toml
[tool.poetry.group.pkg-celery_worker.dependencies]
celery = {extras = ["redis", "sqs"], version = "^5.3.6"}

# BUILD file
python_sources(
    name="pkg",
    sources=["celery/**/*.py"],
    dependencies=[
        "3rdparty/python:poetry#boto3",
        "3rdparty/python:poetry#celery",
        "3rdparty/python:poetry#redis",
    ],
)