Your process is free to write wherever in the sandbox

but an output directory (say literally named

output

) should be empty simply because the sandbox starts out empty, and is populated with just the input digest

Is there any way to get more structured output from commands like

dependees

? I am trying to get a map of dependees for a bunch of files and am either faced with running through a loop and calling dependees for each (slow) or using xargs to

./pants

and calling dependees once, losing track of which target the dependee came from. Ideally I could call

dependees

(and maybe other commands like it) once with a bunch of targets and get something like a json map of which target has which dependee.

Hey Pants folks! I’m trying to reproduce the documentation of using AWS Lambda and Docker with Pants, but I’m not able to get it running. When I try running my function I get:

{
  "errorMessage": "the 'package' argument is required to perform a relative import for '.app.lambdex_handler'",
  "errorType": "TypeError",
  "requestId": "",
  "stackTrace": [
    "  File \"/var/lang/lib/python3.9/importlib/__init__.py\", line 122, in import_module\n    raise TypeError(msg.format(name))\n"
  ]
}

I have re-created the repo based on the docs here: https://github.com/nharada1/lambda-aws-pants-demo My only changes are I’m using Python 3.9 instead of 3.8

Hi, Although, after add package into requirments.txt and create lockfile and package, file sill cannot see the package. Where do I put more code? I am using example-django version 1. requirement.txt

django-ninja>=0.19.1

2. lockfile

// --- BEGIN PANTS LOCKFILE METADATA: DO NOT EDIT OR REMOVE ---
// {
//   "version": 2,
//   "valid_for_interpreter_constraints": [
//     "CPython>=3.7"
//   ],
//   "generated_with_requirements": [
//     "ansicolors>=1.0.2",
//     "django-ninja>=0.19.1",
//     "django-stubs>=1.10.0",
//     "django<4,>=3.2.13",
//     "gunicorn>=20.1.0",
//     "protobuf>=3.11.3",
//     "pytest>=6.0.1",
//     "requests>=2.25.1",
//     "setuptools>=42.0.0",
//     "translate>=3.2.1",
//     "types-requests>=2.25.1"
//   ]
// }

3. BUILD

# Copyright 2021 Pants project contributors.
# Licensed under the Apache License, Version 2.0 (see LICENSE).

python_sources(
    name="lib",
    dependencies=[
        "helloworld/person/migrations",
    ]
)

python_tests(
    name="tests",
    dependencies=[
        "helloworld",  # For settings.py.
    ],
)

python_test_utils(
    name="test_utils",
)

4. api.py

from ninja import NinjaAPI

api = NinjaAPI()

@api.get("/add")
def add(request, a:int, b:int):
    return {"result": a + b}

Hello. I've observed some unexpected behaviour around task cancellation/retries and I'm trying to work out if I've misconfigured pants or not. In particular: I'm noticing that any file system change will cause any running tasks to be retried, even if the file system change doesn't affect the running tasks. For instance, the task might be a long-running test or a server (i.e. runs forever), and the change might be: • touching any file without changing contents (equivalent of

touch path/to/file

) • changing a file that the running task(s) do not (transitively) depend on For either of these, I'd expect the running tasks to keep running, and it's annoying that fiddling on some unrelated part of a codebase forces things to restart. Is this the expected behaviour?

Thanks, for the explanation John.

Hello. I have the following issue: I have a monorepo where the pex_binary includes a manage.py (django) and includes a dependency in the same path. Since the dependency includes the absolute path to this dependency, the pex has a different "root" path. Hence the dependency cannot be found, since the path to this dependency starts at root and not in the same folder as the manage.py (I hope that makes sense what I am saying). Is there a way to tell the pex binary and its dependencies to only look at the dependencies from this relative place? Otherwise I get the error:

Note that only Django core commands are listed as settings are not properly configured (error: No module named 'test_config').

Even though

./pants depedencies ::

shows that test_config is included, just as

backend/test_config/

.

Hello again - One more question. In django, everything seems to be correctly set up, when running shell (e.g.

./pants run backend/modules/mod1:manage -- shell

). However when I do runserver (e.g.

./pants run backend/modules/mod1:manage -- runserver

) it complains that django is not installed. (No module found...) Any idea why? it seems to disregard all installed side-packages...

Hi, as a suggestion, it could be nice if running

./pants check ::

would list any dangling targets, ie targets that aren't used anywhere because they are misconfigured.

Do I have to use

BUILD

files in all subdirectories that I need or can I also just aggregate all files with:

python_sources(
   name="lib",
   sources=["folder/**/*.py"],
)

When I do this, I get the error

ImportError: No module named manage

, but I am not sure if this is because this is not allowed or because I am doing something else that is wrong.

I've been playing with remote cache for the last few days and I have two thoughts: 1. Some processes should not be remote cached? like finding binaries on the running machine? maybe new flag

remote_cache

which default to true and false on some very light processes? 2. Should some process not speculate (local process race with remote cache) because they are too heavy to start? I'm thinking mainly nailgun, java processes or tests.

My colleague runs into this problem. Anyone knows how to resolve this? It's Ubuntu 20.04. I've asked him to add these lines in

pants.toml

but that doesn't help:

[python-bootstrap]
search_path = ["/usr/bin/python3"]

I have a python binary that uses GDAL to do some image processing. In the past, we've lived with using GDAL 2.4.0 but we are now trying to use GDAL 3.3.2. The GDAL package needs to be installed from source as the docker images we use only support up to 2.4 unless you install yourself. If I do a ./pants repl on the binary target, I can import gdal just fine and it reports version 3.3.2, however if I try and run the PEX, python says it can't find GDAL 2.4.0. Is there anything special I need to do to tell pants that I installed GDAL manually? My BUILD file references 3.3.2 which should match what's installed in the container

What is the right way to include local wheel files? I have a project where I want to package a local wheel file (transitive dependency) with the pex_binary. I was going about it this way.

python_requirement(
    name="client-lib",
    requirements=["client-lib.whl"],
)
pex_binary(
    name="service",
    entry_point="service.py",
    platforms=[
        'current',
        'manylinux2014-x86_64-cp-39-cp39',
        'macosx-12.0-x86_64-cp-39-cp39',
  ],
  dependencies=[":client-lib"],
)
This errors out with `Requirement 'client-lib....whl' looks like a filename, but the file does not exist`. 
1. Is this the right way to package a local wheel into a pex_binary
2. If so, what is the right way to specify the path?
3. If not, would love some help on how to go about doing this.
TIA

[mypy].extra_type_stubs

doesn't seem to contribute to any lockfiles. Is this intentional? (2.12.0)

I'm writing a python "pipeline" script that stitches together several other python scripts. What's the correct way to do that in pants? (I've tried to call

./pants run

with python

subprocess

, but it doesn't allow me to do that since a pants process is already running. I've also tried to

subprocess.run

python (

sys.executable

) my scripts but it looks like that doesn't pick up the dependencies correctly (fail to import third party packages))

Hi everyone, I’m used to using GCP’s dataflow for data workflows, but a project I’m working on is using AWS instead. I think ultimately I’d like to use EMR and PySpark for this, but there’s a lot of moving pieces with EMR, Spark, Docker, Pants/Pex, etc. What’s the best way to do this at a high level? I’m guessing there’s something with

./pants package

that lets me bundle everything together, and then use EMR’s Docker support to run my code remotely? I’m not sure how to create an environment for the workers, although I assume there’s a way to install the Pants repo into the docker such that it’s available for the workers? I think if I wasn’t using Pants, what I would do is setup a docker container such that my repo has been

pip install

’d, that way all of the imports work for the Python.

I'm having issues with how

pants repl

interacts with distribution targets. I have a sample library that depends on

pyyaml

. That library has a target named

lib

which is just a

python_sources

goal and a target named

example_lib

which is a

python_distribution

target. If I do:

./pants repl src/example_lib/:lib

everything works fine. But suppose I want a repl with all the libs in my monorepo available. The obvious way to do that would be

./pants repl ::

but that gives me:

$ ./pants repl ::
Traceback (most recent call last):
  File "/home/oliver/.pyenv/versions/3.10.0a7/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/oliver/.pyenv/versions/3.10.0a7/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/oliver/.cache/pants/named_caches/pex_root/unzipped_pexes/aa78b3981cc88964a18a80d8fa2d708ffe5155b2/__main__.py", line 103, in <module>
    bootstrap_pex(__entry_point__, execute=__execute__, venv_dir=__venv_dir__)
  File "/home/oliver/.cache/pants/named_caches/pex_root/unzipped_pexes/aa78b3981cc88964a18a80d8fa2d708ffe5155b2/.bootstrap/pex/pex_bootstrapper.py", line 601, in bootstrap_pex
    pex.PEX(entry_point).execute()
  File "/home/oliver/.cache/pants/named_caches/pex_root/unzipped_pexes/aa78b3981cc88964a18a80d8fa2d708ffe5155b2/.bootstrap/pex/pex.py", line 519, in execute
    self.activate()
  File "/home/oliver/.cache/pants/named_caches/pex_root/unzipped_pexes/aa78b3981cc88964a18a80d8fa2d708ffe5155b2/.bootstrap/pex/pex.py", line 156, in activate
    self._activated_dists = self._activate()
  File "/home/oliver/.cache/pants/named_caches/pex_root/unzipped_pexes/aa78b3981cc88964a18a80d8fa2d708ffe5155b2/.bootstrap/pex/pex.py", line 143, in _activate
    activated_dists.extend(env.activate())
  File "/home/oliver/.cache/pants/named_caches/pex_root/unzipped_pexes/aa78b3981cc88964a18a80d8fa2d708ffe5155b2/.bootstrap/pex/environment.py", line 321, in activate
    self._activated_dists = self._activate()
  File "/home/oliver/.cache/pants/named_caches/pex_root/unzipped_pexes/aa78b3981cc88964a18a80d8fa2d708ffe5155b2/.bootstrap/pex/environment.py", line 671, in _activate
    resolved = self.resolve()
  File "/home/oliver/.cache/pants/named_caches/pex_root/unzipped_pexes/aa78b3981cc88964a18a80d8fa2d708ffe5155b2/.bootstrap/pex/environment.py", line 502, in resolve
    for fingerprinted_distribution in self.resolve_dists(all_reqs)
  File "/home/oliver/.cache/pants/named_caches/pex_root/unzipped_pexes/aa78b3981cc88964a18a80d8fa2d708ffe5155b2/.bootstrap/pex/environment.py", line 589, in resolve_dists
    raise ResolveError(
pex.environment.ResolveError: Failed to resolve requirements from PEX environment @ /home/oliver/.cache/pants/named_caches/pex_root/unzipped_pexes/452071acbd3c0c2b132caf245582cb4d49e6d821.
Needed cp310-cp310-manylinux_2_31_x86_64 compatible dependencies for:
 1: pyyaml
    Required by:
      example-lib 1.0.0
    But this pex had no ProjectName(raw='pyyaml', normalized='pyyaml') distributions.
 2: types-PyYAML
    Required by:
      example-lib 1.0.0
    But this pex had no ProjectName(raw='types-PyYAML', normalized='types-pyyaml') distributions.

Note that the actual built distribution has its dependencies specified correctly:

$ tar -O -zxvf dist/example_lib-1.0.0.tar.gz example_lib-1.0.0/setup.py
example_lib-1.0.0/setup.py

# DO NOT EDIT THIS FILE -- AUTOGENERATED BY PANTS
# Target: src/example_lib:example_lib

from setuptools import setup

setup(**{
    'install_requires': (
        'pyyaml',
        'types-PyYAML',
    ),
    'name': 'example_lib',
    'namespace_packages': (
    ),
    'package_data': {
    },
    'packages': (
        'example_lib',
    ),
    'python_requires': '==3.10.*',
    'version': '1.0.0',
})

Is there no way to do a

pants repl

for all libraries in a repo that has one or more distribution targets?

Hi - I'm running a Django application in Pants and I'm trying to understand the best way to include non-py files when creating a PEX binary. I have the following in my BUILD file:

python_sources(
    name="lib",
    dependencies=[
        "//:root#auth0-python",
        "//:root#django-admin-rangefilter",
        ...,
    ],
)

pex_binary(
    name="manage",
    entry_point="manage.py",
    dependencies=[
        ":lib",
    ],
    restartable=True,
)

My Django application has various HTML files, SQL files, and other non-Python files that are required for it to function. What's the typical pattern to make sure that these files are present in the PEX when running

./pants package src/myapp/manage.py

?

has anybody had any issues with grpcio as of late? Been chasing build issues due to an unrelated change. But under the aws/codebuild/amazonlinux2-standard:3 image

./pants package something/that/uses:grpcio

works. but under our "other" image we're being forced to use, something gets upset and tries to compile grpcio, and subsequently fails. Yet when I run

pip3 install grpcio

it seems to download a compiled version.

I am trying to write a "rule" (I assume I am on the correct path) so learn about plugins with the ultimate aim of writing some nice versioning backend logic that brings in many different versioning strategies (semver, calver, gitcommish, conventional-commits-semver) To get there I want to build up a few practical examples: The first place I wanted to understand is how plugins works and specifically, want to just tag a docker image.

docker_image(
    name="docker_image",
    repository="my-image-name",
    image_tags=[the_version()],
    dependencies=[":the_python_pkg", ":docker_resources"],
)

where

the_version()

is coming from a rule generated in a plugin This is the "test plugin" i have written:

# pants-plugin/versioning/register.py
import os
from pants.engine.rules import collect_rules, rule

@rule
async def the_version() -> str:
    return os.popen('git --no-pager describe --always --dirty --broken').read()

def rules():
    return collect_rules()

However when

./pants package path/to:docker

is run I am expecting the docker_image tag to pickup the "string" from the

the_version()

however pant` does not register the rule as a symbol ? (maybe it's not a rule I need)

MappingError: Failed to parse ./services/my-image/BUILD:
Name 'the_version' is not defined.

If you expect to see more symbols activated in the below list, refer to <https://www.pantsbuild.org/v2.13/docs/enabling-backends> for all available backends to activate.

All registered symbols: ['_generator_sources_helper', 'archive', 'build_file_dir', 'docker_image', 'experimental_run_shell_command', 'experimental_shell_command', 'file', 'files', 'helm_artifact', 'helm_chart', 'helm_unittest_test', 'helm_unittest_tests', 'http_source', 'parametrize', 'pex_binaries', 'pex_binary', 'pipenv_requirements', 'poetry_requirements', 'python_artifact', 'python_distribution', 'python_requirement', 'python_requirements', 'python_source', 'python_sources', 'python_test', 'python_test_utils', 'python_tests', 'relocated_files', 'resource', 'resources', 'setup_py', 'shell_source', 'shell_sources', 'shunit2_test', 'shunit2_tests', 'target']

My goal for this first step is to get

pants

to "auto collect" the "GIT_COMMIT" version ref: https://www.pantsbuild.org/v2.11/docs/tagging-docker-images#using-env-vars-to-include-dynamic-data-in-tags and provide it as a variable or via a function (like I tried above). Such that

./pants package src/example:demo

Builds a docker image with

demo:$(git rev-parse HEAD)

Is it possible to set a PEX

--runtime-pex-root

for a

pex_binary

target? I don’t see any mention of this option on the docs page. On most hardened Linux systems (eg US STIG) the home dir is mounted non-executable (noexec) which prevents any wheels with .so libraries from executing.

I have everything set up with pants and it works like a charm - Thanks for the help @curved-television-6568 and @happy-kitchen-89482. One last question remains to me: First party dependency which are packaged separately. My currently workflow is to have a poetry pyproject.toml to specify the first party dependency (and to override it in the dev dependencies with an editable install). Pants doesn't understand editable installs, therefore I also include the dependency in my BUILD file, which works as intended. (The dependency + editable install I still have from the days before pants). If I now what to package all my modules and libraries I need to keep the dependency declared in my pyproject.toml, otherwise the generated setup.py in the distribution doesn't know anything about this dependency. If I don't declare the editable install in the dev dependency, pants generate lockfiles doesn't work anymore. (which makes sense). So if I keep the dependency in the BUILD file, as well as in the pyproject.toml + editable install as dev-dependency everything works - But it feels kind of like a workaround. My thought process was, that if libraryA declares a distribution in its BUILD file and moduleB depends on this libraryA, it should add it to the install_requires section of the setup.py (As mentioned here https://www.pantsbuild.org/docs/python-distributions#mapping-source-files-to-distributions). Or does this not work because of me using poetry?

is it a good idea (and is it possible?) to collect all files from dependencies and subdependencies into a digest? im writing a tailwind plugin and tailwinds needs all html files to be able to purge unneeded css rules.

It seems like a very basic question, but I can't find any way to override the

target-system

for lockfile generation. I am only interested in

linux

as a target system, but Pants seems to fail because it wants dependencies to be resolved for all platforms.

I posted this at the end of the day yesterday and I think maybe people didn't notice it as a result. Please excuse the re-post: https://pantsbuild.slack.com/archives/C046T6T9U/p1663632448606739

Is there any way to parametrize a pair of fields together? 🧵

Hi guys. we are on week 4 of trying to get our Django app to working pants. We are now trying to figure out how to get local development to work. At the moment, none of the

admin.py

files are getting pulled in (as well as other magic dependencies). Does anybody have any tips on how to solve this?

Hello. Does anyone know if there's a good way to modify Tailor's behavior across directories? I was trying to get the BUILD files to all look a certain way throughout specific project directory trees. Right now it looks like I need to modify the generated files by hand or add custom generation.