https://2022.pycon.kr/en/program/talks/25 Here is my talk intro about migration of Backend.AI to mono-repo using Pantsbuild in Korean, for PyCon KR. On Oct 15th, I will also give the same talk in English in PyCon JP.

Hello. I have a weird behavior and I suspect some caching somewhere but that's just a feeling. We're building container images using podman, a rootless replacement for docker. It emulates the docker cli so nothing should change besides one exception, it needs to executables named newuidmap and newgidmap. Both are on /usr/bin. When executing

./pants package ::

, the build fails saying it can't find the executable 'newuidmap' on $PATH. I tried to add the following to the pants.toml file:

[docker]
tools = [
    "newuidmap",
    "newgidmap"
]

and sometimes it works, sometimes it doesn't. When it starts to work, I can remove the section and it continues to work. Sometimes the problem comes back. I'm not sure how to reproduce that easily and I can't guarantee that nothing changed in my environment. I tried removing the caches, but it doesn't seem to have an effect. Any idea on what's going on or what I should do to dig deeper?

Hi, I am having trouble running tests that use local files. What is the correct way to include files as resources for testing? My directory layout is like this • pants.toml • pants • service ◦ BUILD ◦ examples ▪︎ data • data.csv ◦ sub_dir_1 ▪︎ BUILD ▪︎ <other files> ◦ test ▪︎ sub_dir_2 • BUILD • test_service.py service/BUILD has this:

resources (
    name="test-data",
    sources=["./examples/data/*"],
)

pex_binary(
    name="service",
    entry_point="service.py",
    platforms=[
        "current",
        "manylinux2014-x86_64-cp-39-cp39",
    ],
    dependencies=[":test-data"],
)

test/sub_dir_2/BUILD has this:

pex_binary(
    name="test_service",
    entry_point="test_service.py",
    dependencies=["service:test-data"],
)

I am trying to package everything as a pex file. A test in test_service.py uses the files as such:

path = str(pathlib.Path(__file__, "../../../examples/data/data.csv").resolve())
do_something_with(path)

Checking the sandbox, indeed, the files don't exist(actually no resources exist). However, I do see the files in the packaged .pex archives. I am trying to run the test as such

./pants --keep-sandboxes=on_failure test service/test/sub_dir_2/test_service.py -- -k test_csv

I am using pants 2.13.0. I would really appreciate any pointers on how to go about including files in tests. TIA, CSN

I'm interested in working on this and getting familiar with the mechanics of dependency inferrence. I'm not sure where/how to hook into this.

@careful-address-89803 I love the idea you propose, and that would be awesome to have your help! How do you feel about what the next steps are, based on our replies? I'm happy to write out more clear tips if that would be useful

Do we see collecting the existing debug goals into a common goal to be a prerequisite step? and then we can plug python into that?

I don't think it needs to be a prereq step. I think JVM users often != Go users != Python users, no need to bundle them together

I think I've asked this before, but when (if at all) does pants determine a lockfile needs to be updated?

Separate question related to the thread above: is it possible to easily override the help message that comes out when an invalid lockfile error is raised?

generate-lockfiles

lets you supply a

--custom-command

to appear in the lockfile header. I'd like developers to use that. However, if they see an invalid lockfile error, they are presented with a command that I don't want them to run.

I'm trying to set up multiple resolves where the only "real" difference is the set of

--extra-index-urls

each uses (because pytorch has CUDA and non-CUDA libs which are differentiated only by the index url). But it appears that

python-repos.indexes

is a global property and can't be set per-resolve. Is there a way to do this? (see https://pantsbuild.slack.com/archives/C046T6T9U/p1664233267501069 for some more context if necessary)

What's the correct syntax for using a local package that has a

setup.py

? The use case here is moving code to a monorepo, and trying to use code that was previously installed with a

pip install

. From some old Github conversations, I believe this should be possible, but I couldn't find documentation for how to set things up. Details in 🧵.

Hi, with pants 2.14, I see lot of warnings like. I suppose this is because of namespace packages. What should I do to handle such warnings?

15:49:36.77 [WARN] Pants cannot infer owners for the following imports in the target src/package-local/tests/utils/test_nlp.py:

  * tests.resources.fake (line: 5)

In this case

src/package-local

has structure like. We have multiple of such local packages each with their own tests.

src/package-local
- package_local
- tests

What tool does

export

use to make a virtual environment?

I have left my computer running for a few hours and when I got back, I could no longer run pants, because it said "too many open files". I killed the pantsd process, and then it started working again. I guess that file descriptors are leaking somewhere. Is this a known issue?

Hello, I am learning Pants, and I have an issue running a simple test because it fails to import the module. Here is the repository: https://github.com/pmuller/pants-example-monorepo And this is the error I get:

$ ./pants test ::
00:31:26.68 [ERROR] Completed: Run Pytest - packages/libhello/tests/test_hello.py failed (exit code 2).
============================= test session starts ==============================
platform linux -- Python 3.9.14, pytest-7.0.1, pluggy-1.0.0
rootdir: /tmp/pants-sandbox-8eRczK
plugins: cov-3.0.0
collected 0 items / 1 error

==================================== ERRORS ====================================
____________ ERROR collecting packages/libhello/tests/test_hello.py ____________
ImportError while importing test module '/tmp/pants-sandbox-8eRczK/packages/libhello/tests/test_hello.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
packages/libhello/tests/test_hello.py:1: in <module>
    from libhello import hello
E   ModuleNotFoundError: No module named 'libhello'
- generated xml file: /tmp/pants-sandbox-8eRczK/packages.libhello.tests.test_hello.py.xml -
=========================== short test summary info ============================
ERROR packages/libhello/tests/test_hello.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.06s ===============================

What am I getting wrong?

I'm using 2.13 and seeing weird behavior in

.cache/pants/setup

. https://www.pantsbuild.org/docs/using-pants-in-ci recommends a simple script for nuking cache directories when too large. It recommends

pants/setup

as one of the directories to monitor. When we do that for our builds when they kick off, we start seeing errors like this

/home/buildbot/.cache/pants/setup/bootstrap-Linux-x86_64/2.13.0_py38/bin/python: No such file or directory

Any recommendations on what to do?

Question about the use of

python_requirements

. I'm aware that you can create an arbitrarily large lockfile without having to worry about everything in the lockfile getting dragged in to each Python target. But do you have to also create separate

python_requirements

for each binary / library if you want to restrict the number of dependencies they pull in? Or can Pants still resolve which ones to pull in via import statements?

not sure if i’m doing something wrong or if this isn’t supported, but is it possible to add extras to a

python_requirement

target? e.g.

python_requirement(
    name="apache-airflow",
    requirements=[
        "apache-airflow[amazon]==2.2.2",
    ],
    modules=["airflow"],
)

this doesn’t seem to include the

amazon

extra (if i run pylint against code that uses this as a dependency, it fails to import

from amazon.providers.amazon

) however, including the extra as a separate package seems to work, e.g.

python_requirement(
    name="apache-airflow",
    requirements=[
        "apache-airflow==2.2.2",
        "apache-airflow-providers-amazon==2.4.0",
    ],
    modules=["airflow"],
)

the downside is that i have to manually specify the compatible version of the package, which i’d like to avoid

Not sure how to resolve this issue, at least until the ansible plugin is ready. Using ansible to deploy a bunch of pexes. After packaging, they end up in

dist/...

- however, to deploy them from my ansible role, I need them in specific locations. I haven't been able to resolve symlinks (it just uploads the symlink to the target). Is there a Pantsian way to manipulate the host filesystem?

Is it possible to change the path of the virtualenv generated by "pants export ::"? I'd like its path to be fully generic (i.e. not include Python version) so I can include a generic VSCode configuration in the repository, making it easier for other people to setup their environment.

When I run

./pants lint ::

everything is fine. But when I run pylint through VSCode using

python.linting.pylintPath=./dist/export/python/virtualenvs/tools/pylint/bin/pylint

, I get pylint's

impor-error

because the

tools/pylint

apparently doesn't understand my imports, despite

PYTHONPATH

being defined in

.env

. BTW, the

.env

looks fine as VSCode itself has no issue resolving those imports. Any idea how I could troubleshoot this further?

pybind11 + pex + docker query:

I tried to adopt Pants for plugins implemented in Python for GNU TeXmacs: https://github.com/texmacs/plugins-in-python/pull/6 For example, using Pants and PEX, the installation of a SymPy plugin of GNU TeXmacs is much more easier! Thanks! Pants is an awesome tool!

TL;DR: Too many

BUILD

files. Hi, I’m trying to setup Pantsbuild (2.13) for the first time in an existing Python monorepo. At the root of the repo, I’ve added an empty

BUILD_ROOT

file and a

pants.toml

file with:

[GLOBAL]
backend_packages = ["pants.backend.python"]

[source]
root_patterns = ["/X_*/"] # "X" is the name of the company and "X_" is the prefix of each project

There are about 50 projects in the repo. I’ve run

pants tailor ::

and then

git status | wc -l

and got 644. Do I really need to commit 640

BUILD

files? Is there any way around it? (maybe creating one per project?) Thanks in advance 🙂

with the new

run_goal_use_sandbox

is there a way to make

restartable

usable too? atm it only works on

pex_binary

but thats slowing down the dev iterations massively.

im pretty sure im running in the wrong direction, again :) i try to automate a deployment with ansible-runner. so i wrote an

deployments/project_name/main.py

script and made it invokable with

python_source

. i'm able to run it with

./pants run deployments/project_name

with and without the sandboxing feature. pretty nice but ansible-runner wants to call

ansible-playbook

via subprocess. i tried to include it as a requirement in my

python_source

but without any luck. the next try was: make a fake

ansible-playbook

executable inside my

~/.local/bin folder

. result: my

deployments/project_name/main.py

is able to find and execute it. is it the right solution to globally install ansible so it's findable by my

deployments/project_name/main.py

script or is there something i could do to include it into the path to be able to rely on the automatic fetching of ansible?

is there a more succinct way to glob all files that are not Python modules than getting all files first and then negating

resources(
    name="testdata",
    sources=["tests/testdata/**/*", "!tests/testdata/**/*.py"],
)

?

We're seeing very slow docker builds with pants, and was wondering if we're doing something obviously bad:

19:41:23.94 [INFO] Long running tasks:
  643.75s	Building docker image <http://amramedical.com/client-session-tagger:latest|amramedical.com/client-session-tagger:latest> +2 additional tags.

These are small services, with 4 requirements (largest being boto3). The time spent is mostly focused on building "requirements" pexes, meaning

pex_binary

targets in the container with only requirements included. We use a lockfile, with_tools=True and layout="packed". We use only wheels.

#13 [source 4/4] RUN PEX_TOOLS=1 /usr/local/bin/python client-session-tagger-handler.pex venv --compile /app
#13 sha256:3dcefd9a9645feb44367da0c45b3370685d8cc30427c1a07e8d99fe9111ab559
#13 DONE 3.7s
#9 [dependencies 4/4] RUN PEX_TOOLS=1 /usr/local/bin/python client-session-tagger-requirements.pex venv     --scope=deps     --compile /app
#9 sha256:ef99e535e560ee6d3decd7805cb3c81c0d17b05123d0f2b0505b157e6efb1697
#9 DONE 637.8s

Hi there, I'm trying to use the

flake8-requirements

plugin. Since my

requirements.txt

is not in the project's root directory, I need to pass the

--requirements-file

option pointing to the requirements in

3rdparty/python

. So I added this to `[flake8].args`:

[flake8]
args = [
   ...,
    "--requirements-file=3rdparty/python/requirements.txt",
]

However, the plugin doesn't seem to be able to read the text file. I suspect an issue with the sandbox. Any ideas?