I'm trying to figure out how I should think about

python_library()

targets that

pants tailor

generates for me. In particular, it is generating them for "scripts" in our repo that we'd nominally intend not to be importing. Should I think about them as just "there's python code here pants needs to know about"? Or if there is code that we don't intend to import is there a better way to structure the targets?

Is there an equivalent of

::

that I can use in the

sources

field of a

files

target? I'm trying to package up an entire directory for use on spark. I know the better long term solution is to have specific targets for a given job, but I'd like to have something that just over-estimates to get something good enough without explicitly listing all the targets. We currently have a hack using a

files(sources=[""**/*.py""])

but that does not nice things to target discovery, so I'm hoping there is a better way to do this?

ignore-adding-targets doesn’t seem to work for

requirements.txt

. For example, in an extremely minimal repo:

# pants.toml
[GLOBAL]
pants_version = "2.8.0rc4"
backend_packages = [
  "pants.backend.python",
]

[tailor]
ignore_adding_targets = ["//:requirements.txt"]

reports

./pants tailor --check
Would update BUILD:
  - Add python_requirements target requirements.txt

if there is a

requirements.txt

in the same directory. (In the middle of incremental adoption of pants and we’ve got a bunch of

requirements.txt

spread around that I need to ignore because they have duplicated targets). Am I doing something silly wrong, or is there a bug here?

Hi pants devs. So I see the nuance for why

pex_binaries

don't include

files

targets, but converting all of our uses to be compatible with

resources

is a larger project. I'm playing with

docker_image

targets, but I have found that I can pretty easily get the execution env compatible with what I need by depending on the

files

targets I need from the

docker_image

and then using

COPY . /app

in my

Dockerfile

. If a

docker_image

target depends on

pex_binary

, could it then automatically also depend on

files

targets depended on by

python_library

in that

pex_binary

?

hello! We recently had a build failing on circleci with:

22:43:57.74 [INFO] Starting: installing node project dependencies
The pantsd process was killed during the run.

If this was not intentionally done by you, Pants may have been killed by the operating system due to memory overconsumption (i.e. OOM-killed). You can set the global option `--pantsd-max-memory-usage` to reduce Pantsd's memory consumption by retaining less in its in-memory cache (run `./pants help-advanced global`). You can also disable pantsd with the global option `--no-pantsd` to avoid persisting memory across Pants runs, although you will miss out on additional caching.

If neither of those help, please consider filing a GitHub issue or reaching out on Slack so that we can investigate the possible memory overconsumption (<https://www.pantsbuild.org/docs/getting-help>).
Traceback (most recent call last):
  File "/home/circleci/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/bin/pants_loader.py", line 95, in run_default_entrypoint
    exit_code = runner.run(start_time)
  File "/home/circleci/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/bin/pants_runner.py", line 86, in run
    return remote_runner.run()
  File "/home/circleci/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/bin/remote_pants_runner.py", line 99, in run
    return self._connect_and_execute(pantsd_handle)
  File "/home/circleci/.cache/pants/setup/bootstrap-Linux-x86_64/2.7.0_py38/lib/python3.8/site-packages/pants/bin/remote_pants_runner.py", line 131, in _connect_and_execute
    return PyNailgunClient(port, executor).execute(command, args, modified_env)
native_engine_pyo3.PantsdClientException: The pantsd process was killed during the run.

Memory usage stayed reasonable throughout the build so it wasn’t being killled by our OOM killer The core dump shows that it was instead killed with sigabort so… I assume the call was coming from inside the house somewhere

circleci@3b6b9430e214:~/project$ eu-readelf --notes  core.1609.\!home\!circleci\!.cache\!pants\!setup\!bootstrap-Linux-x86_64\!pants.y9Lzwt\!install\!bin\!python3.8  | head

Note segment of 151880 bytes at offset 0x97d8:
  Owner          Data size  Type
  CORE                 336  PRSTATUS
    info.si_signo: 6, info.si_code: 0, info.si_errno: 0, cursig: 6
    sigpend: <>
    sighold: <>
    pid: 1779, ppid: 1, pgrp: 1608, sid: 1608
    utime: 0.004000, stime: 0.004000, cutime: 1.475434, cstime: 0.262632
    orig_rax: 14, fpvalid: 1

Removing the pants cache & re-running allowed the build to go through but I’m curious as to what would be causing the process to abort?

Any way for

pants

to ignore linting/formatting/checking an entire directory tree? Our current build doesn't try and lint/fmt/check any directory trees that aren't XYZ, and for migration I'd like to keep this behavior. Hopefully adding a handful of

skip_ABC

in each

BUILD

isn't the way 😬

I have an interesting

fmt

conflict between black and isort. 🧵

I think I might be seeing an inference error:

a/
├─ b/
│  ├─ c/
│  │  ├─ d/
│  │  │  ├─ __init__py
│  │  ├─ foo.py

# foo.py
from a.b.c import d

Running

pylint

from inside

pants

gives:

E0611(no-name-in-module), ] No name 'd' in module 'a.b.c'

But running

pylint

on the file doesn't have the issue. Setting

[python-infer].inits=true

doesn't seem to alleviate the issue

Last set of

pylint

differences: I'm seeing

./pants lint

and

pylint

(in my

ve

) disagree.

pylint

in ve is fine,

./pants

says

[E1101(no-member), AClass.delete] Instance of 'scoped_session' has no 'commit' member

The code in question is essentially:

from flask_sqlalchemy import SQLAlchemy

db = SQLAlchemy()
...
db.session.commit()

I've set my

pants.toml

to match the

ve

's deps (I think it's a representative set)

[pylint]
version = "pylint==2.6.0"
lockfile = "3rdparty/pylint_lockfile.txt"
extra_requirements.add = [
    "astroid==2.4.2",
    "flask==1.1.2",
    "flask-sqlalchemy==2.4.4",
    "mccabe==0.6.1",
    "pylint-flask-sqlalchemy==0.2.0",
]

hey team, I would like to skip running a linter/type checker/formatter on all projects by default without modifying existing

python_library

declarations to be, e.g.

python_library(skip_mypy=True)

or replacing them with a macro (not feasible with the combinations of linter/formatter/type checker true/false). I wasn’t able to find a way to do this, I am afraid.

Hi all! I am having issues running

pants lint <target>

with the

black

backend. flake8 and bandit run as expected, but black fails on install with

Building black.pex with 2 requirements: black==21.5b2, setuptools

. it's not a poetry dependency (nor are flake and bandit) and I didn't put any

[black]

section in the pants.toml, I tried specifying other versions too, it always fails with this error. I am a bit lost on debugging this now, what can I do to understand this error a bit better?

Is pants named_caches ever cleaned? I missed to look how much space it took before starting wiping it, but when it wasn’t done in a minute, I stopped, and then it was still at 1G.. 😮

Pants 2.7.1 is building my

.pex

with the shebang

#!/usr/bin/env python3.8

despite my pants.toml telling it to use python3.9

[python-setup]
interpreter_constraints = ["CPython==3.9.*"]

And the platforms being set to 3.9

pex_binary(
    name="server",
    platforms=["linux_x86_64-cp-39-cp39", "macosx_11.0_x86_64-cp-39-cp39"],
    entry_point="some_project/server.py:main"
)

The same happens if I set the bex_binary's

interpreter_constraints

directly

I'm also having a problem resolving dependencies:

> ./pants dependencies baz:baz
baz/baz/application.py:../baz
baz/baz/server.py:../baz
library/core/core/mongo.py:../core  <--- This requires pymongo
library/foo/foo/jwtcherry.py:../foo
thirdparty/python:cherrypy

pymongo is not getting bundled into my

.pex

but it does correctly appear as a dependency of

core

, so I assumed this would have been resolved automatically

> ./pants dependencies library/core
<other deps>
thirdparty/python:pymongo
<some more deps>

I know there's docs regarding running

pylint

& friends in N processes instead of one big one (and letting the tool subprocess), but can't seem to find it. Perhaps it should be linked on this page? https://www.pantsbuild.org/v2.8/docs/python-linters-and-formatters

Hey all. 👋 I am working with pants

1.26.0

  (from the olden days) and I am trying to bump the thrift version that 👖 is using to thrift

0.13.0

based on this security tickethttps://www.cvedetails.com/cve/CVE-2019-0205/ It looks like pants support thrift in its https://binaries.pantsbuild.org/ endpoint up to thrift 

0.11.0

 but not greater than that version  . Does anyone have any context around how/if you can include other paths for pants to download thrift binaries like http://archive.apache.org/dist/thrift/0.13.0/? Thanks y'all 🙏 ERROR:

Invalidated 1 target.18:57:34 [INFO] Attempting to fetch thrift binary from: <https://binaries.pantsbuild.org/bin/thrift/mac/10.13/0.12.0/thrift> ...
18:57:35 [INFO] Attempting to fetch thrift binary from: /Users/eraines/dev/data-airflow-java/build-support/bin/thrift/mac/10.13/0.12.0/thrift ...

11:57:34 00:05       [execute]
FAILURE: ("Error resolving binary request BinaryRequest(supportdir='bin/thrift', version='0.12.0', name='thrift', platform_dependent=True, external_url_generator=None, archiver=None): Failed to fetch thrift binary from any source: (Failed to fetch binary from <https://binaries.pantsbuild.org/bin/thrift/mac/10.13/0.12.0/thrift>: Fetch of <https://binaries.pantsbuild.org/bin/thrift/mac/10.13/0.12.0/thrift> failed with status code 404, Failed to fetch binary from /Users/eraines/dev/data-airflow-java/build-support/bin/thrift/mac/10.13/0.12.0/thrift: Problem reading data from /Users/eraines/dev/data-airflow-java/build-support/bin/thrift/mac/10.13/0.12.0/thrift: [Errno 2] No such file or directory: '/Users/eraines/dev/data-airflow-java/build-support/bin/thrift/mac/10.13/0.12.0/thrift')", BinaryNotFound("Failed to fetch thrift binary from any source: (Failed to fetch binary from <https://binaries.pantsbuild.org/bin/thrift/mac/10.13/0.12.0/thrift>: Fetch of <https://binaries.pantsbuild.org/bin/thrift/mac/10.13/0.12.0/thrift> failed with status code 404, Failed to fetch binary from /Users/eraines/dev/data-airflow-java/build-support/bin/thrift/mac/10.13/0.12.0/thrift: Problem reading data from /Users/eraines/dev/data-airflow-java/build-support/bin/thrift/mac/10.13/0.12.0/thrift: [Errno 2] No such file or directory: '/Users/eraines/dev/data-airflow-java/build-support/bin/thrift/mac/10.13/0.12.0/thrift')"))

Hi. We have been looking into using Pants for a multi-project python repo at the firm I work for. The hosts that would run Pants do not have access to the internet, but we were able to get Pants fully installed and bootstrapped after redirecting it to our internal Artifactory and ran tailor on our proof of concept repo to generate BUILD files. However, when trying to lint (or run any goals really), we are running into issues with Pants trying to download things directly (instead of using redirected pip/PyPI). See screnshot where it tries to (update?) pex when running ./pants lint with mypy and black backends configured.

Is there anyway to know the external dependencies ahead of time so we can take care of those requirements w/o Pants trying to access internet?

Yes, run

./pants help tools

. You'll get information about each tool we use and either the Python requirements that we grab the tool via or else the URLs we fetch the tool from.

That's pretty new, available in Pants 2.8.x and newer: https://pypi.org/project/pantsbuild.pants/2.8.0rc5/

Ah, at the moment we only have 2.7.1 available internally.

Can you show me the commit for that CLI option? Maybe I can just copy it into my local install if the feature is isn't too scattered.

Also had Pex 2.1.54 installed already in the virtual environment; is this not enough to satisfy Pants or does it insist on downloading a specific binary (that hashes the same)?

Pants tries very hard to ignore your environment and be fully hermetic so a build that passes on your machine passes on mine. Anything installed in a venv used to run Pants is - hopefully - ignored. Except for the python interpreter and its stdlib.

The tools commit is here: https://github.com/pantsbuild/pants/commit/6c8ee5e824

Gotcha. That explains why even after pip installing, it still went through the entire process again (and forced us to configure it to use internal Artifactory/SSL). So where should I place these binaries it does not fetch from PyPI if the only option is to retrieve them directly myself (not sure how esay it will be for us to host binaries in a downloadable location, unless a network file location would work).

Yeah - you must provide a shared filesystem or an http server if you can't get these from the internet. Each tool will need to be (re)configured to point at that internal location.

Here's the list from main:

$ ./pants help tools

External Tools
--------------

autoflake          The Autoflake Python code formatter (<https://github.com/myint/autoflake>).
                   Version: autoflake==1.4

black              The Black Python code formatter (<https://black.readthedocs.io/>).
                   Version: black==21.9b0

coursier           A dependency resolver for the Maven ecosystem.
                   Version: v2.0.16-169-g194ebc55c
                   URL template: <https://github.com/coursier/coursier/releases/download/{version}/cs-{platform}.gz>

coverage-py        Configuration for Python test coverage measurement.
                   Version: coverage[toml]>=5.5,<5.6

docformatter       The Python docformatter tool (<https://github.com/myint/docformatter>).
                   Version: docformatter>=1.4,<1.5

dockerfile-parser  Used to parse Dockerfile build specs to infer their dependencies.
                   Version: dockerfile==3.2.0

download-pex-bin   The PEX (Python EXecutable) tool (<https://github.com/pantsbuild/pex>).
                   Version: v2.1.54
                   URL template: <https://github.com/pantsbuild/pex/releases/download/{version}/pex>

flake8             The Flake8 Python linter (<https://flake8.pycqa.org/>).
                   Version: flake8>=3.9.2,<4.0

ipython            The IPython enhanced REPL (<https://ipython.org/>).
                   Version: ipython==7.16.1

isort              The Python import sorter tool (<https://pycqa.github.io/isort/>).
                   Version: isort[pyproject,colors]>=5.9.3,<6.0

mypy               The MyPy Python type checker (<http://mypy-lang.org/>).
                   Version: mypy==0.910

poetry             Used to generate lockfiles for third-party Python dependencies.
                   Version: poetry==1.1.8

pytest             The pytest Python test framework (<https://docs.pytest.org/>).
                   Version: pytest>=6.2.4,<6.3

scala              Scala programming language
                   Version: 2.13.6

scc                The Succinct Code Counter, aka `scc` (<https://github.com/boyter/scc>).
                   Version: 3.0.0
                   URL template: <https://github.com/boyter/scc/releases/download/v{version}/scc-{version}-{platform}.zip>

setuptools         Python setuptools, used to package `python_distribution` targets.
                   Version: setuptools>=50.3.0,<58.0

shellcheck         A linter for shell scripts.
                   Version: v0.7.1
                   URL template: <https://github.com/koalaman/shellcheck/releases/download/{version}/shellcheck-{version}.{platform}.tar.xz>

shfmt              An autoformatter for shell scripts (<https://github.com/mvdan/sh>).
                   Version: v3.2.4
                   URL template: <https://github.com/mvdan/sh/releases/download/{version}/shfmt_{version}_{platform}>

twine              The utility for publishing Python distributions to PyPi and other Python repositories.
                   Version: twine==3.4.2

Use `./pants help $tool` to get help for a specific tool.

Ok this is helpful. I can fetch all of these from my desktop and then place them on an accessible network drive Pants (running on a Linux host) will have access to. When you say configure each tool, is that in pants.toml and which field am I setting (don't have access to the docs rn, but I think it's in the tools or 3rd part deps section?)