Hi everybody! I have a problem that.. I think might be related to something we were experiencing earlier (that I think has gotten fixed in a more recent version of pants?) Anyway we run our builds on circleci and are using their caching facilities to store/retrieve the pants cache we are getting - maybe every week and a half or two weeks - this error in our builds:

"Error storing Digest { hash: Fingerprint<734bd41218ea87070362108ef2424945f2c99700f6523a06ae7ee5f4ef2a09d4>, size_bytes: 2263 }: MDB_CORRUPTED: Located page was wrong type"

(I have the full logs if that's useful but I don't know if it contains much more information (the command was

./pants --no-dynamic-ui --spec-files=.targets.txt package

) Has anybody else experienced this? Is this is an issue with what & how we are saving in the cache? is it possible we need to like.. flush some file or something explicitly so the database is consistent? Or more generally does anyone have any idea if this is an issue with our code/files, circleci, pants or underlying in LMDB?

Is there supposed to be some protection against more than one target owning a source? Somehow I got it into my head that there was an error that came up when that happened, but it seems easy to to make two targets own the same source (forgive me for using BUILD files in my root dir, I'll repent soon, I promise):

❯ ./pants list aiven/admin/__init__.py
//aiven/admin/__init__.py:../../aiven
//aiven/admin/__init__.py:../../fake-target

Was there ever protection against this? Is this considered an okay situation to be in?

Is there a TLDR on settings

pants.toml

for

build_patterns

marker_filenames

and

root_patterns

? I think

marker_filenames

and

root_patterns

both set the source root, via different mechanisms, and then

build_patterns

is what Pants looks for to identify something pants-able. However, I just did a re-factor, and I'm not sure why I lost access to one of my libraries. /core/BUILD.pants

python_sources(
    name="libcore",
    sources=["**/*.py"],
)

/helloworld/BUILD.pants

python_sources(
    name="libhelloworld",
    sources=["**/*.py"],
    dependencies=[
        "core:libcore",
    ],
)

This above, works, however, nested everything under

examples/python

(without adding new BUILD files) and it falls over. My

pants.toml

is as follows:

[GLOBAL]
pants_version = "2.11.0.dev2"
build_patterns = ["BUILD.pants", "BUILD"]

[source]
marker_filenames = ["BUILD.pants"]
root_patterns = [
    "examples/python",
    "pants-plugins",
]

...

After re-reading https://www.pantsbuild.org/docs/existing-repositories#migrating-from-other-build-tools-set-custom-build-file-names - I think I can drop the

build_patterns

entirely, and while I can add another top-level BUILD to the

examples/python

directory, I'm not sure why it worked top level but then fails nested... Note: My

pants.toml

had "/" as a root_pattern, when everything was top-level

Hey, following up on this thread, I tried to build a wheel for a python package that only has an sdist, and pants doesn't seem to be able to find it, even though I can see the extra index URL and I get the package when I run

pip download

with the same settings. Can I get some help? Details in thread.

I noticed that the file permissions of .pex files that Pants produces are different between the 2.7.1 and 2.9.0, please see the thread.

Tool

isort

doesn't handle

src_paths

from

.isort.cfg

correctly. If you have namespaced packages implemented in different parts of the repo, then in order for isort to consider any modules exposed in these namespaces first-party imports, you need to add the paths explicitly in the configuration (https://github.com/PyCQA/isort/issues/1704). When running isort with Pants, this does not work (version 5.10.1):

./pants fmt --only=isort path_to_file_that_imports_firstparty_namespaced_modules

reports isort made no changes, but

isort path_to_file_that_imports_firstparty_namespaced_modules

makes the expected change in grouping. One clue is

isort --show-config

reveals the

src_paths

field to be fully qualified, even though they are only specified relative to the configuration root, whereas the same command ran by Pants shows they're still relative. Pants v2.10rc2.

I notice that we don't document the special-case of the DEFAULT section in pants.toml. This was a holdover from the pants.ini days, where, for backwards compatibility with Python's configfile parser, if you request a value for

bar

in section

foo

, and it doesn't exist, but there is a

bar

in section

DEFAULT

, then that latter value will be returned. There doesn't seem to be much need for this functionality any more (it made a little more sense when we had recursive options) so I propose deprecating it, unless people are strongly relying on it. Please comment in thread if you have opinions on this, thanks!

I am trying to generate lockfiles using pants

2.10.0rc5

but keep getting error. Please see the 🧵 for details

EDIT: Ignore this, Pants is working fine 😄 Just a basic fact or rather something I’d like to have someone’s opinion on: • When you have tests and a

conftest.py

in one directory with a BUILD-file and in a deeper level you have more tests with their own BUILD-file, the usual “pytest” behavior would be to go up the folders and use all

conftest.py

-Files up the tree. This is kind of obsolete with Pants, right? So I could either import the stuff I need from

conftest.py

above into my tests, or define the

python_test_utils

from “above” in my BUILD-file. Does this make sense? 😅

Does Pants only consider the root

.gitignore

when deciding what to augment its global ignore option with? I notice that if, e.g.,

*.csv

is in

.gitignore

and

!exempt_csv.csv

is in

./proj/apps/someapp/assets/.gitignore

, then git will track

./proj/apps/someapp/assets/exempt_csv.csv

but Pants will continue to ignore and exclude it from packaging.

Hey everyone, just tried out the new

2.11.0rc0

version to generate lock files with

pex

lockfile-generator, and the part about generating lockfiles works fine ^^, but found a bug when trying to install the resolved lockfile, wrote a ticket about it: https://github.com/pantsbuild/pants/issues/14859

Hi everyone, How can I run and debug pants directly from vscode? Is it possible?

Hi all, I have an issue when I try to package a pex_binary. I get the following error:

ProcessExecutionFailure: Process 'Building src.python.herzensgut.accounts/serve.pex with 12 requirements: bcrypt==3.2.0, databases[postgresql]==0.5.3, email-validator==1.1.3, fastapi-health==0.4.0, fastapi-jwt==0.1.7, fastapi-mail==1.0.2, fastapi==0.70.0, passlib==1.7.4, pyjwt==2.0.1, python-dotenv==0.19.2, python-multipart==0.0.5, uvicorn==0.16.0' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement fastapi-jwt==0.1.7
ERROR: No matching distribution found for fastapi-jwt==0.1.7

The strange thing is, that running the

pex_binary

is working. This is my BUILD file:

python_requirement(
    requirements=[
        "bcrypt==3.2.0",
        "fastapi==0.70.0",
        "fastapi-health==0.4.0",
        "fastapi-jwt==0.1.7",
        "fastapi-mail==1.0.2",
        "email-validator==1.1.3",
        "passlib==1.7.4",
        "pyjwt==2.0.1",
        "python-dotenv==0.19.2",
        "python-multipart==0.0.5",
        "uvicorn==0.16.0",
    ]
)

resources(
    name="queries",
    sources=[
        "**/*.sql",
    ],
)

resources(
    name="templates",
    sources=[
        "**/*.html",
    ],
)

python_sources(
    name="lib",
    sources=[
        "**/*.py",
    ],
    dependencies=[
        ":queries",
        ":templates",
        "src/python/herzensgut/shared:lib",
    ],
)

python_distribution(
    name="dist",
    dependencies=[
        ":lib",
    ],
    wheel=True,
    sdist=True,
    provides=setup_py(
        version="0.0.1",
        name="herzensgut.accounts",
        description="",
    ),
)

pex_binary(
    name="serve",
    restartable=True,
    entry_point="app/api/server.py",
)

Do you have an idea what the problem is? Thanks!

FYI, for current (and future 🙂 ) contributors: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1647816687766429

We're at 1928 github stars. Let's get to 2k! https://github.com/pantsbuild/pants

good evening / afternoon.

Wondering - where do i find the plans for • arm docker buildx ? • rust builds - ideally kick off a cargo build 🙂

Does anyone know if Pants works in Conda environments? I've got a teammate who's trying to bootstrap Pants in an existing repo on a Conda 3.9.7 environment and hitting an error resolving the 2.9.0 version of Pants.

Trouble in the example repo:

Got a docker question. I'd like to avoid proliferating our AWS ECR registry ID around our repo (since it's complicated). The

pants.toml

setting was great to set up something like

[docker.registries.aws-ecr]

so that I could refer to them. However, one of my images depends on a base image we store in that registry. It seems like I still have to do

FROM <my long registry ID>/<image name>:<image tag>

instead of just

FROM <image name>:<image tag>

. My target configuration looks like this

docker_image(
    name = "gnc_ground",
    registries = ["@aws-ecr"],
    tags = ["ground_software_service"],
)

Is there some flag I need to set to enable lockfiles for tools (on 2.9.0) ?

./pants generate-lockfiles --resolve=terraform-hcl2-parser
Unknown flag --resolve on global scope
Did you mean --level?
Use `./pants help` to get help.
11:57:54.70 [ERROR] Unknown flags --resolve on global scope


Use --print-stacktrace for more error details and/or -ldebug for more logs. 
See <https://www.pantsbuild.org/v2.9/docs/troubleshooting> for common issues.
Consider reaching out for help: <https://www.pantsbuild.org/v2.9/docs/getting-help>

Can somebody give me a hint on how to generate a pytest lockfile with specific versions of dependencies? We’re using some plugins for pytest & django (i.e.

pytest-django

) which are loaded when pytest starts. When I generate a lockfile using

./pants generate-lockfile

I get

pytest-django==3.10.0

but

pytest-django==4.1.0

is what I need and what is written in our

requirements-dev.txt

actually 🤔

I have a eng who is having installation issues for a Python dep 🧵

2022-03-22T12:54:13,150 ERROR: Could not find a version that satisfies the requirement audioread==2.1.8
2022-03-22T12:54:13,150 ERROR: No matching distribution found for audioread==2.1.8

Maybe a stupid question, but we just switched from

django-dotenv

to

python-dotenv

and with that from absolute to relative and a-lot-easier

.env

-File loading. But now my testing with Pants stopped working due to the fact (I guess) that it’s hermetic and the

.env

-File cannot be found. Is there a common way to use

.env

-Files inside Pants? At least I couldn’t find anything related to that topic inside the docs 🙂.

Another odd case from one of our engs. They report touching a test file, then seeing us run the lint tools on the one file. Then a few pants runs (with the same exact cmd) later they see us run the lint tools on the "bucket" of files that test file is in.

Soliciting your opinions about the

No applicable files or targets matched.

WARN log: Today when you do things like

./pants test ::

or

./pants --changed-since=main --changed-dependees=transitive test

and no test files match, you get a WARN log along the lines of

17:18:14.39 [WARN] No applicable files or targets matched. The `test` goal works with these target types:

  * go_package
  * junit_test
  * python_test
  * scala_junit_test
  * scalatest_test
  * shunit2_test

However, you only specified targets with these target types:

  * python_source

Please specify relevant files and/or targets. Run `./pants filter --target-type=go_package,junit_test,python_test,scala_junit_test,scalatest_test,shunit2_test ::` to find all applicable targets in your project, or run `./pants filter --target-type=go_package,junit_test,python_test,scala_junit_test,scalatest_test,shunit2_test :: | xargs ./pants filedeps` to find all applicable files.

I am currently wondering whether this warning is desirable. • Arguments for it include that it may help catch cases where your Pants command did not match your intent. • Arguments against it include that the empty set is not a special case, so if "run all tests that match this spec" ends up being fulfilled vacuously then so be it. This is particularly salient for

--changed-*

or when generating command specs from other data.

Hey, a UX question, what is the fastest way to run (e.g. during the development) a

console_scripts

endpoint of

python_distribution

? Similar to

./pants run <pex_target>

? Or is it incouraged to develop useing pex binaries? In this case, how to deal with those which contain huge amount of third-party heavy dependencies (think

pytorch

)? thanks in advance 🙏

Just want to make a quick announcement @happy-kitchen-89482 @enough-analyst-54434. I was able to get some dedicated servers for our team with TB's of local disk storage, and so now we are able to checkout SCM workspaces to the local filesystem as well as point Pants and PEX to those locations for their caches (all while preserving the convenience of having user home directories still mapped to NFS). I was able to get over 10+ people fully setup with a single onboarding script yesterday with zero issues for anyone. One of the most promising things is the script packages and tests the entire project (which includes

python_distribution

and

pex_binary

targets), and this time there were no weird hash digest errors, runtime deadlocks, skipped dependencies, etc. and goals ran much more quickly. I suppose the moral of the story is don't use NFSv3. With that said, I just got word that our organization is starting the process of migrating various NFS shares to NFSv4. I don't know the implications of that, but when it happens I will probably try to give NFS another chance, as it seems there are material changes to the file locking mechanisms as it's now baked directly into the protocol. When that gets rolled out and I have an opportunity to test it, I'll pass along another update. It will be worthwhile for us to still get things working on NFS since it is so deeply embedded in the organization, and getting setup on local storage is more operationally intensive (permissions, committing to a single host, etc.). I also heard we made a new hire to get a pilot going for WSL2, which I think will be the ultimate solution. But we'll see about that one...! Thanks for all your help since we started this process. I think I can see a light at the end of the tunnel now!

Another question for understanding. Imagine the following structure:

package_a/
  a.py
  BUILD -> python_distribution(name = "package_a")
package_b/
  b.py -> imports package_a.a
  BUILD -> python_distribution(name = "package_b")

My previous understanding was that when building wheels for

package_b

, it would detect automatically the dependency on

package_a

(via the respective file

a.py

) and I don't need to list this dependency explicityly in

python_distribution

i.e.

package_b

will just include the sources and transitive deps of the

package_a

However, some recent experiments show it is not true, so I must include

//package_a

as a

dep

of

package_b

. Is that correct? Doesn't this defeat the idea of pants taking over dep management, so that if later the actual import of

a.py

is removed, there's not dangling dependency on

package_a

remaining? Thanks! I hope I explained my issue clearly, LMK if I can refine.

Hey, so I started setting up pants in CI, and from reading the doc https://www.pantsbuild.org/docs/using-pants-in-ci It is recommended to cache the

$HOME/.cache/pants

folder between steps/build for faster execution. The catch is that even when running a

./pants

goals on only changed targets (with

--changed-since=origin/main

), pants starts by downloading all dependencies in the impacted lockfiles, and with a

data-science

resolve that can easily get to ~10Gbs of cache to upload/download. Is there any way around this ? or Am i doing something wrong^^, since it's indeed slower to push/pull 10Gigs of cache each time