Seems not a day passes when I don't find something else to not understand 🙂 🙂 I'm playing with the new "pants-hash" functionality for docker image tags (2.10.0.dev1) Love it love it love it. Thanks a ton! ...and as usual, more questions 😄 first-- Is it possible to also tag with the existing git hash (or tag with an environment variable, which would be very flexible)? The reason I ask is that we've been tagging our production containers with the SHA of the commit which produced them in case we have to do hot-fixes (makes it terribly easy to figure out what tree they were built from). We've used good old

git log -1 --pretty=%H

to get the SHA. So the idea is that we'd tag each container both with a "git hash" (which would be different for each build) and a "pants hash" (which ideally wouldn't change unless the input context changed). We wouldn't deploy a container unless the "Pants hash" changed, but with the "git hash" on there we could look up the commit. Second--this is possibly better handled by the docker tooling, but since it relates--is it possible to get information about targets (in this case, such as the pants-hash) in any form, particularly human-readable? In other words, it would be nice (again, no need for extra work yet!) to say "okay, once we've built these, what would the image names and tags be of all the built containers in this repository?" so that we could then programmatically update deployments. I know some of this may fall outside the purview of the build tool itself, but this is great stuff for us and it's also just fun to learn.

Random "I did not expect that" of the day: I had a

files

target for building a docker container to include an entire source tree:

files(sources=["**/*"], name="all_files")

... it didn't seem to include a "yarn.lock" file in that directory which was necessary for the build to proceed. I added it into the sources directly and it still didn't seem to be there. I ran with

--no-process-cleanup

and discovered that

client_secrets.json

and

yarn.lock

were both missing (ie present in the original source tree, but not in the execution environment). Adding them to the

sources

field directly didn't seem to help. I'm lightly baffled; any idea what I'm missing here?

./pants peek [that-target]

even lists them in

sources_raw

(ie I'm asking for them) but they don't show up anywhere else in the

peek

output.

another bump for https://pantsbuild.slack.com/archives/C18RRR4JK/p1641338890053700 ! we’ve gotten a lot of great ideas already, but we’d love to hear yours too! submissions close wednesday, when we switch to voting

Question about caching... I'm trying to customize cache locations to isolate some executions. I want to confirm that the env vars/options I need to do this are: •

PANTS_SETUP_CACHE

•

PANTS_LOCAL_STORE_DIR

•

PANTS_NAMED_CACHES_DIR

Is there anything else I need to set? Is there an easier way to set this all at once (other than overriding

$XDG_CACHE_HOME

)?

Running into a weird caching issue when trying to pipe things:

./pants dependees --transitive some/file.py | xargs ./pants filter --target-type=python_tests --granularity=file

If I run this once, it takes a while to compute, which I expect. But, on subsequent tries it looks like Pants usually needs to completely recompute the depedency graph every time I run it. (Well, using

-ldebug

shows that it's hitting cache for each module but it just takes a while to traverse the whole graph, even if it's using cache each time). My question is... why doesn't it just remember the final output of that recent

./pants dependees

command? Does this have something to do with pantsd?

I'm hitting this warning a lot in my pants runs, and I suspect it's slowing down execution times:

15:52:47.86 [WARN] Completed: Find Python interpreter to bootstrap PEX - No bootstrap Python executable could be found from the option `interpreter_search_paths` in the `[python-setup]` scope. Will attempt to run PEXes directly.

My config looks like this:

[python-setup]
interpreter_search_paths = ["/usr/bin/python3"]

What's wrong with this search path? That Python definitely exists. What could make it invalid?

Hi Pants! I'm on Pants 2.8.0 and trying to onboard myself to

experimental_resolves_to_lockfiles = { default = "constraints.txt"}

. I have a few different requirements.txt files, say

**/requirements.txt

. How do I let pants auto-generate my lockfile based on the requirements files?

hey folks, FYI: my team at Twitter just shipped the first series of a new blog post: "Advancing Jupyter Notebooks at Twitter - Part 1" where we talk about our Jupyter Notebook tooling for DS/ML practitioners and the intersection with monorepos, pants, pex, the pantsbuild/pants-jupyter-plugin integration and more. Posted in #announce

Can a

docker_image

depend on another

docker_image

? I.e. can we use pants to build a base image?

Wondering how this should be handled. I packaging a python

pex_binary

in a

docker_image

. My python application includes a python package (

pybedtools

) which has some native components. My dev system is ubuntu21.04. The docker image base is python:3.8.12-slim. When I package the pex it builds components of

pybedtools

against the GLIBC 2.33 on my dev system, which then fails inside the docker image where 2.31 is installed:

root@e107dc414f9c:/color# python3 $PEX_BINARY 
...
  File "/root/.pex/installed_wheels/bf89cafd4094cc8d47719f572e0edb81d1c39ef9/pybedtools-0.8.2-cp38-cp38-linux_x86_64.whl/pybedtools/__init__.py", line 9, in <module>
    from .cbedtools import (
ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /root/.pex/installed_wheels/bf89cafd4094cc8d47719f572e0edb81d1c39ef9/pybedtools-0.8.2-cp38-cp38-linux_x86_64.whl/pybedtools/cbedtools.cpython-38-x86_64-linux-gnu.so)
root@e107dc414f9c:/color# ls -lah /lib/x86_64-linux-gnu/libc.so.6 
lrwxrwxrwx 1 root root 12 Oct  2 12:47 /lib/x86_64-linux-gnu/libc.so.6 -> libc-2.31.so

Possible solutions (none perfect): 1. Match my docker image base to my dev machine? (brittle) 2. Run the

./pants package path/to/Dockerfile

in a docker image? (gets convoluted fast) 3. Rebuild wheels somehow at docker image build time or runtime? (unclear how)

This question is in line with the previous one as I am also exploring potential solutions. Currently our team often pulls docker images for testing purposes but building is done within our CI/CD pipeline. Is there a way I can prevent developers to build a docker image with

pants package ::

when they not on a specific platform, but allow building of pex files? I assume I would have to write a plugin for that, just curious how I should approach this

Just noting that the “How to upload your distributions to a package index” callout on https://www.pantsbuild.org/v2.9/docs/python-distributions is no longer accurate, given that we have

./pants publish …

Any suggestion on which docker image to use during out CI/CD runs? As I can see it needs at least python3.8 and docker installed (when one wants to build docker images).

There is another Python package import issue I came across: Is is possible to use system installed site-packages (and make them available inside pants)? More details in the 🧵.

hey folks! another bump for https://pantsbuild.slack.com/archives/C18RRR4JK/p1641338890053700 : add an idea or two before submissions close tomorrow!

Like Pants? Here are some ways you can help more people benefit: • Tell peers why you use Pants and who you think will benefit from Pants too. Spread the word! Whether that be in a casual chat, tweet, blog post, tutorial, presentation, or video. • Contribute to the collaborative spirit of the community by helping a fellow user on Slack or GitHub. • Star the project on GitHub https://github.com/pantsbuild/pants. It's an easy action that helps build exposure and momentum. • Contribute to the docs using the "Suggest Edits" button. • Contribute to the codebase by suggesting an idea, making a feature request, giving feedback on an issue or dev release, opening an issue, or starting a pull request. • List your organization or open source project on the Who Uses Pants page https://www.pantsbuild.org/page/who-uses-pants • Add a testimonial https://www.pantsbuild.org/docs/testimonials • Publish a case study of how your goals are being served by Pants. This is easily one of the most powerful things you can do! • Ask your organization to consider partnering with us on things like adding support for JavaScript https://www.pantsbuild.org/page/language-support, other languages, or a key new feature. The more resources, the faster we can add that one more thing you've been eagerly waiting for...

I saw an error message today when bootstrapping Pants in CI (Linux), but can’t reproduce locally (the error is gone for a fresh build of a Git checkout now). Thought to share anyway in case it’s a symptom of something important:

Urk, I'm sorry to re-ask this, but there was a snippet that allows export of requirements (python_requirement dependencies) to a "requirements.txt" like format, and I can't find it; I'm assuming the new way would be to use something like

./pants dependencies --transitive some_target | xargs ./pants filter --target-type=python_requirement | some_export_stanza

but I'm a bit vague on that last part... (BTW, very nice allowing for the overrides in

python_requirements

for things like the notorious missing setuptools/pkg_resources and other undeclared third-party dependencies; that worked wonderfully and cleaned up a lot)

Hey Pants Team, We're having issues reported from M1 mac users regarding

psutil/_psutil_osx.cpython-39-darwin.so

. See this gist for more debug info. We followed directions in https://www.pantsbuild.org/v2.7/docs/installation#macos but no luck. We tried removing the library and installing it directly but still having the same issue. Any help is greatly appreciated!

hi, I'm trying to include bare files in my pex file with pants 2.9rc6. somehow it doesnt work:

# noinspection PyUnresolvedReferences
files(
    name="test-libs",
    sources=[
        '../test_libs/**'
    ],
)

# noinspection PyUnresolvedReferences
pex_binary(
    name="itg_pex",
    entry_point="hub/main.py",
    dependencies=[":test-libs"],
)

those files are not imported directly and there may be some resources in that directories My intention is to have them present in the pex file

hi, getting a coredump from the JVM when running

./pants lint

on our whole repo (which has around 1500 Scala files):

Error: 6.11 [ERROR] Completed: Lint with scalafmt - scalafmt failed (exit code -6).
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fb1f0e766f1, pid=5768, tid=5791
#
# JRE version: OpenJDK Runtime Environment (11.0+28) (build 11+28)
# Java VM: OpenJDK 64-Bit Server VM (11+28, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0xbf46f1]  Node::add_req(Node*)+0xc1
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /tmp/process-executionIMybnt/core.5768)
#
# An error report file with more information is saved as:
# /tmp/process-executionIMybnt/hs_err_pid5768.log
#
# Compiler replay data is saved as:
# /tmp/process-executionIMybnt/replay_pid5768.log
#
# If you would like to submit a bug report, please visit:
#   <http://bugreport.java.com/bugreport/crash.jsp>
#



✓ Black succeeded.
✓ Docformatter succeeded.
✓ Flake8 succeeded.
✓ Google Java Format succeeded.
✓ Shellcheck succeeded.
✓ gofmt succeeded.
✓ hadolint succeeded.
✓ isort succeeded.
𐄂 scalafmt failed.
✓ shfmt succeeded.

just checking if you are aware of this

I assume this is intentional, but I'm kind of puzzled as to why:

09:16:23.41 [WARN] The constraints file requirements.txt does not contain entries for the following requirements: pantsbuild-pants.

Ignoring `[python_setup].resolve_all_constraints` option.
09:16:23.46 [ERROR] 1 Exception encountered:

  ValueError: [python].run_against_entire_lockfile was set, but could not find a lockfile or constraints file for this target set. See <https://www.pantsbuild.org/v2.9/docs/python-third-party-dependencies> for details.

Doesn't the version in my

pants.toml

act like a pin for pants' version? Do I have to keep both in lockstep? Should

pantsbuild-pants

be special-cased for

run_against_entire_lockfile

?

When running Pants commands I'm seeing `Filesystem changed during run: retrying

Lint

in 500ms...` happen seemingly sporadically. Is there any way to dive into which file(s) changed? Admittedly my setup is weird as we transition to Pants. I'm running Pants in a subprocess from Python being run as a

bazel run myscript.py

command. So there's probably shenanigans happening, but understanding what changed would help me narrow the shenanigan-er

Found this on 2.9.0rc6:

❯ ./pants dependencies --transitive aiven/pheme
11:44:23.36 [ERROR] 1 Exception encountered:

  BinaryNotFoundError: Was not able to locate a Python interpreter to execute rule code.
Please ensure that Python is available in one of the locations identified by `[python-bootstrap] search_path`, which currently expands to:
  ['/usr/bin/python3.9']

This can be fixed by changing the path to

/usr/bin/python

which is a symlink to

/usr/bin/python3.9

. Seems like the interpreter search is picky about how the file is named maybe? Is this expected?

In #welcome we see a lot of people enter the community at the point where they are sorting out either/both (a) whether and how to convert to a monorepo or (b) whether and how to adopt a formal build system instead of an ad hoc solution. Would it be helpful for folks if we created some tutorials about either/both of those topics? If so, what specifically are questions that would be helpful to address in greater detail than current resources on pantsbuild.org and elsewhere? What gaps could we fill to make it easier for you to make good decisions for your org, and move forward with those confidently?

I’m getting the following output running

./pants publish

for a docker image

17:32:51.78 [TRACE] Completed: pants.core.goals.publish.package_for_publish
log: failed to find directory: exec: "sh": executable file not found in $PATHThe push refers to repository [<http://888845.dkr.ecr.us-east-1.amazonaws.com/8/example|888845.dkr.ecr.us-east-1.amazonaws.com/8/example>]
780d01aa651c: Preparing

Any ideas? This is with 2.9.0rc6.

today is the last chance to submit ideas for 2022 roadmapping! https://pantsbuild.slack.com/archives/C18RRR4JK/p1641338890053700 … tomorrow we will switch to voting.

Also, any tips for auth with AWS ECR? I saw a change to add ‘secrets’ but not sure how to use.

Is Terraform out of experimental? I see code that's out of the "experimental" package, but I don't see any docs on the site.