pantsbuild #development

Channels

welcome

github-notifications

development

early-jelly-55072

03/22/2022, 8:16 PM

I am running into an issue using the bleeding edge 2.11.0rc0. When I do a reset of pants cache and pants.d and set python to

python>=3.8

and

.python-version

has 3.8.13 set, I get an error saying the SHA256 of

pex

is not as expected. I also have pex-cli.url_template set to a local path because I'm behind an aggressive proxy. If I set python to 3.9 it works. I can also revert to 3.8 and it will continue to work. The prerequisites for 2.11 says 3.7, 3.8 or 3.9 required.

fast-nail-55400

03/23/2022, 2:21 PM

wrote up a “design needed” issue for streaming notifications for BSP. https://github.com/pantsbuild/pants/issues/14885

witty-crayon-22786

03/23/2022, 5:26 PM

@fast-nail-55400: i saw you create a PR yesterday in order to kick off a SHA build: i assume that was because just pushing the branch to origin wasn’t enough? it used to be, but maybe our branch filters changed recently.

witty-crayon-22786

03/23/2022, 5:35 PM

public note-to-self: an important part of the speedups that should be possible with PEX native lockfiles is skipping the building of the “repository pex”

--pex-repository

, and directly consuming the lockfile to build each subset of the resolve. i haven’t followed up on this because there is a lot in flight, but doing that is likely something that would be worth getting into 2.11.x as a key benefit to the switch

👍 2

ancient-vegetable-10556

03/23/2022, 6:18 PM

Is there a way to get processes to echo their stdout/stderr automatically, or do we need to explicitly log their captured output? (@witty-crayon-22786)

happy-kitchen-89482

03/24/2022, 7:01 PM

Note, I've updated https://www.pantsbuild.org/docs/contributor-overview with the new requirement to add a

category:

label to each PR.

👍 1

hundreds-father-404

03/25/2022, 1:53 AM

@polite-garden-50641 the dependabot PRs are failing because they need the label

category:internal

- is that possible to configure?

bitter-ability-32190

03/25/2022, 12:00 PM

Moving this to a more public place where we can collaborate. I'm trying out an upcoming change where we'll be making N (#cores) VenvPEXs with overlapping sets of deps (using the PEX lockfile, so we know versions and metadata beforehand). And at some point will run into

Copy code

File "/home/joshuacannon/.cache/pants/named_caches/pex_root/installed_wheels/0457d0c3fb526f3f246a04ebe5fa67dfa4f0877fe617e106f23f3b083f7b1ab1/pex-2.1.73-py2.py3-none-any.whl/pex/common.py", line 470, in atomic_directory
    fcntl.lockf(lock_fd, fcntl.LOCK_EX)  # A blocking write lock.
OSError: [Errno 35] Resource deadlock avoided

I suspect the fact that we have overlapping sets of deps being requested in N

pex

processes is what's causing this.

busy-vase-39202

03/25/2022, 9:08 PM

We now have a maintainer Josh and a contributor Josh on the team. (Yay!) So when referring to either person, especially in conversations with newer members of the community, please try to namecheck a handle so folks are clear on who you mean. Thanks!

happy-kitchen-89482

03/25/2022, 11:01 PM

Will cut 2.12.0.dev0 soon, now that #14911 is reverted, unless anyone wants me to wait for anything.

happy-kitchen-89482

03/26/2022, 3:42 AM

Please no merges to main until I get the release merged

happy-kitchen-89482

03/26/2022, 3:45 AM

OK, merged the release prep so main is unblocked

happy-kitchen-89482

03/26/2022, 3:46 AM

BTW, in my spare time I will noodle on a custom goal to completely automate the release prep, now that we have the classification via labels

🙌 2

happy-kitchen-89482

03/26/2022, 5:02 AM

Seeing this issue running

PANTS_PEX_RELEASE=STABLE ./build-support/bin/release.sh build-universal-pex

on MacOS Big Sur (x86_64):

Copy code

[=== 00:17 Checking prebuilt wheels for 2.12.0.dev0+git2229d781 ===]
All 2 pantsbuild.pants packages were fetched and are valid.
ERROR: Could not find a version that satisfies the requirement PyYAML<7.0,>=6.0 (from pantsbuild.pants==2.12.0.dev0+git2229d781) (from versions: none)
ERROR: No matching distribution found for PyYAML<7.0,>=6.0 (from pantsbuild.pants==2.12.0.dev0+git2229d781)
ERROR: Could not find a version that satisfies the requirement ijson==3.1.4 (from pantsbuild.pants==2.12.0.dev0+git2229d781) (from versions: none)
ERROR: No matching distribution found for ijson==3.1.4 (from pantsbuild.pants==2.12.0.dev0+git2229d781)
ERROR: Could not find a version that satisfies the requirement PyYAML<7.0,>=6.0 (from pantsbuild.pants==2.12.0.dev0+git2229d781) (from versions: none)
ERROR: No matching distribution found for PyYAML<7.0,>=6.0 (from pantsbuild.pants==2.12.0.dev0+git2229d781)

happy-kitchen-89482

03/26/2022, 5:02 AM

But before that there was, e.g.,

Fetching <https://binaries.pantsbuild.org/wheels%2F3rdparty%2F2229d781a02ca2e3891c3595eee22c11d5c37113%2F2.12.0.dev0%2Bgit2229d781%2Ftypes_PyYAML-6.0.3-py3-none-any.whl>

hundreds-father-404

03/26/2022, 6:06 PM

@happy-kitchen-89482 fyi an awkward edge with the required PR labels: only Contributor and Maintainer team members are currently able to add them. Would it make sense to let anyone add GH labels?

happy-kitchen-89482

03/27/2022, 4:20 PM

OK, figured out why building the release pex is broken. The culprit is https://github.com/pantsbuild/pants/pull/14813.

❗ 1

bitter-ability-32190

03/28/2022, 12:40 AM

Should we deprecate

[python].run_against_entire_lockfile

enable_resolves

True

, and we detect it's a PEX lockfile (once we have the new "everything in a venv" solution)? There's very little reason to keep it around, the only one I can think of is backwards-compatibility (explained in 🧵)

busy-vase-39202

03/29/2022, 3:46 AM

📣 If you are a Pants Contributor or Pants Maintainer, Toolchain would like to invite you to a team meetup online to get better acquainted with fellow team members, discuss 2022 priorities, and take a closer look at the codebase. To make participation viable across Pants team members' many time zones, it will be held at 9am Pacific / 18:00 CEST. Are there any particular weekdays during the first 3 weeks of April that would be easier/harder for you to join everyone for this meetup?

👍 3

fast-nail-55400

03/29/2022, 1:38 PM

is there any way to specify an address spec to choose a particular resolve? for example, maybe something like

src/jvm::@resolve=foo

— this could select all targets at or under

src/jvm

that have

resolve=foo

bitter-ability-32190

03/29/2022, 3:53 PM

Would it make sense to elide

RequirementsPexRequest

references to just use

PexFromTargetsRequest

(and move it's functionality into the rule that resolves the latter). It's strictly a subset (and ends up creating a

PexFromTargetsRequest

with the defaults) and I'm working on a PR which would benefit from this simplification

fast-nail-55400

03/29/2022, 4:13 PM

do we prefer TOML or YAML for ancillary config files?

hundreds-father-404

03/29/2022, 11:49 PM

Thoughts on deprecating requirement constraints in Pants 2.11 in favor of

[python].resolves

, now that we have Pex lockfiles?

hundreds-father-404

03/30/2022, 12:27 AM

can a releaser please do the pants.pex release for 2.9.1 for me? (As mentioned before, I have issues w/ my M1)

👍 1

wide-midnight-78598

03/31/2022, 1:04 AM

Is this the best way I should be doing this?

Copy code

ansible_pex = await Get(
        Pex,
        PexRequest,
        ansible.to_pex_request(),
    )

This takes about 200 seconds on my macbook, while pip installing ansible is about 45 seconds. I know this is doing more than just installing, but I feel like maybe I'm missing an optimization

bitter-ability-32190

03/31/2022, 11:17 AM

I feel like I remember someone posting a way to help debug graph errors ❓ I'm encountering them from a refactor: https://github.com/pantsbuild/pants/pull/14965#issuecomment-1084438930

Encountered 730 rule graph errors:

wide-midnight-78598

03/31/2022, 2:35 PM

This may coincidentally be related to Josh's message above - but I kinda recall seeing a web UI that helps with either graphs or plugins or something, and being super excited about it. @happy-kitchen-89482 Was that you? Can't remember what to search for.

fast-nail-55400

03/31/2022, 4:25 PM

I am going to land the Rust v1.59.0 upgrade. If you use macOS, you will probably have to upgrade to XCode 13.3 in order to avoid a crash in

cargo clippy

due to https://github.com/rust-lang/rust-clippy/issues/8470.

👍 2

polite-garden-50641

03/31/2022, 4:55 PM

I keep getting an error when I try to run pants in the pants repo. I am on latest main branch with no changes.

polite-garden-50641

03/31/2022, 4:55 PM

Copy code

Compiling store v0.1.0 (/data/home/asher/projects/pants/src/rust/engine/fs/store)
   Compiling cache v0.0.1 (/data/home/asher/projects/pants/src/rust/engine/cache)
   Compiling mock v0.0.1 (/data/home/asher/projects/pants/src/rust/engine/testutil/mock)
   Compiling process_execution v0.0.1 (/data/home/asher/projects/pants/src/rust/engine/process_execution)
error: could not compile `process_execution`

Caused by:
  process didn't exit successfully: `rustc --crate-name process_execution --edition=2021 process_execution/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no -C codegen-units=1 -C debuginfo=2 -C metadata=7ff608cd959af0e6 -C extra-filename=-7ff608cd959af0e6 --out-dir /data/home/asher/projects/pants/src/rust/engine/target/release/deps -L dependency=/data/home/asher/projects/pants/src/rust/engine/target/release/deps --extern async_lock=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libasync_lock-f3c40e7088c0c48f.rmeta --extern async_oncecell=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libasync_oncecell-ba76a37e063cdcf2.rmeta --extern async_trait=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libasync_trait-4aa641a94165ba90.so --extern bincode=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libbincode-d7ced88b617693f1.rmeta --extern bytes=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libbytes-9cc9c96cedb1e24c.rmeta --extern cache=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libcache-981fde84d4228c07.rmeta --extern concrete_time=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libconcrete_time-2f1249d69a527629.rmeta --extern deepsize=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libdeepsize-d7d1622868673ce8.rmeta --extern derivative=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libderivative-2619785e95535a53.so --extern fs=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libfs-1467bb6869e5e94e.rmeta --extern futures=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libfutures-f5900e09ff5253c6.rmeta --extern grpc_util=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libgrpc_util-cb81e9d7595f4c5b.rmeta --extern hashing=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libhashing-0c5c81344e6ef287.rmeta --extern itertools=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libitertools-b959b6a86b797ed6.rmeta --extern lazy_static=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/liblazy_static-96fed952b13c5817.rmeta --extern libc=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/liblibc-fb921b68e1797e44.rmeta --extern log=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/liblog-0e6f3cf61610f2cf.rmeta --extern nails=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libnails-69d3659aaf5b7f25.rmeta --extern nix=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libnix-72c9300964ad8386.rmeta --extern parking_lot=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libparking_lot-8494b1e05e5340f5.rmeta --extern prost=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libprost-b2632190cd94e73c.rmeta --extern prost_types=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libprost_types-766516f3f0692716.rmeta --extern protos=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libprotos-10288ea90b99fbda.rmeta --extern rand=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/librand-e522ca9e79e79f18.rmeta --extern regex=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libregex-16beee172cc64ee4.rmeta --extern serde=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libserde-af7124666ff28546.rmeta --extern sha2=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libsha2-6c4576a0a8a87561.rmeta --extern shell_quote=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libshell_quote-79baabf6b367c402.rmeta --extern store=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libstore-0a05c3d63588ad45.rmeta --extern strum=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libstrum-bf9b6d9c2899c7ed.rmeta --extern strum_macros=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libstrum_macros-6bbefd75229ed662.so --extern task_executor=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libtask_executor-9adf70db6244cadc.rmeta --extern tempfile=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libtempfile-7112cb76b40b1e08.rmeta --extern tokio=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libtokio-f474fdc1fa8c947e.rmeta --extern tokio_rustls=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libtokio_rustls-abb22639f413c4f0.rmeta --extern tokio_util=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libtokio_util-6b86a41b5a9e7bfe.rmeta --extern tonic=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libtonic-0d71199eb5c55548.rmeta --extern tryfuture=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libtryfuture-8db37b3fa3d16876.rmeta --extern uname=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libuname-2be2494598846851.rmeta --extern uuid=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libuuid-307326ec059951a0.rmeta --extern walkdir=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libwalkdir-ac8d7ebc8af51cf8.rmeta --extern workunit_store=/data/home/asher/projects/pants/src/rust/engine/target/release/deps/libworkunit_store-c1d0dcf514ffb29b.rmeta -L native=/data/home/asher/projects/pants/src/rust/engine/target/release/build/ring-f23070cefae4de00/out -L native=/data/home/asher/projects/pants/src/rust/engine/target/release/build/lmdb-rkv-sys-e2f79afb3647292a/out` (signal: 9, SIGKILL: kill)
warning: build failed, waiting for other jobs to finish...
error: build failed