oof, this is really bad. @curved-television-6568: if you have any additional information about how to reproduce this one, it would be really helpful

I’m on a relatively new Mac M1, using

╰─❯ pyenv version
3.9.13 (set by /Users/andreas.stenius/.pyenv/version)
3.8.13 (set by /Users/andreas.stenius/.pyenv/version)

but apart from that it’s nothing special. I did a repor off of

main

yesterday. I’m happy to run some diagnostics to help pin this down if you have any for me..

well: the first question is probably whether it is a busy wait (is the CPU pegged?) or a deadlock / very long hang

checking…

if the former, i’m curious where CPU time is being spent… if the latter, we’d need to figure out who is waiting for who, which might involve thread dumps

lacking emojis for emotions 😛 /me intrigued

I was not able to repro this on a Mac M2 FWIW

it’s a deadlock it looks like, no spike on CPU from what I can see on the Activity Monitor at least..

how do I get a thread dump on this thing? (learning new stuff today! 😄 )

oh, it’s in the docs… reading 😉

install `py-spy`: https://github.com/benfred/py-spy … then dump threads with

sudo py-spy dump -p $pid

… probably for the test process first

I’ll switch over to DM from here on…

or a new thread, yea

@curved-television-6568: continuing debugging of the hung test:

Hey @witty-crayon-22786, I've been thinking a lot about environment matchers and want to make sure I understand the motivating use cases:

It occurs to me we should also do the

partition

-ing schema for `fmt`/`lint` for

check

as well so that each are consistent. Unfortunately I'm starting to lose steam, so if anyone is interested in picking it up, LMK (and I'd VERY much like to keep this change in a single release, so timing does matter) Also RIP fmt/lint/check plugin authors. I'm so sorry 😭

Need some type hint help:

_GoalSubsystemT = TypeVar("_GoalSubsystemT", bound=GoalSubsystem)

async def _get_subpartitions(
    subsystem: type[_GoalSubsystemT],
): ...

...
    _get_subpartitions(lint_subsystem)

# Note: `LintSubsystem` is `class LintSubsystem(GoalSubsystem):`

yields

error: Argument 1 to "_get_subpartitions" has incompatible type "LintSubsystem"; expected "Type[<nothing>]"  [arg-type]

https://mypy.readthedocs.io/en/stable/kinds_of_types.html#the-type-of-class-objects makes me think what I'm doing is OK

(Short write-up of the `fmt`/`lint` changes, assuming the tool name is

Polite

) `lint.py`: • Plugins register rules using

*PoliteRequest.registration_rules()

(there are multiple unions being registered) • Plugins define 2 rules* (*future PR will provide a "default" implementation of the first rule if opted-into) ◦ A "partitioner" rule. This takes all the specs FieldSets (for targets) / files (for target-less), and returns

Partitions

: a mapping from

key: Any

to

elements: tuple[T, ...]

. (Note the elements can by anything) Most tools have an implementation of "if skip, return empty. Otherwise return one partition of all the inputs". Some plugins do actual partitioning though, and those get to do interesting thing. This is also an opportunity to do expensive work once. ◦ A "runner" rule. This rule takes a

SubPartition

(the key, and a subset of the partition) and returns a

LintResult

• The rules look like: ◦

PoliteRequest.PartitionRequest -> Partitions

◦

PoliteRequest.SubPartition -> LintResult

`fmt.py`: • Same idea as

lint.py

with one change: the partition elements are filepaths for both target-aware tools and targetless tools • The snapshot is retrieved in the "runner" rule by doing

await PoliteRequest.SubPartition.get_snapshot(request)

@fast-nail-55400 for

terraform fmt

is it OK if we're formatting a subset of files in a directory? E.g. if a directory has

[a, b, c]

and we fmt

[a, b]

in one process and

[c]

in another?

@witty-crayon-22786 Following our discussion the other day, I believe this does everything we want — let me know what you think: https://github.com/pantsbuild/pants/pull/16965

looks good: thanks.

@fast-nail-55400 I’m working on migrating options to the new runtime environments support, and am looking at the

GolangSubsystem

. There’s a bunch of options that look like they might be environment-sensitive, can you confirm that my suggestions here make sense? https://docs.google.com/document/d/1vXRHWK7ZjAIp2BxWLwRYm1QOKDeXx02ONQWvXDloxkg/edit#heading=h.f7dousqg1kef

PSA for docs: it is not safe to use new lines to split up source code, unless you actually want Readme.com to render those line breaks: https://github.com/pantsbuild/pants/pull/16976

another docs open question: we got feedback that it isn’t always obvious that tabbed code blocks might represent a complete working example:

I have two PRs coming in that I’d very much like to have in the next 2.14.x release if possible: https://github.com/pantsbuild/pants/pull/16934 https://github.com/pantsbuild/pants/pull/16977

IT LIVES! The formatter schema change PR is up: https://github.com/pantsbuild/pants/pull/16980 Say hello to targetless formatters (and very soon, say hello to

fix

)

Need some opinions on https://github.com/pantsbuild/pants/pull/16926 Specifically, the question of "Whether to add the

args

options to tools which don't take argv _today_" Pros of adding it: • Future-proof. The user who is using Pants today can upgrade the version of the tool to one that does take

argv

, and can now pass args ◦ Otherwise the user would have to either fork the support in-repo, or upstream the change and wait and upgrade Pants 😞 • Consistency across all tool subsystems. They all have

args

option Cons of adding it: • Funky UI for users of versions that don't take an argv. They try and pass something and the tool goes boom

@witty-crayon-22786 @hundreds-father-404 In the Environment Redesign doc, there’s a few instances of fields on targets that look like those fields are going to be environment-sensitive. Do we have a design for such a thing yet?

I’m reviewing the remaining Environment Design TODOs, and I have a question about

[GLOBAL].ca_certs_path

. This is a BootstrapOption, which is inherently non-environment-sensitive. There’s clearly places where we need an environment-sensitive value, but we also need a value which is fixed to the environment where Pants is being run (e.g. in

scheduler.py

). Is the solution here to add a new option for process-level certs paths (which defaults to

ca_certs_path

), or something else?

The

chevron==0.14.0

entry in

requirements.txt

has the comment: Should only be used by build-support. What is the context for that comment? (I'd like to use a templating library in a rule in the Go backend and using a templating library that is already in the lockfile would be preferred.)