https://pantsbuild.org/ logo
p

proud-dentist-22844

02/10/2023, 8:14 AM
@bitter-ability-32190 Does this asserted assumption mean that fmt plugins cannot add files? https://github.com/pantsbuild/pants/blob/2.15.x/src/python/pants/core/goals/fix.py#L392-L394 That conflicts with a feature (prob an accidental feature) I tried to expose as an official feature that was released in 2.14: https://github.com/pantsbuild/pants/pull/15846 Also, I think that's what is preventing me from changing elements in my fmt Batch to be FieldSets instead of files.
b

bitter-ability-32190

02/10/2023, 12:55 PM
As of today yes. Thanks for the link, that sparked my memory. The code is very hierarchical, which helps and hurts. But ultimately is to reduce mistakes. So linters are opaque in the handoff between partitioner and runner. Fixers (which include formatters) don't get the luxury because we have to run the tools serially so we have to reduce partitions into a common base.
That being said, I wouldn't be surprised if there's a way to hack it to work. I think what you'd need is a field set type whose Source isn't hydrated? Or is hydrated via your codegen?
But yeah, I think you're shoving a square peg in a round hole. The hole used to be bigger than the square, but it got narrower.
p

proud-dentist-22844

02/10/2023, 3:47 PM
Ok. I think I've got things sorted so I don't need the field set any more. But what about Fmt/fix plugins adding files?
b

bitter-ability-32190

02/10/2023, 3:50 PM
YMMV on that.
We can try and set up some time to discuss it in a call. Maybe figure where best something like that belongs
p

proud-dentist-22844

02/10/2023, 3:51 PM
Yes please.
I've beaten my head on this for awhile and would love some help. When might be best for you?
b

bitter-ability-32190

02/10/2023, 4:04 PM
Let's figure that out next week? Ive been OoO all week so I have no clue what next week looks like 😬
p

proud-dentist-22844

02/10/2023, 4:58 PM
LOL. The joys of after-vaca-catch-up. Let me know when you know 😄
Yes. That assertion prevents adding files.
Copy code
15:35:09.47 [WARN] Completed: Update contrib/schemas/*.json with st2-generate-schemas - st2-generate-schemas made changes.
  contrib/schemas/policy.json
15:35:09.47 [ERROR] 1 Exception encountered:

Engine traceback:
  in `fmt` goal

AssertionError: Expected ('contrib/schemas/action.json', 'contrib/schemas/alias.json', 'contrib/schemas/pack.json', 'contrib/schemas/policy.json', 'contrib/schemas/rule.json', 'contrib/schemas/sensor.json') to match ('contrib/schemas/action.json', 'contrib/schemas/alias.json', 'contrib/schemas/pack.json', 'contrib/schemas/rule.json', 'contrib/schemas/sensor.json')
b

bitter-ability-32190

02/10/2023, 9:38 PM
Yeah I'm not surprised. Fmt/Fox wasn't designed to add files
Remember that the core code has to serialize the calls and partitions as well. We need to know what we're operating on ahead of time 😟
p

proud-dentist-22844

02/10/2023, 9:45 PM
Now I'm thinking about adding a way for a subsystem to opt-in to being able to add files during fmt. (essentially something that switches that assert from
==
to
issuperset
b

bitter-ability-32190

02/10/2023, 9:47 PM
IMHO That seems like it's best handled through other means. Codegen or a new goak
p

proud-dentist-22844

02/10/2023, 9:47 PM
Nope
The whole point is it has to fit into the normal flow of what people do all the time
No new goal. No
./pants run
. No special script. Because we already have that (an ugly makefile target and a script) and people don't run it.
b

bitter-ability-32190

02/10/2023, 9:50 PM
Well in this case you aren't formatting or fixing files. Sooooooo
p

proud-dentist-22844

02/10/2023, 9:50 PM
Yeah - logically it fits into codegen... But codegen is so isolated it's not useful.
b

bitter-ability-32190

02/10/2023, 9:50 PM
I disagree. I actually use it a lot in our repo 🙂
p

proud-dentist-22844

02/10/2023, 9:50 PM
Restricting the export-codegen to only touch
dist/
makes it pretty worthless (for this usecase)
b

bitter-ability-32190

02/10/2023, 9:51 PM
I think there's a ticket for that 😉
p

proud-dentist-22844

02/10/2023, 9:54 PM
It would have to be some kind of codegen that runs during lint (just like fmt) and fails if codegen would make changes to the in-repo generated files. I suppose if we had something that prompted people to run
./pants export-codegen ...
similar to the prompt to run
./pants fmt
then that would work. I'm starting to burn out on this migration.
I'll attempt to be patient and wait till we can chat next week. Sorry to bother you.
b

bitter-ability-32190

02/10/2023, 9:55 PM
In this case id just wrap the pants invocation in a shell script. We gobble those like candy
p

proud-dentist-22844

02/10/2023, 9:55 PM
ick. no
No wrapping anything in shell scripts
I'm trying to delete all of those
"No more Makefiles!" is my mantra
b

bitter-ability-32190

02/10/2023, 9:56 PM
I guess I still don't understand how these are codegenerated but not in repo but also not codegenerated-the-pants-way
p

proud-dentist-22844

02/10/2023, 9:58 PM
The migration to pants is still in-progress (ouch. this is taking awhile) So, until that migration is complete, all the old infra has to continue working. Once I can show that all tests are passing both under a pants exported venv and under the old stuff, then I'll be able to start dropping the old Makefiles and actually lean into using pants more directly. As is, things have to work both under pants and under the Makefile.
b

bitter-ability-32190

02/10/2023, 9:58 PM
We have codegenerated files in repo. We check them with a unit test. If it doesn't match expected we give a nice error message about what to run
p

proud-dentist-22844

02/10/2023, 9:59 PM
It will be awhile before I can migrate the tests so that they can actually run under pytest and therefore be run by pants.
b

bitter-ability-32190

02/10/2023, 10:00 PM
But I still don't understand how fmt is supposed to format a file that doesn't exist before it's run
p

proud-dentist-22844

02/10/2023, 10:03 PM
It's more like "populate this directory" than "format this file". No matter what gets passed in, the correct files get spit out. If someone deletes a file, (or if they add code for a new type where there is no existing file), then it should just show up. The developer experience has been so bad, I'm trying to minimize the number of things people have to learn.
lint
fmt
and
test
are the only bits I'm planning to document, and then point interested folks at the pants docs for anything beyond that. Yeah. I think I'm just being a Karen at this point. I should get off my soap box -- the world does not revolve around my use cases.
b

bitter-ability-32190

02/10/2023, 10:04 PM
There's value and knowledge in every use case
p

proud-dentist-22844

02/13/2023, 7:32 PM
@bitter-ability-32190 when you get a chance, would you check out my proposal: https://github.com/pantsbuild/pants/discussions/18235 It builds on some of the insights from this thread.
👀 1
2 Views