Here’s a fun one. This turns off Gets entirely, if...
# development
h
Here’s a fun one. This turns off Gets entirely, if an env var is set: https://github.com/pantsbuild/pants/pull/22670
The env var is not currently set, so this should be a no-op. We can then set it in a branch and see how CI behaves.
f
Is there documentation (or, better yet, a utility) for plugin authors to convert their existing Gets into call-by-name?
h
The utility exists (
pants migrate-call-by-name <paths>
). The next step is for me to write that documentation…
That is my next task
f
Nice! Is that in 2.28 or should I wait for something newer?
w
Note: migrate-call-by-name doesn’t work out-of-repo yet. I explicitly made it work in-repo, as out of repo had some other issues. … I should probably update that
@happy-kitchen-89482 The top of this issue is the 80/20 on how that script/system works: https://github.com/pantsbuild/pants/issues/21065 I think really the harder part is when that fails, what to do (which is like, MOSTLY in the
Get(Process)
case when you have to manually move
implicitly
around, but there are the cases where even the rule graph transformer thing that Stu wrote won’t see the Gets - e.g. in rule_helpers and whatnot. I’d wager that the overwhelming majority of plugins should be super straight-forward by comparison
@fast-school-44220 The goal itself has been around for several versions now (https://github.com/pantsbuild/pants/blob/main/src/python/pants/goal/migrate_call_by_name.py)
👍 1
f
I'll give it a try next week.
h
@wide-midnight-78598 do you think it’s worth making that script work out of repo?
I am currently rewriting all the documentation… And we should create a migration page as well
w
In short, yes. Super helpful. I don’t think it’s that much work on the script side, I just hadn’t looked at it until we were done in the main repo. The workaround dirty hack would be to just move your plugin code temporarily into the main pants repo, migrate it, then copy it out again - but meh. I’m almost caught back up with day job, so will be looking at my Pants tasks starting this week
f
qq before the documentation is available: what's the call by name version of
MutliGet()
?
w
Concurrently
That’s a find/replace… err… and import
f
thanks. import from pants.engine.rules still?
w
Oh, yes, I think so - I guess it literally is a find/replace. If you need examples, check out the PRs from mid-May - a metric ton of changes during the CBN week: https://github.com/pantsbuild/pants/pulls?q=is%3Apr+“call-by-name”+author%3Asureshjoshi These are just mine because I named them all a certain way
is:pr “call-by-name” author:sureshjoshi
f
"No results matched your search"? But I'll just do the s// and see what happens.
w
Yeah, GitHub is weird - I think there’s a special way to copy out the escaped search, but yeah either way
f
hmm. apparently not:
ImportError: cannot import name 'Concurrently' from 'pants.engine.rules'
w
Lower case?
f
ah, much better!
thanks
w
concurrently
- my iPad thinks it knows best with my capitalization
I’d be curious if the
migrate-call-by-name
call works. I haven’t tried in quite some time, and was going to look into it later this week.
pants migrate-call-by-name pants-plugins/something::
or wherever. The code that the rule engine processes might give grief if it’s out of the pants main repository. It definitely did that when I was trying to work on the goal from mainline, to an out-of-source plugin
From this issue:
# Ensure the backend is enabled in pants.toml # backend_packages = [ # ... # "pants.backend.WHATEVER", # ... # ] pants migrate-call-by-name src/python/pants/backend/WHATEVER:: # Manually check for unnecessarily `implicitly`s and replace pants fix fmt lint check test src/python/pants/backend/WHATEVER::
f
I'm in the middle of changing the parallelism from horizontal to vertical, but once that's done, I'll try the Get conversion.
👍 1
w
I had intended to come back and mess around with plugins later this week, so if you want to maybe save some frustration, I can try running through the door first
“Later today” not later this week
f
I'm in meetings all afternoon, so that might just naturally happen
w
😆 Fair enough! I’ll report back in this thread any success or failure of note
Okay, as I suspected, call-by-name works fine in plugins, the migration goal falls over with the intentional error I left a while back.
Copy code
venvs/2.29.0rc0/lib/python3.11/site-packages/pants/backend/python/util_rules/entry_points.py' is not in the subpath of 'oss/pants-plugins' OR one path is relative and the other is absolute.
I need to finally dig into this, and see why the rule graph was all janky with external plugins.
f
Hi, SJ, any luck? I'm ready to start migrating my plugins
w
Just have a dayjob PR to finish, then I was hoping to look at it this afternoon
f
dayjobs. always getting in the way. :)
🎯 1
w
Okay, so in a bit of a surprise, this worked in my separate pants-plugins repo
Copy code
PANTS_SOURCE=../pants-from-github pants migrate-call-by-name pants-plugins/experimental/scie::
Trying to do the migration out of repo does suffer from some other problems to, which is the poor typechecking compared to mainline
@fast-school-44220 So, I ran a call-by-name on my repo here: https://github.com/sureshjoshi/pants-plugins/compare/main...call-by-name I did it using this line:
PANTS_SOURCE=../pants-from-github pants migrate-call-by-name pants-plugins/experimental/scie::
which involved cloning the main pants repo, building it, and then this. That worked, with only a couple of hiccups: • Updated the repo to a recent release • Updated my plugin resolve to use Python 3.11 •
pants generate-lockfiles --resolve=pants-plugins
pants export ::
- to use
./dist/export/python/virtualenvs/pants-plugins/3.11.13/
as my virtualenv in VSCode to detect certain problems ahead of time The only migration problem I had, was that
result = await Get(ProcessResult, Process, process)
this somehow got migrated into
result = await fallible_to_exec_result_or_raise(**implicitly(process))
which call-by-name can't type correctly. It needed to be
result = await fallible_to_exec_result_or_raise(**implicitly({ process: Process }))
or, what I eventually did (and what we do a lot in-repo)
Copy code
result = await fallible_to_exec_result_or_raise(
        **implicitly(
            Process(
                argv=argv,
                input_digest=input_digest,
                description="Run science on the input digests",
                output_files=output_files,
                level=LogLevel.DEBUG,
            )
        )
    )
Eventually, we'd also add an update from
fallible_to_exec_result_or_raise
to
execute_process_or_raise
which is an alias in the same module. I was hoping the method I tried wouldn't work, as that would make it easy to debug so that we can run that command on the release itself
f
one of my teammates just cloned and built the repo, conveniently. Let me see if I can borrow it from him and try this.
w
👍 I'd be curious to hear the results. Im going to fiddle around with some way to correctly reproduce the problem I'm trying to solve, without relying on releases
Candidate fix here for the migration script: https://github.com/pantsbuild/pants/pull/22701 - to test this, I just applied this change in my installed Pants venv, and it migrated my code
f
mostly worked, but I got one "failed to find dependencies" warning twice.
could that be because I was already calling it by name?
a quick review of the diffs looks like it did the right thing
which backend do I need active for the
pants fix fmt lint check test src/python/pants/backend/WHATEVER::
to clean up the `implicity`s?
w
That's more representative of what code you're doing. Since this was in the pants repo, and we have so many backends. If you're doing it in your own repo, I'd guess all the backends are already enabled.
mostly worked, but I got one "failed to find dependencies" warning twice.
Yeah, it's not 100% fool-proof, more like 80% and requires manual cleanup. Some of the items are actually just that the rule graph can't find a handful of items, depending on how they're ordered and whether they're nested, etc. Sometimes it's a cause of the AST work, which was targeting the bulk of the 5000 or so changes Pants had to make, but not all of them
f
oh, is cleaning up unneeded `implicitly`s something that any formatter can do?
w
yeah, any of the python ones
f
fwiw, my plugins still appear to be fully functional after using your goal to convert them to call by name. Now I just need a document to explain what
**implicitly()
is actually doing. 🙂
🔥 1
w
Now I just need a document to explain what
**implicitly()
is actually doing
Don't we all 😆 Hang on, Benjy did a big document re-write for the docs
"Fill in the blanks" is a pretty good way to describe it. A form a dependency injection, etc. Same idea as what was happening with the
Get
mechanism, but now it's more... ummm "explicily implicit"
f
thanks. that page is helpful.