https://pantsbuild.org/ logo
#development
Title
# development
p

proud-dentist-22844

04/06/2023, 7:35 PM
I’ve got a change that makes pants do editable installs of all relevant `python_distribution`s when exporting a mutable virutalenv (which essentially turns pants into a PEP-660 frontend and backend - albeit delegating a few pieces of that to pip and to the configured PEP-517 backend). Or at least I’ve got all of the export tests passing now, including integration tests. I haven’t used this on the st2 repo yet 🙂 Scope of my change: This is only possible fore mutable venvs, so symlinked venvs will not get this feature. Tool resolves (which are deprecated in 2.16) do not ever have dists, so this does not apply there either. And I’m not touching the legacy export codepath (where you do
export <addresses>
) that will stay as is; this will only apply when using
export --resolve=
). So, here’s my question: Should we have a new export option to enable/disable editable installs? If so, should it default to enabled or disabled (disabled is the backwards compatible option)? Or should we always do editable install of dists when exporting a resolve that includes dists?
@curved-television-6568 I believe you mentioned in an issue that you tried to do something like this - would you want this to always be enabled?
c

curved-television-6568

04/06/2023, 7:41 PM
what’s the downside to having it always enabled? It’s been quite a while since I needed this, and it may come up again but not something I’d need on a regular basis for sure.
p

proud-dentist-22844

04/06/2023, 7:46 PM
python_distribution
does not have a
resolve
field. So figuring out which resolve a
python_distribution
belongs to is expensive: calculating the owned deps of all distributions, and for each distribution, look through those deps until one of them has a resolve field, and use that for that dist’s resolve. Plus there’s the cost of building the PEP-660 wheels - if the configured PEP-517 build backend doesn’t support the PEP-660 methods, then it falls back to a method that is, sadly, optional in PEP-517. If that method isn’t there, then it falls back to telling that backend to build the whole wheel, and then it extracts just the dist-info directory from it and discards the rest. So, installing these editable wheels isn’t free. It’ll slow down the export, though I’m not sure by how much.
c

curved-television-6568

04/06/2023, 7:50 PM
hmm… feels like an option is a good idea then; possibly defaulting to
disabled
based on the relatively few times we’ve seen this feature requested..?
p

proud-dentist-22844

04/06/2023, 7:51 PM
Yeah. I would enable that option in pants.toml for the st2 project, but 🤷 others apparently don’t need it as much?
👍 1
What should the option name be? •
--include-editable-local-dists
--include-editable-dists
--editable
--include-pep660-local-dists
--include-pep660-dists
--pep660
• something else?
😬 1
@happy-kitchen-89482 wdyt?
oh. Should this be one of those
resolve_to
options? I’m thinking yes. hmm. that’s more complex.
h

happy-kitchen-89482

04/06/2023, 8:48 PM
Is it a property of the resolve though, or of the python_distribution?
p

proud-dentist-22844

04/06/2023, 8:52 PM
oh. I hadn’t thought about adding a field to
python_distribution
to control inclusion.
The resolve gets exported atomically (basically), so the option has to be set at the resolve level. But I suppose individual
python_distribution
targets could opt out of such inclusion.
h

happy-kitchen-89482

04/06/2023, 8:59 PM
Oh, right, this is when exporting an entire resolve
Oh wait, but why do we even need python_distributions? can we not "export" the relevant source roots as editable installs? Or does that require a setup.py?
I have no idea how editable installs work
p

proud-dentist-22844

04/06/2023, 9:00 PM
We need to be able to generate the
dist-info
directory with the PEP-517 backend, so we need the
provides=python_artifact()
details.
An editable install consists of the
dist-info
directory and a
.pth
file that injects the source directory into the PYTHONPATH (or more accurately, into
sys.path
).
h

happy-kitchen-89482

04/06/2023, 9:19 PM
gotcha
p

proud-dentist-22844

04/06/2023, 9:20 PM
Technically, we could also inject a
.pth
file that just adds all of the source roots (that contain code in the exported resolve). But people can set PYTHONPATH for that as well. The major value of editable installs is to get the
dist-info
details.
@happy-kitchen-89482 and @curved-television-6568 Could I get some feedback on https://github.com/pantsbuild/pants/pull/18639 before I mark it ready for review?
I’m waiting to mark it ready for review because: one part of it, the refactoring in
pants.backend.python.util_rules.setup_py
should probably go in a separate PR. If it does, then it could be backported to 2.16 as a behavior-less refactor. Doing that means that I can copy a lot of the code in the PR into a plugin for use in the st2 repo while I wait for 2.17 to stabilize enough to upgrade.
But I would like feedback on the approach first (before splitting off the refactor PR and before creating the plugin).
I marked it as ready for review. I'm very excited about this change. 😄
👀 1