https://pantsbuild.org/ logo
p

proud-dentist-22844

03/27/2023, 8:56 PM
From rule code, I would like to modify the exported venv whenever someone does
./pants export --resolve=st2
. Or perhaps just add another
--export-py-*
option. @happy-kitchen-89482 @curved-television-6568 what would you think of modifying this code to do the following: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/goals/export.py#L236-L256 • allow other rules to inject things into the digest (probably via a new union) ◦ Like this rule which could inject
*.dist-info/entry_points.txt
files for the `python_distribution`s in the current resolve: ▪︎ https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/framework/stevedore/rules.py#L43-L45 ◦ Or a new rule that could inject
.pth
files for each of the `python_distribution`s in the resolve (which would mean we don't have to mess with
PYTHONPATH
and is essentially the solution described in PEP 660 for editable wheels. • make
post_processing_cmds
more configurable somehow (maybe add more post processing commands before removing the reqs+pex pexes) ◦ like something that modifies the
PYTHONPATH
in the
activate
script to include the first party source roots included in that resolve (Instead of adding the .pth files above) ◦ adjust the requirements pex venv command line args (or maybe the env) so that I can specify the version of pip/setuptools By manipulating
PYTHONPATH
and adding
entry_points.txt
, we could use this to simulate editable installs of the first party sources.
h

happy-kitchen-89482

03/28/2023, 3:02 AM
Note that none of this can be allowed for symlinked venvs, so I assume this is all for newly-build venvs?
p

proud-dentist-22844

03/28/2023, 6:06 PM
correct. only for
mutable_virtualenv
.
h

happy-kitchen-89482

03/28/2023, 6:23 PM
If there's a way to do this somewhat streamlined? I fear turning this into a swiss army knife, but if we focus on 1-2 use cases here (e.g., simulating editable installs) to start?
p

proud-dentist-22844

03/28/2023, 6:23 PM
Yeah. simulating editable installs is my focus.
Studying this, I think the best extension point might be to allow rules to add files to the digest before the post-process command runs. Assumption I need to validate: I think we’ll be able to add files into the virtualenv before the virtualenv is actually created. If not, then the rules will have to add the files to a separate temp directory with another post-processing command that copies them into the virtualenv after it gets created.
As far as editing
post_processing_cmds
,
pex-tools
doesn’t have very many options for the
venv
sub command. I mentioned editing
post_processing_cmds
because I thought it would be something like the
virtualenv
command where I need to specify pip/setuptools/wheel versions. But, that’s not necessary - I’m adding those to my requirements instead (since they are runtime deps for st2), and then https://github.com/pantsbuild/pex/pull/2107 will allow my pex-provided versions to be used instead of whatever default
pip
/
setuptools
versions happen to get included in the venv thanks to the
--pip
flag. ps: I really want whatever version of pex includes that feature to make it into pants 2.16 if at all possible These pex-tools venv flags might also need to be configurable somehow: •
--prompt
(maybe we just need to set this and not make it configurable. I think it should have the resolve name instead of defaulting to the python version (the dir name of
dist/virtualenvs/<resolve>/<python_version>
like it does now) •
--non-hermetic-scripts
(not sure if the dev env might need this at some point) •
--copies
(this is only needed if some libs in the venv don’t work with the symlinks. I have encountered this in the past, though I don’t think I need it now) And then looking at the pex sources, this option could be interesting in pants if it gets exposed via pex-tools
venv
subcommand: •
--system-site-packages
(not sure if that would make sense for the dev environment) ◦ `Virtualenv.create(…, system_site_packages=False, …)`: https://github.com/pantsbuild/pex/blob/main/pex/venv/virtualenv.py#L141 ◦ which gets called in pex-tools
venv
here: https://github.com/pantsbuild/pex/blob/main/pex/tools/commands/venv.py#L190-L196
h

happy-kitchen-89482

03/28/2023, 10:58 PM
We could just optionally add all the source roots, no?
With no configurability?
p

proud-dentist-22844

03/29/2023, 12:13 AM
I don't understand what you mean by configurability. That other rules should not be able to inject things into the export? Or that the user shouldn't have to use more CLI flags?
h

happy-kitchen-89482

03/29/2023, 12:19 AM
I mean that letting users configure arbitrary post_processing_cmds seems like overkill? We can give you one option that adds all the source roots as "editable installs"?
p

proud-dentist-22844

03/29/2023, 12:21 AM
Yeah. Making that configurable doesn't provide as much value as I expected. My comments are, afaict, an exhaustive list of what value could be extracted from it (ie not much)
Creating editable installs is the way to go. Do we do that per source root or per
python_distribution
? Somehow it should be filtered to only include the source roots or `python_distribution`s if they contain something in the target resolve.
If a source root does not include a
python_distribution
, then we do not have enough info to add
dist-info
directories of an editable install. We could add the source roots to the
sys.path
via
.pth
files, but we risk adding source roots that are irrelevant, or, worse, that mask module names in an unrelated source root. So, I don't want to just add all the source roots to the path.
Extracting entry_point details from anything other than a
python_distribution
(like a
pex_binary
) seems very problematic, because we would not have a package name for where to put the metadata.
2 Views