https://pantsbuild.org/ logo
Join Slack
Powered by
From rule code, I would like to modify the exporte...
# development
p
From rule code, I would like to modify the exported venv whenever someone does 
./pants export --resolve=st2
. Or perhaps just add another 
--export-py-*
option. @happy-kitchen-89482 @curved-television-6568 what would you think of modifying this code to do the following: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/goals/export.py#L236-L256 • allow other rules to inject things into the digest (probably via a new union) ◦ Like this rule which could inject 
*.dist-info/entry_points.txt
files for the `python_distribution`s in the current resolve: ▪︎ https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/framework/stevedore/rules.py#L43-L45 ◦ Or a new rule that could inject 
.pth
files for each of the `python_distribution`s in the resolve (which would mean we don't have to mess with 
PYTHONPATH
and is essentially the solution described in PEP 660 for editable wheels. • make 
post_processing_cmds
more configurable somehow (maybe add more post processing commands before removing the reqs+pex pexes) ◦ like something that modifies the 
PYTHONPATH
in the 
activate
script to include the first party source roots included in that resolve (Instead of adding the .pth files above) ◦ adjust the requirements pex venv command line args (or maybe the env) so that I can specify the version of pip/setuptools By manipulating 
PYTHONPATH
and adding 
entry_points.txt
, we could use this to simulate editable installs of the first party sources.
h
Note that none of this can be allowed for symlinked venvs, so I assume this is all for newly-build venvs?
p
correct. only for 
mutable_virtualenv
.
h
If there's a way to do this somewhat streamlined? I fear turning this into a swiss army knife, but if we focus on 1-2 use cases here (e.g., simulating editable installs) to start?
p
Yeah. simulating editable installs is my focus.
Studying this, I think the best extension point might be to allow rules to add files to the digest before the post-process command runs. Assumption I need to validate: I think we’ll be able to add files into the virtualenv before the virtualenv is actually created. If not, then the rules will have to add the files to a separate temp directory with another post-processing command that copies them into the virtualenv after it gets created.
As far as editing 
post_processing_cmds
, 
pex-tools
doesn’t have very many options for the 
venv
sub command. I mentioned editing 
post_processing_cmds
because I thought it would be something like the 
virtualenv
command where I need to specify pip/setuptools/wheel versions. But, that’s not necessary - I’m adding those to my requirements instead (since they are runtime deps for st2), and then https://github.com/pantsbuild/pex/pull/2107 will allow my pex-provided versions to be used instead of whatever default 
pip
/ 
setuptools
versions happen to get included in the venv thanks to the 
--pip
flag. ps: I really want whatever version of pex includes that feature to make it into pants 2.16 if at all possible These pex-tools venv flags might also need to be configurable somehow: • 
--prompt
(maybe we just need to set this and not make it configurable. I think it should have the resolve name instead of defaulting to the python version (the dir name of 
dist/virtualenvs/<resolve>/<python_version>
like it does now) • 
--non-hermetic-scripts
(not sure if the dev env might need this at some point) • 
--copies
(this is only needed if some libs in the venv don’t work with the symlinks. I have encountered this in the past, though I don’t think I need it now) And then looking at the pex sources, this option could be interesting in pants if it gets exposed via pex-tools 
venv
subcommand: • 
--system-site-packages
(not sure if that would make sense for the dev environment) ◦ `Virtualenv.create(…, system_site_packages=False, …)`: https://github.com/pantsbuild/pex/blob/main/pex/venv/virtualenv.py#L141 ◦ which gets called in pex-tools 
venv
here: https://github.com/pantsbuild/pex/blob/main/pex/tools/commands/venv.py#L190-L196
h
We could just optionally add all the source roots, no?
With no configurability?
p
I don't understand what you mean by configurability. That other rules should not be able to inject things into the export? Or that the user shouldn't have to use more CLI flags?
h
I mean that letting users configure arbitrary post_processing_cmds seems like overkill? We can give you one option that adds all the source roots as "editable installs"?
p
Yeah. Making that configurable doesn't provide as much value as I expected. My comments are, afaict, an exhaustive list of what value could be extracted from it (ie not much)
Creating editable installs is the way to go. Do we do that per source root or per 
python_distribution
? Somehow it should be filtered to only include the source roots or `python_distribution`s if they contain something in the target resolve.
If a source root does not include a 
python_distribution
, then we do not have enough info to add 
dist-info
directories of an editable install. We could add the source roots to the 
sys.path
via 
.pth
files, but we risk adding source roots that are irrelevant, or, worse, that mask module names in an unrelated source root. So, I don't want to just add all the source roots to the path.
Extracting entry_point details from anything other than a 
python_distribution
(like a 
pex_binary
) seems very problematic, because we would not have a package name for where to put the metadata.
3 Views
Open in Slack
PreviousNext
Powered by