proud-dentist-22844
03/27/2023, 8:56 PM./pants export --resolve=st2
. Or perhaps just add another --export-py-*
option.
@happy-kitchen-89482 @curved-television-6568 what would you think of modifying this code to do the following:
https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/goals/export.py#L236-L256
• allow other rules to inject things into the digest (probably via a new union)
◦ Like this rule which could inject *.dist-info/entry_points.txt
files for the `python_distribution`s in the current resolve:
▪︎ https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/framework/stevedore/rules.py#L43-L45
◦ Or a new rule that could inject .pth
files for each of the `python_distribution`s in the resolve (which would mean we don't have to mess with PYTHONPATH
and is essentially the solution described in PEP 660 for editable wheels.
• make post_processing_cmds
more configurable somehow (maybe add more post processing commands before removing the reqs+pex pexes)
◦ like something that modifies the PYTHONPATH
in the activate
script to include the first party source roots included in that resolve (Instead of adding the .pth files above)
◦ adjust the requirements pex venv command line args (or maybe the env) so that I can specify the version of pip/setuptools
By manipulating PYTHONPATH
and adding entry_points.txt
, we could use this to simulate editable installs of the first party sources.happy-kitchen-89482
03/28/2023, 3:02 AMproud-dentist-22844
03/28/2023, 6:06 PMmutable_virtualenv
.happy-kitchen-89482
03/28/2023, 6:23 PMproud-dentist-22844
03/28/2023, 6:23 PMpost_processing_cmds
, pex-tools
doesn’t have very many options for the venv
sub command.
I mentioned editing post_processing_cmds
because I thought it would be something like the virtualenv
command where I need to specify pip/setuptools/wheel versions. But, that’s not necessary - I’m adding those to my requirements instead (since they are runtime deps for st2), and then https://github.com/pantsbuild/pex/pull/2107 will allow my pex-provided versions to be used instead of whatever default pip
/ setuptools
versions happen to get included in the venv thanks to the --pip
flag. ps: I really want whatever version of pex includes that feature to make it into pants 2.16 if at all possible
These pex-tools venv flags might also need to be configurable somehow:
• --prompt
(maybe we just need to set this and not make it configurable. I think it should have the resolve name instead of defaulting to the python version (the dir name of dist/virtualenvs/<resolve>/<python_version>
like it does now)
• --non-hermetic-scripts
(not sure if the dev env might need this at some point)
• --copies
(this is only needed if some libs in the venv don’t work with the symlinks. I have encountered this in the past, though I don’t think I need it now)
And then looking at the pex sources, this option could be interesting in pants if it gets exposed via pex-tools venv
subcommand:
• --system-site-packages
(not sure if that would make sense for the dev environment)
◦ `Virtualenv.create(…, system_site_packages=False, …)`: https://github.com/pantsbuild/pex/blob/main/pex/venv/virtualenv.py#L141
◦ which gets called in pex-tools venv
here: https://github.com/pantsbuild/pex/blob/main/pex/tools/commands/venv.py#L190-L196happy-kitchen-89482
03/28/2023, 10:58 PMproud-dentist-22844
03/29/2023, 12:13 AMhappy-kitchen-89482
03/29/2023, 12:19 AMproud-dentist-22844
03/29/2023, 12:21 AMpython_distribution
? Somehow it should be filtered to only include the source roots or `python_distribution`s if they contain something in the target resolve.python_distribution
, then we do not have enough info to add dist-info
directories of an editable install.
We could add the source roots to the sys.path
via .pth
files, but we risk adding source roots that are irrelevant, or, worse, that mask module names in an unrelated source root. So, I don't want to just add all the source roots to the path.python_distribution
(like a pex_binary
) seems very problematic, because we would not have a package name for where to put the metadata.