https://pantsbuild.org/ logo
#general
Title
# general
p

proud-dentist-22844

03/27/2023, 4:13 PM
I have pants generating
setup.py
for wheels (
python_distribution
). In an exported venv,
./pants export ...
, I would like to use editable mode installations of all the exported resolve's code (Ie the code that will end up in that resolve's `python_distribution`s). So, is there a way to get the pants-generated
setup.py
and do something like
setup.py develop
.? Or perhaps a PEP 660 install? One constraint to the solution: Setuptools gained support for PEP 660 in v64, but that is python 3.7+, and I need python 3.6 for the StackStorm project (for now - I need to figure out editable install to unblock some other things before we can work on finally dropping 3.6).
e

enough-analyst-54434

03/27/2023, 4:15 PM
I think you're on your own here blazing new trails @proud-dentist-22844.
p

proud-dentist-22844

03/27/2023, 4:16 PM
Not the first time 😛
Does pex have any kind of editable wheel support?
c

curved-television-6568

03/27/2023, 4:17 PM
I’ve been down this path too in the past: https://github.com/pantsbuild/pants/issues/11812#issuecomment-810831415 😉 (long since abandoned so have no up-to-date ideas.. 😕 )
p

proud-dentist-22844

03/27/2023, 4:18 PM
(BTW, this is slightly different than my venv request from a few days ago, which was more about final installation, whereas this is explicitly for development, so the exported venv is just fine if I can get it to work)
e

enough-analyst-54434

03/27/2023, 4:22 PM
@proud-dentist-22844 I'm not sure what that would mean even since a PEX file is really a lockfile + contents. You freeze things in and so editable makes roughly no sense. You can use local project directories as input requirements, which snapshots the local project at that point in time - but the project is snapshotted, edits won't interfere with the now hermetic PEX capture of the project.
What does one use editable mode for? There is
pants run
and
pants repl
- is it a 3rd case, or just because those 2 cases are too slow?
p

proud-dentist-22844

03/27/2023, 4:27 PM
Yup. this would be a very non-hermetic thing that would only really make sense in venv mode. I went through all of the various
--help
options in
pex
and
pex-tools
(and subcommands) to see what flexibility pex has. I'm not sure where editable anything might be shoehorned into pex unless maybe pex could be a PEX 660 frontend and install the PEX 660 wheels? Though I guess it would use pip for that. Sorry for the train of thought.
e

enough-analyst-54434

03/27/2023, 4:30 PM
I guess you're hacking around our provided
pants export
hack for IDE hacky support?
If so the hack tower is too tall. There is work on a proper VS Code plugin from SJ.
p

proud-dentist-22844

03/27/2023, 4:31 PM
My problem is: I can't yet migrate tests to run under pants (in particular I need to migrate from nosetest to pytest and ST2's use of eventlet makes migrations like that more difficult). Today, the makefile creates a virtualenv, installs all the requirements, and then runs
setup.py develop
for each of the wheels. I want to replace that venv with one generated somehow via pants so that the current Makefile can still be used to run tests while transitioning over to pytest + pants. And that means that I need editable installs. Also, our code relies heavily on setuptools metadata (esp entrypoints that are managed by openstack/stevedore), so PYTHONPATH alone is insufficient for testing our code.
e

enough-analyst-54434

03/27/2023, 4:32 PM
Oh Jesus. Ok. You're having to deal with crazy land.
Yeah - not sure what to say about that. You are definitely on your own here.
p

proud-dentist-22844

03/27/2023, 4:32 PM
The entire ST2 migration to Pants has been crazy land. And I'm apparently the mayor there...
e

enough-analyst-54434

03/27/2023, 4:33 PM
Ha!
As far as PEP-660 - yeah, that would only apply in
--venv
mode, but it still doesn't really apply. The whole point of a PEX is to pack it up and ship it; so where it lands will almost never have the local project checked out in the same relative location for symlinking.
p

proud-dentist-22844

03/27/2023, 4:35 PM
Yeah, but now pants use of PEX is pushing it squarely into the "development" phase instead of just the "deployment" or "install" phase of software dev lifecycle.
Really pushing its limits...
e

enough-analyst-54434

03/27/2023, 4:35 PM
Well, the venv is the right tool. That's the place to solve this for sure.
So instead of the develop mode, venv + PYTHONPATH=subproj1:subproj2 nets the same thing - is that right?
That should give live-edit with correct sys.path linking local project 1st party code to its 3rdparty reqs.
Meaning - I'd think the Makefile could work that way too. Obviously it would require edits which you may be loathe to do - but something is going to need edits here.
p

proud-dentist-22844

03/27/2023, 4:40 PM
venv + PYTHONPATH=subproj1:subproj2 nets the same thing - is that right?
Nope. not quite. We need the egg-info
dist-info
directories to expose the setuptools entry points.
e

enough-analyst-54434

03/27/2023, 4:40 PM
God.
p

proud-dentist-22844

03/27/2023, 4:41 PM
I like Him. 🙂
e

enough-analyst-54434

03/27/2023, 4:41 PM
I do not!
I just check in my dist-info and treat it as a resource.
Your python_distribution need not depend on that style thing. Perhaps a tad copypasta, but maybe an out?
p

proud-dentist-22844

03/27/2023, 4:43 PM
I wonder if I can hook into
export
somewhere to inject the
entry_points.txt
files... I already have the logic to generate that: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/framework/stevedore/rules.py#L43-L45
If I can generate the
entry_points.txt
files, then venv + PYTHONPATH would be sufficient.
e

enough-analyst-54434

03/27/2023, 4:44 PM
Or if you do like I suggested and stomach the non-DRY. That will work too.
p

proud-dentist-22844

03/27/2023, 4:44 PM
I just check in my dist-info and treat it as a resource.
Fascinating idea. I'll cogitate on it.
I doubt I can get a copy/pasta solution merged into ST2.
e

enough-analyst-54434

03/27/2023, 4:45 PM
Ok. Well - good luck to you. This is all very ... I think you'll continue to be voted in as Mayor for life.
p

proud-dentist-22844

03/27/2023, 4:46 PM
LOL
c

curved-television-6568

03/27/2023, 4:50 PM
I’m willing to sit as second chair in your town hall @proud-dentist-22844 .. as my linked to issue comment was based in my desire to leverage precisely that too (setup entry_points data)
p

proud-dentist-22844

03/27/2023, 4:51 PM
Welcome! LOL
🙃 1
2 Views