https://pantsbuild.org/ logo
#pex
Title
a

aloof-angle-91616

07/29/2018, 8:19 PM
(this isn't necessarily a pex question, but it is about python packaging in pants -- let me know if i should move to #general) i'm trying to adding an option that evaluates
setup.py
files for pants
python_dist()
targets in a context that gives them access to pants code (in this case, so that the setup.py can import a pants module which extends distutils/setuptools classes. it is entirely possible to write the distutils/setuptools extensions as a completely separate pypi module, as they don't currently need to rely on other parts of pants, and could then just be accessed using
setup_requires
in the
python_dist()
target (and this is actually pretty reasonable). however, it seems like it should be possible to somehow expose the running pants environment to the setup.py invocation we do in
BuildLocalPythonDistributions
and/or
SetupPyRunner
, so the setup.py can just do:
Copy code
from setuptools import find_packages
from distutils.core import Extension
from pants.some.module import setup

setup(
  # ...,
  ext_modules=[Extension(...)],
  packages=find_packages(),
I assume this would be done either by creating a pex with the pants requirement, or by adding something to
PYTHONPATH
in the executing environment for the setup.py. this probably isn't that hard when using pants in any other repo besides the pants repo because we can expect a pex with all the relevant pants code to be available, since that is how pants is released, but i can't figure out how to expose the python code from the running pants to a setup.py invocation (i may be trying to go about this all wrong). there's a very little more context in https://github.com/pantsbuild/pants/issues/5661
e

enough-analyst-54434

07/29/2018, 8:30 PM
Right now setup.py is run directly with a python interpreter with minimal sys.path modification via Installers. You'd want to just switch to building a loose pex of the python_dist and executing setup.py using that pex
If the python_dist has pants deps, they be included in the PEX and thus visible to setup.py
a

aloof-angle-91616

07/29/2018, 9:21 PM
i just realized we could just make the
python_dist()
depend on a
pants_requirement()
in the
setup_requires
kwarg in the BUILD file which already works great thanks to chris -- my concern was how to make it reflect the running version of pants without having to write down the version string anywhere, mostly
e

enough-analyst-54434

07/29/2018, 9:42 PM
You need nothing so indirect. Simply depending on pants targets is enough.
a

aloof-angle-91616

07/29/2018, 9:43 PM
well,
python_dist()
can't depend on
python_library()
yet
e

enough-analyst-54434

07/29/2018, 9:43 PM
Aha, ok. That should probably be fixed.
👍 1
a

aloof-angle-91616

07/29/2018, 9:43 PM
currently it's just c/c++ targets actually
yes
e

enough-analyst-54434

07/29/2018, 9:44 PM
The SetupPy task has the logic needed. That could be extracted.
a

aloof-angle-91616

07/29/2018, 9:44 PM
i would do it but i don't know the model of what would be expected to happen (would we require the
python_library()
to have a
provides=
? would we copy the sources over into the dist before building?)
and ok
oh wow ok i'm looking at
SetupPy
now and this is great
e

enough-analyst-54434

07/29/2018, 9:46 PM
Yeah, see the model in SetupPy. The graph is walked and and dependencies that do not provides are sucked in, if they do provides, they're added as install_requires.
a

aloof-angle-91616

07/29/2018, 9:46 PM
this is all very exciting
that's interesting, i always was under the impression every transitive
python_library()
needed to have a
provides=
to be setup.py-able. i'm not sure why i thought that, but the description you just gave sounds optimal to me
a

aloof-angle-91616

07/29/2018, 9:50 PM
great!
i just put up this PR: https://github.com/pantsbuild/pants/pull/6273, which is different than what we have been discussing in this thread. just to be clear. going to look at fixing
setup_requires
and then circle back around to this