https://pantsbuild.org/ logo
Powered by
h

happy-kitchen-89482

09/20/2021, 10:24 PM
I'm taking a quick look at https://github.com/pantsbuild/pants/issues/12930, any thoughts on how to specify "build system requirements"?
I.e., requirements that need to be in the environment in which we run setuptools to create the dist?
So 
numpy
in the example in that ticket (because numpy builds fortran extensions for you)
If we create a 
build_dependencies
field that is a 
SpecialCasedDependencies
then that is nice and uniform - you specify build deps as targets
But then project introspection goals will see those as dependencies of the python_distribution, which is not the right way to look at them
It also allows 
build_dependencies
to be not just python requirements, but any code we can install into a virtualenv, opening the door to having custom build tools (in the PEP 518 sense) in the repo.
But the simple thing to do for right now is just have a list-of-strings-valued 
build_system_requirements
field that we treat as opaque and copy into a generated 
pyproject.toml
[build-system].requires
Or we can say you have to have an explicit 
pyproject.toml
with the build requirements
But then, where does that live? If it's at the repo root then those requirements will apply to every distribution, which is probably not what you want
Anyway, thoughts?
e

enough-analyst-54434

09/20/2021, 10:31 PM
The ticket leads with "handwritten" - is that so? If so I say require pyproject.toml instead of setup.py if you have build requirements because setup.py setup_requires (the progenitor of the PEPs), is widely deemed unsafe.
➕ 1
Is that sort of hard line too hard?
h

happy-kitchen-89482

09/20/2021, 10:33 PM
Well, I think we can allow pyproject.toml as well as setup.py? So that's not so hardline
i.e., pep 518 without pep 517
I think that's a thing?
e

enough-analyst-54434

09/20/2021, 10:33 PM
It is.
h

happy-kitchen-89482

09/20/2021, 10:34 PM
setuptools will take the pyproject.toml into account?
That might be the easiest path to a working solution
e

enough-analyst-54434

09/20/2021, 10:35 PM
It's the other way around. The pyproject.toml takes precedence when both are present. For example: https://github.com/pypa/pip/blob/9c3286873b5af7038ddc09fe034737bc7500a2be/pyproject.toml#L1-L3 https://github.com/pypa/pip/blob/main/setup.py
👍 1
h

happy-kitchen-89482

09/20/2021, 10:35 PM
Basically take any and all of your 
setup.py
, 
setup.cfg
, 
pyproject.toml
throw them in a virtualenv and let God sort them out...
e

enough-analyst-54434

09/20/2021, 10:35 PM
So that's Pip using a modern wrapper on their old setup.py build.
h

happy-kitchen-89482

09/20/2021, 10:36 PM
So that is using pep 517 then
in a minimal way
IIUC
e

enough-analyst-54434

09/20/2021, 10:36 PM
pyproject.toml doesn't need a venv per-say. A PEP-compliant builder is supposed to create those and populate them as needed to do the build.
h

happy-kitchen-89482

09/20/2021, 10:36 PM
ok, so I think you're on to something
e

enough-analyst-54434

09/20/2021, 10:36 PM
I ~refuse to remember 517 vs 518. One of those, yeah.
h

happy-kitchen-89482

09/20/2021, 10:37 PM
518 is "describe the build requirements", 517 is "describe how to run the builder"
e

enough-analyst-54434

09/20/2021, 10:37 PM
Yeah - It was never clear to me why seperate. I don't know how you meaningfully use just one of those.
➕ 1
h

happy-kitchen-89482

09/20/2021, 10:54 PM
But then what entry point do you run, if not your setup.py?
It looks like the canonical thing that consumes pep 517/518 is pip, when it builds sdists into wheels at install time
But that's not the case here
h

hundreds-father-404

09/20/2021, 10:58 PM
other installers like Poetry know how to consume PEP 517/518 as well
h

happy-kitchen-89482

09/20/2021, 11:00 PM
sure, but we are not installing anything here
we are building for deployment
in pep 517 parlance, it's not clear to me what the "build frontend" is here
the driving use-case in PEP 517 is building a wheel from an sdist, in pip
w

witty-crayon-22786

09/20/2021, 11:03 PM
But then project introspection goals will see those as dependencies of the python_distribution, which is not the right way to look at them
i’m less sure about this: it depends why you are using project introspection, but a lot of folks use it in concert with 
--changed-since
to implement CI. for that usecase, you do want these deps.
e

enough-analyst-54434

09/20/2021, 11:03 PM
Use Pex
Erm, that will require two commands.
I'd just use pip.
w

witty-crayon-22786

09/20/2021, 11:05 PM
(relatedly: i think that there is a connection between SpecialCasedDependencies and “scopes” in JVM land / “extras” in python land: scopes/extras allow for transitive filtering of the graph while walking it… Dependencies could be subtype-based filtering, maybe)
h

happy-kitchen-89482

09/20/2021, 11:06 PM
yeah, that scopes/extras thought occurred to me
👍 1
Using pip breaks our current target model (where we provide 
setup_py_commands
in the 
python_distribution
target) although maybe that's fine
And I'm not sure how we'd use pip to generate a deployable artifact (as opposed to installing one in the venv in which we run pip)
e

enough-analyst-54434

09/20/2021, 11:10 PM
pip wheel
- but hang on a sec for something more minimal
h

happy-kitchen-89482

09/20/2021, 11:12 PM
What about sdists?
e

enough-analyst-54434

09/20/2021, 11:14 PM
Experiment setup:
Copy code
$ python -mvenv /tmp/build.test.venv
$ source /tmp/build.test.venv/bin/activate
$ pip install build
$ cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools", "wheel", "requests"]
build-backend = "setuptools.build_meta"
EOF
$ cat <<EOF > setup.py
import requests
import wheel
from setuptools import setup

setup(name="foo", version="0.1.0")
EOF
Run experiment:
Copy code
$ python -mbuild .
* Creating venv isolated environment...
* Installing packages in isolated environment... (requests, setuptools, wheel)
* Getting dependencies for sdist...
running egg_info
creating foo.egg-info
writing foo.egg-info/PKG-INFO
writing dependency_links to foo.egg-info/dependency_links.txt
writing top-level names to foo.egg-info/top_level.txt
writing manifest file 'foo.egg-info/SOURCES.txt'
reading manifest file 'foo.egg-info/SOURCES.txt'
writing manifest file 'foo.egg-info/SOURCES.txt'
* Building sdist...
running sdist
...
writing manifest file 'foo.egg-info/SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Building wheel...
running bdist_wheel
...
adding 'foo-0.1.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Successfully built foo-0.1.0.tar.gz and foo-0.1.0-py3-none-any.whl
Where 
build
is from PyPA.
Copy code
$ python -mbuild -h
usage: python -m build [-h] [--version] [--sdist] [--wheel] [--outdir OUTDIR] [--skip-dependency-check] [--no-isolation] [--config-setting CONFIG_SETTING] [srcdir]

    A simple, correct PEP 517 package builder.

    By default, a source distribution (sdist) is built from {srcdir}
    and a binary distribution (wheel) is built from the sdist.
    This is recommended as it will ensure the sdist can be used
    to build wheels.

    Pass -s/--sdist and/or -w/--wheel to build a specific distribution.
    If you do this, the default behavior will be disabled, and all
    artifacts will be built from {srcdir} (even if you combine
    -w/--wheel with -s/--sdist, the wheel will be built from {srcdir}).

positional arguments:
  srcdir                source directory (defaults to current directory)

optional arguments:
  -h, --help            show this help message and exit
  --version, -V         show program's version number and exit
  --sdist, -s           build a source distribution (disables the default behavior)
  --wheel, -w           build a wheel (disables the default behavior)
  --outdir OUTDIR, -o OUTDIR
                        output directory (defaults to {srcdir}/dist)
  --skip-dependency-check, -x
                        do not check that build dependencies are installed
  --no-isolation, -n    do not isolate the build in a virtual environment
  --config-setting CONFIG_SETTING, -C CONFIG_SETTING
                        pass options to the backend.  options which begin with a hyphen must be in the form of "--config-setting=--opt(=value)" or "-C--opt(=value)"
So, this would not work with 
setup_py_commands
but that's part of what you buy with a decision to force PEP-517/518 for builds. You leave the distutils commands infra behind.
I mean, you stil use it under the covers, but it's purposefully and properly buried.
Something like Poetry or Pip or what have you is too much tool for this.
h

happy-kitchen-89482

09/20/2021, 11:19 PM
Yah
build
seems like the answer
c

curved-television-6568

09/21/2021, 4:58 AM
+1 for labeled edges in the target graph, and for ‘pants dependencies’ to optionally show and/or filter on those labels. (Regarding new specialcased field)
👍 1
Powered by