Might be a simple question but anyone know what `E...
# general
a
Might be a simple question but anyone know what
ERROR: Links are not allowed as constraints
means?
h
That's an error from pip, right?
e
Via https://bugs.launchpad.net/tempest/+bug/1912178 -> https://bugs.launchpad.net/devstack/+bug/1906322/comments/2 it looks like you have things other than (pinned) requirements in a constraints file. Like Pip flags.
a
yeah it is coming from pip
e
@acoustic-librarian-3937 you'd ideally be seeing the same output quoted in the second bug, but there is a sordid story there. Pex is currently hiding that from you (though not by choice), making debugging less obvious than it need be. The sordid details are documented in the issues below, but, in short, Pip has an effectively random mix of stderr and stdout usage and this is hard to deal with without writing a parser to grep out (known) error messages: + https://github.com/pantsbuild/pex/issues/1267 + https://github.com/pypa/pip/issues/9420
a
I think it is mad about lines like this?
Copy code
boto3 @ file:///Users/rmulcahy/Library/Caches/pypoetry/artifacts/ff/2e/6c/7b182c791dc935494d3a1b73699181129dcfb6f38eec4532ddc91e9901/boto3-1.20.42-py3-none-any.whl
e
Yup!
A-OK in a requirements file, no longer OK in a constraints file.
a
weird, my constraints file is generated by a call to
pip freeze --all
seems strange that it would output something it cannot consume
e
It can consume, just not as --constraint, only as -r
a
I wonder if somehow I used a different version of pip to generate it by mistake
e
They are intended to serve different purposes.
That's not it Ryan.
pip freeze
just prints what is installed in your venv. That information can be used by:
-r, --requirement <file>    Install from the given requirements file. This option can be used multiple times.
but it can no longer always be used by
-c, --constraint <file>     Constrain versions using the given constraints file. This option can be used multiple times.
because a file path is not a version.
a
ah ok
thanks
the pants docs are down rn, but we just copied in a script to generate the constraints from the suggestions there. It looks like this
Copy code
#!/usr/bin/env bash
# <https://www.pantsbuild.org/docs/python-third-party-dependencies#tip-set-up-a-virtual-environment-optional>
# this script was modified to use `virtualenv` instead of `venv`
# this script was modified to check if `jq` is installed
set -euo pipefail
set -x

if ! command -v jq &> /dev/null
then
    echo "jq could not be found"
    exit
fi

# You can change these constants.
PYTHON_BIN=python3
VIRTUALENV=build-support/.venv
REQUIREMENTS_FILE=requirements.txt
CONSTRAINTS_FILE=constraints.txt

"${PYTHON_BIN}" -m virtualenv --version || "${PYTHON_BIN}" -m pip install virtualenv
"${PYTHON_BIN}" -m virtualenv "${VIRTUALENV}"
. "${VIRTUALENV}"/bin/activate
pip install --upgrade pip

# Install all our requirements.txt, and also any 3rdparty
# dependencies specified outside requirements.txt, e.g. via a
# handwritten python_requirement_library target.
pip install \
  -r "${REQUIREMENTS_FILE}" \
  -r <(./pants dependencies :: |
    xargs ./pants filter --target-type=python_requirement |
    xargs ./pants peek |
    jq -r '.[]["requirements"][]')
echo "# Generated by build-support/generate_constraints.sh on $(date)" > "${CONSTRAINTS_FILE}"
pip freeze --all >> "${CONSTRAINTS_FILE}"
Should I modify this to generate my constraints file differently?
e
I think we have warnings about direct references (https://www.python.org/dev/peps/pep-0440/#direct-references) and VCS requirements in that doc but forget.
Ok, just so I don't assume wrongly, can you provide all the python configuration in your pants.toml? I forget all the wiring here - its complex - and that would help understand your setup.
a
sure!
pants.toml
h
If you're able to use Pants 2.10, the world gets a bit better. The docs site is down w/ explanation...but we added a new
[python].resolves
feature that has several improvements over
[python].requirement_constraints
. One of them is it installs the "lockfile" by using
-r lock.txt
rather than
-c constraints.txt
, which means that the file you have generated from
pip freeze
will work. Upgrade to Pants 2.10.0rc1 and run
./pants help-advanced python
for more info Now, you wouldn't be able to use Pants's
generate-lockfiles
goal to generate the lockfile for you, which is what we intend when using this resolves feature. That's because it does not yet support VCS & local requirements. You'd need to set
[python].invalid_lockfile_behavior = 'ignore'
, and continue to manually manage the lockfile with your current workflow Your lockfile also could not have
--hash
in it, which is to reduce risk of supply chain attacks. That's because of a pip limitation that if one entry has
--hash
, everything must. @enough-analyst-54434 has been leading a project to teach PEX to generate lockfiles a la pip. He's sketched out some thoughts on how to get Pex to support VCS/local requirements in a lockfile, which would be state-of-the-art https://github.com/pantsbuild/pex/issues/1556
e
In case Ryan can't, do you recall the remedy here @hundreds-father-404? Is it just delete those lines from the constraints file?
a
This isn't a blocker for me right now, we commit our constraints file and I can just use what we had until I need to change a dependency, I just wanted to know what was going on
👍 1
e
I just re-read the code @acoustic-librarian-3937 and I think deleting the line is the answer.
👍 1
a
really appreciate you explaining this stuff to me, I am very new to the python packaging ecosystem
sounds good thanks
I will also look into using v 2.10
h
I can't remember if the workaround would be 1) removing the entry entirely, vs 2) keeping the entry but w/o the URL stuff
removing the entry entirely
One quirk of constraints file is that it need not be exhaustive. Normally, that's a bad thing because it means you might have unpinned things -> less stability + more supply chain attack risk. But here, that can be to your advantage by working around the problem however it does mean that Pants can't make an important performance enhancement where it installs your constraints.txt once, and then extracts the relevant subset of deps when doing things like running tests. This is pretty important for most users to have better perf, otherwise Pants will do the correct-but-slow thing of resolving each unique combination of requirements your project uses as a distinct process
keeping the entry but w/o the URL stuff
I think that means that Pants's performance optimization of first resolving your entire constraints.txt will not be able to work properly. But I'm not certain
I will also look into using v 2.10
Cool, I recommend this the most. See https://www.pantsbuild.org/docs/upgrade-tips
❤️ 1