https://pantsbuild.org/ logo
Join Slack
Powered by
How do I specify a VCS requirement in the constrai...
# general
p
How do I specify a VCS requirement in the constraints file? Given this requirements line:
Copy code
st2-auth-ldap @ git+<https://github.com/StackStorm/st2-auth-ldap.git@master>
Then, using 
pip freeze --all
I end up with a line in my 
constraints-pants.txt
like:
Copy code
st2-auth-ldap @ git+<https://github.com/StackStorm/st2-auth-ldap.git@078334ca4767a552232c1a2563761c7f73c1d4e9>
But when I try to run 
./pants lint ::
(where pylint.pex is actually building successfully 🎉 ), I get this:
Copy code
DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement.. You can find discussion regarding this at <https://github.com/pypa/pip/issues/8210>.
ERROR: Links are not allowed as constraints
So, I tried commenting out the VCS portion in 
constraints-pants.txt
like this:
Copy code
st2-auth-ldap #@ git+<https://github.com/StackStorm/st2-auth-ldap.git@078334ca4767a552232c1a2563761c7f73c1d4e9>
But, then I get this error instead:
Copy code
Traceback (most recent call last):
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.4.0rc3_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 229, in _run_inner
    return self._perform_run(goals)
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.4.0rc3_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 168, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.4.0rc3_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 190, in _perform_run_body
    poll_delay=(0.1 if poll else None),
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.4.0rc3_py37/lib/python3.7/site-packages/pants/init/engine_initializer.py", line 136, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.4.0rc3_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 526, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.4.0rc3_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 490, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.lint.lint
  in pants.core.goals.lint.enrich_lint_results
  in pants.backend.python.lint.pylint.rules.pylint_lint
  in pants.backend.python.lint.pylint.rules.pylint_lint_partition
  in pants.backend.python.util_rules.pex.create_pex
  in pants.backend.python.util_rules.pex.build_pex (requirements.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.4.0rc3_py37/lib/python3.7/site-packages/pants/engine/process.py", line 255, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Resolving constraints-pants.txt' failed with exit code 1.
stdout:

stderr:
ERROR: Could not find a version that satisfies the requirement st2-auth-ldap
ERROR: No matching distribution found for st2-auth-ldap
pid 56773 -> /opt/local/Library/Frameworks/Python.framework/Versions/3.6/bin/python3.6 /Users/jafloyd/.cache/pants/named_caches/pex_root/pip.pex/cf4106b4c7898b462f7c172dde686a9747103f1a --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/jafloyd/.cache/pants/named_caches/pex_root --log /private/var/folders/d7/hkbhq46j2_g3f392vkjp407xxl_1cw/T/process-executionnZmtBG/.tmp/tmptqkj94nv/pip.log download --dest /private/var/folders/d7/hkbhq46j2_g3f392vkjp407xxl_1cw/T/process-executionnZmtBG/.tmp/tmpw6u3zj0v/opt.local.Library.Frameworks.Python.framework.Versions.3.6.bin.python3.6 --constraint constraints-pants.txt APScheduler==3.7.0 Babel==2.9.0 Flask==1.1.2 GitPython==3.1.14 Jinja2==2.11.3 MarkupSafe==1.1.1 PyNaCl==1.4.0 PyYAML==5.4.1 Routes==2.5.1 WebOb==1.8.7 Werkzeug==1.0.1 amqp==5.0.6 appdirs==1.4.4 astroid==2.5.3 bcrypt==3.2.0 beautifulsoup4==4.9.3 cachetools==2.0.1 certifi==2020.12.5 cffi==1.14.5 chardet==3.0.4 click==7.1.2 cryptography==3.4.7 debtcollector==2.2.0 decorator==5.0.6 distlib==0.3.1 dnspython==1.16.0 eventlet==0.30.2 fasteners==0.16 filelock==3.0.12 flex==6.14.1 futurist==2.3.0 gitdb==4.0.7 greenlet==1.0.0 gunicorn==20.1.0 idna==2.10 iso8601==0.1.14 itsdangerous==1.1.0 jsonpath-rw==1.4.0 jsonpointer==2.1 jsonschema==2.6.0 kazoo==2.8.0 kombu==5.0.2 lazy-object-proxy==1.6.0 linecache2==1.0.0 lockfile==0.12.2 mail-parser==3.15.0 mock==4.0.3 mongoengine==0.23.0 msgpack==1.0.2 netaddr==0.8.0 netifaces==0.10.9 networkx==1.11 orjson==3.5.1 orquesta oslo.config==1.12.1 oslo.i18n==5.0.1 oslo.serialization==4.1.0 oslo.utils==4.8.0 packaging==20.9 paramiko==2.7.2 passlib==1.7.4 pbr==5.5.1 pip==21.0.1 ply==3.11 prance==0.20.0 prompt-toolkit==3.0.18 psutil==5.8.0 pyOpenSSL==20.0.1 pyasn1-modules==0.2.8 pyasn1==0.4.8 pycparser==2.20 pymongo==3.11.3 pyparsing==2.4.7 python-dateutil==2.8.1 python-editor==1.0.4 python-ldap==3.0.0 python-statsd==2.1.0 pytz==2021.1 repoze.lru==0.7 requests==2.25.1 retrying==1.3.3 rfc3987==1.3.8 semver==2.13.0 setuptools==49.2.1 simplejson==3.17.2 six==1.15.0 smmap==4.0.0 soupsieve==2.2.1 sseclient-py==1.7 st2-auth-backend-flat-file st2-auth-ldap st2-rbac-backend stevedore==3.3.0 strict-rfc3339==0.7 tenacity==7.0.0 tooz==2.8.0 traceback2==1.4.0 tzlocal==2.1 udatetime==0.0.16 ujson==4.0.2 unittest2==1.1.0 urllib3==1.26.4 validate-email==1.3 vine==5.0.0 virtualenv==20.4.3 voluptuous==0.12.1 wcwidth==0.2.5 wheel==0.36.2 wrapt==1.12.1 yaql==1.1.3 zake==0.2.2 --index-url <https://pypi.org/simple/> --retries 5 --timeout 15 exited with 1 and STDERR:
None
pip 
-vvv
output gives me:
Copy code
1 location(s) to search for versions of st2-auth-ldap:
* <https://pypi.org/simple/st2-auth-ldap/>
Fetching project page and analyzing links: <https://pypi.org/simple/st2-auth-ldap/>
Getting page <https://pypi.org/simple/st2-auth-ldap/>
Found index url <https://pypi.org/simple/>
Looking up "<https://pypi.org/simple/st2-auth-ldap/>" in the cache
Request header has "max_age" as 0, cache bypassed
<https://pypi.org:443> "GET /simple/st2-auth-ldap/ HTTP/1.1" 404 13
Status code 404 not in (200, 203, 300, 301)
Could not fetch URL <https://pypi.org/simple/st2-auth-ldap/>: 404 Client Error: Not Found for url: <https://pypi.org/simple/st2-auth-ldap/> - skipping
Given no hashes to check 0 links for project 'st2-auth-ldap': discarding no candidates
ERROR: Could not find a version that satisfies the requirement st2-auth-ldap
ERROR: No matching distribution found for st2-auth-ldap
Exception information:
Traceback (most recent call last):
  File "/Users/jafloyd/.cache/pants/named_caches/pex_root/pip.pex/cf4106b4c7898b462f7c172dde686a9747103f1a/.deps/pip/pip/_vendor/resolvelib/resolvers.py", line 171, in _merge_into_criterion
    crit = self.state.criteria[name]
KeyError: 'st2-auth-ldap'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jafloyd/.cache/pants/named_caches/pex_root/pip.pex/cf4106b4c7898b462f7c172dde686a9747103f1a/.deps/pip/pip/_vendor/resolvelib/resolvers.py", line 318, in resolve
    name, crit = self._merge_into_criterion(r, parent=None)
  File "/Users/jafloyd/.cache/pants/named_caches/pex_root/pip.pex/cf4106b4c7898b462f7c172dde686a9747103f1a/.deps/pip/pip/_vendor/resolvelib/resolvers.py", line 173, in _merge_into_criterion
    crit = Criterion.from_requirement(self._p, requirement, parent)
  File "/Users/jafloyd/.cache/pants/named_caches/pex_root/pip.pex/cf4106b4c7898b462f7c172dde686a9747103f1a/.deps/pip/pip/_vendor/resolvelib/resolvers.py", line 83, in from_requirement
    raise RequirementsConflicted(criterion)
pip._vendor.resolvelib.resolvers.RequirementsConflicted: Requirements conflict: SpecifierRequirement('st2-auth-ldap')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jafloyd/.cache/pants/named_caches/pex_root/pip.pex/cf4106b4c7898b462f7c172dde686a9747103f1a/.deps/pip/pip/_internal/resolution/resolvelib/resolver.py", line 122, in resolve
    requirements, max_rounds=try_to_avoid_resolution_too_deep,
  File "/Users/jafloyd/.cache/pants/named_caches/pex_root/pip.pex/cf4106b4c7898b462f7c172dde686a9747103f1a/.deps/pip/pip/_vendor/resolvelib/resolvers.py", line 453, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/Users/jafloyd/.cache/pants/named_caches/pex_root/pip.pex/cf4106b4c7898b462f7c172dde686a9747103f1a/.deps/pip/pip/_vendor/resolvelib/resolvers.py", line 320, in resolve
    raise ResolutionImpossible(e.criterion.information)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('st2-auth-ldap'), parent=None)]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/jafloyd/.cache/pants/named_caches/pex_root/pip.pex/cf4106b4c7898b462f7c172dde686a9747103f1a/.deps/pip/pip/_internal/cli/base_command.py", line 223, in _main
    status = self.run(options, args)
  File "/Users/jafloyd/.cache/pants/named_caches/pex_root/pip.pex/cf4106b4c7898b462f7c172dde686a9747103f1a/.deps/pip/pip/_internal/cli/req_command.py", line 180, in wrapper
    return func(self, options, args)
  File "/Users/jafloyd/.cache/pants/named_caches/pex_root/pip.pex/cf4106b4c7898b462f7c172dde686a9747103f1a/.deps/pip/pip/_internal/commands/download.py", line 131, in run
    reqs, check_supported_wheels=True
  File "/Users/jafloyd/.cache/pants/named_caches/pex_root/pip.pex/cf4106b4c7898b462f7c172dde686a9747103f1a/.deps/pip/pip/_internal/resolution/resolvelib/resolver.py", line 127, in resolve
    six.raise_from(error, e)
  File "<string>", line 3, in raise_from
pip._internal.exceptions.DistributionNotFound: No matching distribution found for st2-auth-ldap
Ooh! I got it to run!
🔥 1
I commented out all of the VCS entries from the consraints file. Now it finishes resolving and runs pylint. 🎉 But, I get this warning:
Copy code
23:43:02.57 [WARN] The constraints file constraints-pants.txt does not contain entries for the following requirements: orquesta
23:43:02.58 [WARN] Ignoring `[python_setup].resolve_all_constraints` option because constraints file does not cover all requirements.
So, is there a way for the constraints file to cover VCS entries too?
h
I'll look at that tomorrow
OK I have a fix, just figuring out how to test it
@proud-dentist-22844 Out of interest, how are you getting 
pip freeze
to emit 
st2-auth-ldap @ git+<https://github.com/StackStorm/st2-auth-ldap.git@078334ca4767a552232c1a2563761c7f73c1d4e9>
into your constraints files? If I try that I get 
st2-auth-ldap==3.5.dev0
in the freeze output
p
odd. I’m using the generate_constraints.sh script which does 
pip freeze --all
. I tested with pip 20.3.3 (in a different virtualenv) and 21.0.1 (in the venv built by generate_constraints.sh). Both of them use the git hash.
I don’t have any pip config files (
pip config debug
).
One venv has setuptools 40.6.2 and the other has 51.3.3
https://github.com/pypa/pip/pull/7612#issue-364531569 Looks like the commit hash is the expected behavior since pip 20.1
👍 1
What version of pip do you have installed?
h
Possibly an older one
Will try again
anyway, https://github.com/pantsbuild/pants/pull/11907 fixes the issue you mentioned above
There are some caveats, but basically you'd just delete those VCS constraints from your constraints file
And we'll take this issue into account in the implementation of "proper" lockfiles, coming soon!
p
"${PIP}" freeze --all | sed '/git+https/s/^/#/' >> "${CONSTRAINTS_FILE}"
h
basically!
p
proper? yet another lockfile?
h
That way any transitive deps of the VCS deps that are not themselves VCS deps are locked
Well, I mostly mean supporting poetry...
👍 1
pip constraints files aren't real lockfiles
p
ah. Great. 🙂 I like poetry.
➕ 1
Pipenv is not as nice to use.
h
@steep-breakfast-98857 is working on Poetry support as we speak
p
I believe I’m following that issue in the repo
s
Yep - hopefully done by EOW and in a following release. GitHub issue for it specifies the lossy nature of the conversion (see here) btw.
🙌 2
p
As a first cut, the lossy conversion isn’t a problem. Next, I wonder if a pants plugin could integrate poetry even more deeply to avoid that lossiness - preserve the full lockfile details and pass it on to whatever downloads the wheels. or even use poetry itself to build what goes in requirements.pex. Or a plugin for poetry to so you could teach it how to generate the requirements.pex stuff.
h
Yes, that is the phase II I am imagining, where Pants actually generates the lockfile for you. You still check it in and possibly tweak it, but Pants does the gruntwork.
👍 2
OK the fix for the above is in (it took me a few tries to get the test to pass in CI because Requirements is finicky about parsing 
file://
URLs): https://github.com/pantsbuild/pants/pull/11907
It doesn't cherry-pick cleanly onto 2.3.x or 2.4.x. @proud-dentist-22844 are you ok running on tomorrow's dev release?
If that's a problem I can fix up the cherry-pick conflicts.
p
I'm fine with using a dev release
h
Cool, will post here when it's out.
2.5.0.dev2 is out, and should address this issue.
Confirmed that it works (but you have to entirely remove the constraints-pants.txt lines that reference the VCS repos, Pants now adds those back into the resolve for you).
And by "works" I mean I can run 
./pants lint ::
and black and flake8 pass. PyLint itself builds too, but there are PyLint failures that I guess are legit?
p
I just got back from vacation (a nice week of camping without reception). The problem with the pylint failures is that I can’t tell pants to only run pylint on a subset of files (eg only on the most important non-test code), which is what the Makefile does. So, those might be “legit” except that they have been implicitly ignored by only explicitly including other code.
h
Sorry for the late reply, this got swallowed below the fold... There are a couple of ways to only run Pants on specific files
One is to just run pants on those files explicitly, e.g., 
./pants path/to/file.py path/to/dir::
and so on
h
Although, iiuc, Jacob would like to run other linters on those files just not Pylint. Would Pylint's 
--ignore
option work via 
[pylint].args
in your 
pants.toml
? http://pylint.pycqa.org/en/latest/user_guide/run.html#command-line-options
h
Oh yes, that could work as well!
Yeah, re running the other linters you'd need to run Pants twice, once excluding pylint on all files, and then just pylint on the subset where it is expected to pass
This might be a common enough use case that we should allow linter excludes at the target level
➕ 2
It makes sense because "this file cannot pass pylint" is a true property of the file
➕ 1
Like "this file is python2.7 compatible"
@proud-dentist-22844 do any of these solutions work? Glad you had a good time camping!
p
The pylint 
--ignore
option is really limited or works in a way that is a complete mismatch with my mental model 😉. I couldn’t figure out how to use patterns or paths to exclude things.
Yeah. Expecting someone to “know” that they need to run pants twice with special args to first skip and then only run pylint kind of defeats a major purpose of pants: replace the makefile mess.
👍 1
I mean, I’ll always forget, and other people will keep asking, so it will end up codified in some script or other, and then pants isn’t the primary entry point for developer / testing functions.
I really like the idea of excluding the linter at the target level.
h
Update, Benjy and I have been talking about how to support this and indeed I think it makes a lot of sense to do at the target level, and it's not very difficult for us to implement I'm proposing something like
Copy code
python_library(skip_pylint=True)
I'm hoping to get this out Monday, which should make it into the upcoming dev release
➕ 1
h
I can hold off on the dev release until then
👍 1
(I am releaser this week)
p
Ooh! Sweet.
Would I need to put that in every BUILD file under a given directory? Or just the top most?
h
I think every target would need it (until we support inheriting metadata up the filesystem, which is something we're thinking about)
👍 2
h
https://github.com/pantsbuild/pants/pull/12008. Thanks for the suggestion @proud-dentist-22844! This is a really neat way to facilitate adding tools incrementally in large repos
🎉 1
p
Very nice. Is there an issue or PR about metadata inheritance as well?
h
I don't think so, but we've been having discussions the past few months. That is a much much bigger project
p
ok. I look forward to that bigger project. 🙂
➕ 1
h
The inheriting metadata thing is part of getting rid of (explicit) BUILD file targets entirely, unless you need them to specify non-default metadata.
Basically, we believe that users think in terms of files and dirs, and targets are an unnatural construct that adds cognitive load, and sometimes creates an impedance mismatch of sorts
And by "we" I mean "I", but not just me
h
Yeah, my bigger frustration with targets is that it's too hard to change metadata in a fine-grained way without adding lots of boilerplate. This is a pretty horrible statement from our docs https://www.pantsbuild.org/v2.5/docs/python-test-goal#timeouts:
Copy code
This timeout will apply to each file belonging to the python_tests target, meaning that test_f1.py will have a timeout of 120 seconds and test_f2.py will have a timeout of 120 seconds. If you want finer-grained timeouts, create a dedicated python_tests target for each file:
You should be able to change the timeout for 
test_f1.py
without needing to split out new targets and without impacting sibling files My concern is less that "targets are unnatural / targets should possibly not fundamentally exist"
p
getting rid of (explicit) BUILD file targets entirely, unless you need them to specify non-default metadata.
Yes please!
4 Views
Open in Slack
PreviousNext
Powered by