https://pantsbuild.org/ logo
#general
Title
# general
m

modern-monkey-78364

12/13/2022, 2:16 PM
Hi, I had been on 1.1.14 version of poetry and the requirements.txt didn't contain any optional or extras. Today I upgraded poetry and now requirements.txt contains extras and I am seeing the test cases run by pants are failing. Here's my pyproject.toml file -> from which requirements.txt is generated via
poetry export -f requirements.txt -o requirements.txt --without-hashes
Copy code
python_requirement(
    name="setuptools",
    requirements=["setuptools"],
)

python_requirement(
    name="torch",
    requirements=["torch==1.10.0+cpu"],
    dependencies=[":setuptools"],
)

python_requirement(
    name="pytorch-lightning",
    requirements=["pytorch-lightning"],
    dependencies=[":torch"],
)

python_requirement(
    name="torchmetrics",
    requirements=["torchmetrics"],
    dependencies=[":torch"],
)

python_requirements(
    name="reqs",
    module_mapping={
        "python-constraint": ["constraint"],
        "google-api-core": ["google.api_core"],
        "grpcio": ["grpc"],
        "grpcio-health-checking": ["grpc_health"],
        "python-json-logger": ["pythonjsonlogger"],
    },
)
It looks like pants treats requirements.txt as a constraints file(
ERROR: Constraints cannot have extras
). Do I need to upgrade pants or make some changes in BUILD file to resolve the issue?
r

refined-addition-53644

12/13/2022, 2:59 PM
I think you missed to post pyproject.toml. In requirments.txt I have used extras and pants never complained about it
Copy code
fastapi[all]~=0.18.0
How is the generated requirements.txt looks like?
m

modern-monkey-78364

12/13/2022, 3:01 PM
Ah, thanks for notifying @refined-addition-53644. Updated the pyproject.toml link above. Adding here as well: https://github.com/brary/Sample-Pyproject/blob/main/pyproject.toml
I have uploaded the requirements.txt as well: https://github.com/brary/Sample-Pyproject/blob/main/requirements.txt
e

enough-analyst-54434

12/13/2022, 4:29 PM
@modern-monkey-78364 it looks like that requirements file has: 1. A pip flag at the top 2. No extras (
<project name>[<extra name>]...
), but lots of markers (
<project name><specifier>; <marker>
). Can you provide the full command line and output of an example failure along with the Pants version you're using?
m

modern-monkey-78364

12/13/2022, 4:43 PM
Hi @enough-analyst-54434, thanks for taking a look. I was on pants version: 2.10 where I faced the issue, now I have upgraded to 2.14 and the issue is still there. There is a dependency with extras in pyproject.toml ->
pymongo = {extras = ["srv"], version = "^4.3.2"}
which after poetry upgrade, translated to
pymongo[srv]==4.3.2 ; python_version >= "3.7" and python_version < "4.0"
in requirements.txt file. To create requirements.txt I used:
Copy code
poetry export -f requirements.txt -o requirements.txt --without-hashes
I was testing a unit test:
Copy code
./pants test src/backend/project/service/metadata/test_add_metadata.py
And I got this error:
16:18:16.08 [ERROR] 1 Exception encountered:
ProcessExecutionFailure: Process 'Building requirements.pex with 17 requirements: astunparse<2.0.0,>=1.6.3, flask<3.0.0,>=2.0.2, google-api-core<2.0.0,>=1.22.4, marshmallow-sqlalchemy<0.24.0,>=0.23.1, opentelemetry-api<0.18.0,>=0.17b0, opentelemetry-exporter-opencensus<0.18.0,>=0.17b0, opentelemetry-exporter-prometheus<0.18.0,>=0.17b0, opentelemetry-instrumentation-grpc<0.18.0,>=0.17b0, opentelemetry-instrumentation-sqlalchemy<0.18.0,>=0.17b0, opentelemetry-instrumentation<0.18.0,>=0.17b0, opentelemetry-sdk<0.18.0,>=0.17b0, prometheus-client<0.15.0,>=0.14.1, protobuf<4.0.0,>=3.12.2, psycopg2-binary<3.0.0,>=2.8.5, pytest<7.0.0,>=6.0.1, setuptools, sqlalchemy<2.0.0,>=1.3.18' failed with exit code 1.
stdout:
stderr:
pid 133712 -> /.kubetop/home/navinderbrar/.cache/pants/named_caches/pex_root/venvs/57eae4c648bfb80a08e394e056b3ca1a88a06759/c68d021d9c61b1d493ab658b3a6b2c85f19a20a2/bin/python -sE /.kubetop/home/navinderbrar/.cache/pants/named_caches/pex_root/venvs/57eae4c648bfb80a08e394e056b3ca1a88a06759/c68d021d9c61b1d493ab658b3a6b2c85f19a20a2/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /.kubetop/home/navinderbrar/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-cF6juK/.tmp/pex-pip-log.kb1jyhcx/pip.log download --dest /.kubetop/home/navinderbrar/.cache/pants/named_caches/pex_root/downloads/resolver_download.s4xpl8oe/kubetop.home.navinderbrar..pyenv.versions.3.7.15.bin.python3.7 --constraint __constraints.txt astunparse<2.0.0,>=1.6.3 flask<3.0.0,>=2.0.2 google-api-core<2.0.0,>=1.22.4 marshmallow-sqlalchemy<0.24.0,>=0.23.1 opentelemetry-api<0.18.0,>=0.17b0 opentelemetry-exporter-opencensus<0.18.0,>=0.17b0 opentelemetry-exporter-prometheus<0.18.0,>=0.17b0 opentelemetry-instrumentation-grpc<0.18.0,>=0.17b0 opentelemetry-instrumentation-sqlalchemy<0.18.0,>=0.17b0 opentelemetry-instrumentation<0.18.0,>=0.17b0 opentelemetry-sdk<0.18.0,>=0.17b0 prometheus-client<0.15.0,>=0.14.1 protobuf<4.0.0,>=3.12.2 psycopg2-binary<3.0.0,>=2.8.5 pytest<7.0.0,>=6.0.1 setuptools sqlalchemy<2.0.0,>=1.3.18 --index-url <https://pypi.org/simple/> --extra-index-url <https://download.pytorch.org/whl/cpu> --retries 5 --timeout 15 exited with 1 and STDERR:
DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement.. You can find discussion regarding this at <https://github.com/pypa/pip/issues/8210>.
ERROR: Constraints cannot have extras
e

enough-analyst-54434

12/13/2022, 4:45 PM
Can you please re-run but add the flag
--keep-sandboxes=on_failure
? That will print a log line about keeping a
/tmp/...
directory and then we can go look at the contents of the
__constraints.txt
file inside that temp dir.
m

modern-monkey-78364

12/13/2022, 4:56 PM
This is the content of that __constraints.txt file: https://github.com/brary/Sample-Pyproject/blob/main/constraints.txt
e

enough-analyst-54434

12/13/2022, 5:01 PM
Aha - OK, Pip just has an inaccurate error message then I guess. There are 0 extras in that file, only markers.
So, it looks like you're not using
[python] enable_resolves = true
(https://www.pantsbuild.org/docs/reference-python#enable_resolves) - is that correct?
Ah right, damn Poetry. So ... what do you actually use Poetry for that Pants does not provide? Is this just the case of transitioning from Poetry to Pants and you're caught in the middle needing to support both, or is there a use case Poetry satisfies that Pants does not?
m

modern-monkey-78364

12/13/2022, 5:06 PM
Yeah, that's correct @enough-analyst-54434. Not using
[python] enable_resolves = true
and I am planning to migrate, just caught in the middle since it's a big project and already had been using poetry.
e

enough-analyst-54434

12/13/2022, 5:07 PM
Ok, gotcha. Yeah, well besides editing out the markers manually after export you have no options except enabling resolves and using the exported requirements as the lock (instead of as constraints) or finishing the migration completely and switching to Pex lock files, which is the default.
m

modern-monkey-78364

12/13/2022, 7:00 PM
Thanks @enough-analyst-54434, I have few doubts. 1. What's the need behind removing markers? 2. If I use
pip freeze > requirements.txt
instead of
poetry export -f requirements.txt -o requirements.txt --without-hashes
would't that remove the markers instead of doing it manually?
e

enough-analyst-54434

12/13/2022, 7:38 PM
Backing up, you only need to remove environment markers for use of the requirements file as a constraints file. If you stop doing that and use it as a requirements file, all is good. This requires some new pants configuration, namely
[python] enable_resolves = true
+ pointing That subsystem at your requirements.txt as the lock file to use.
m

modern-monkey-78364

12/13/2022, 8:25 PM
I have updated the pants.toml as below:
Copy code
[python]
# The default interpreter compatibility for code in this repo. Individual targets can ovverride
#  this with the `compatibility` field. See
#  <https://pants.readme.io/docs/python-interpreter-compatibility>.
interpreter_constraints = ["==3.7.*"]
# Use a lockfile. See <https://pants.readme.io/docs/python-third-party-dependencies>.
enable_resolves = true
resolves = "requirements.txt"
resolves_generate_lockfiles = false
This is not resolving, even though I have it created manually.
Exception message: Error computing value for --resolves in scope 'python' (may also be from PANTS_* environment variables).
Caused by:
Invalid dict value: requirements.txt
Another issue which I faced when I tried to run pants test after poetry+pants upgrade was:
19:58:22.14 [ERROR] 1 Exception encountered:
`Exception: Unmatched glob from python_requirements(address="//:reqs", description=None, module_mapping=FrozenDict({'python-constraint': ('constraint',), 'google-api-core': ('google.api_core',), 'grpcio': ('grpc',), 'grpcio-health-checking': ('grpc_health',), 'python-json-logger': ('pythonjsonlogger',)}), overrides=None, source=requirements.txt, tags=None, type_stubs_module_mapping=FrozenDict({}))'s field `source`: "requirements.txt"` This above error stopped, when I changed
python_requirements
in my BUILD file to
poetry_requirements
--> Is this the right solution?
My bad on the first part of above query, I think this is the correct way:
Copy code
[python]
# The default interpreter compatibility for code in this repo. Individual targets can ovverride
#  this with the `compatibility` field. See
#  <https://pants.readme.io/docs/python-interpreter-compatibility>.
interpreter_constraints = ["==3.7.*"]
# Use a lockfile. See <https://pants.readme.io/docs/python-third-party-dependencies>.
enable_resolves = true
resolves_generate_lockfiles = false

[python.resolves]
python-default = "requirements.txt"
Please clarify on the second part regarding
python_requirements
vs
poetry_requirements
e

enough-analyst-54434

12/13/2022, 8:53 PM
I think that's right. The error looks like it was simply placing a
python_requirements()
default target in a
BUILD
file that was not a sibling of a file named `requirements.txt`; so I think that could be fixed. That said, using
poetry_requirements()
in a `BUILD`that is a sibling to `pyproject.toml`is fine too, even if the lock is an exported one like you have.
Yeah, actually what you did is more correct. The
python_requirements()
, if properly hooked up, would allow your code to depend on transitive deps directly, which is bad.
m

modern-monkey-78364

12/14/2022, 9:40 AM
Thank you so much @enough-analyst-54434, this conversation has been super helpful . 1. Is there a doc on completely switching over from poetry to pants, which I can follow from migration? 2. I have currently few BUILD files somewhat like these:
Copy code
python_sources()

python_tests(
    name="tests",
)
There are certain tests which I guess were not being run because they were not mentioned in BUILD files(as you see in above code) with 2.10 pants version, but after the upgrade now to 2.14, my test cases are bring run and failing. Can I skip running these tests, which are not explicitly mentioned in BUILD files till I fix those test cases?
e

enough-analyst-54434

12/14/2022, 7:53 PM
There is no such guide that I know of. IIUC the mapping from Poetry to Pants is: + poetry build -> ./pants package + poetry shell -> ./pants repl + poetry install ~> ./pants {fmt,lint,check,test,...} + poetry lock -> ./pants generate-lockfiles + poetry export ~> ./pants export Pants does not have add / remove etc fetaures, it expects you to add requirements yourself; i.e., manually research available requests versions and make a choice and then write
requests==2.0.0
or what have you. Mind you, this is not a one to one mapping. With tools like Poetry / tox, etc - its easier to set up venvs for ad-hoc tools. In fact Poetry / tox, etc only support ad-hoc tools. You have to pick these and set them up (pytest, etc). Pants has a set of plugins to run common tools for
fmt
,
lint
, `check`and
test
operations (goals), you just enable those plugins. It's easy to tweak tool versions for those plugins but its much less easy still to add an arbitrary tool not already supported. You can write a plugin, but that's a bit much compared to how easy it is using Poetry / tox, etc to add a tool. That said, you gain alot once done in Pants - free parallelism, incrementalism and caching amongst others. There are folks working hard though on making adding ad-hoc tool use quite a bit easier as we speak; so the tradeoffs there will probaly shift ~Pants 2.17 I'd guess.
m

modern-monkey-78364

12/15/2022, 10:58 AM
Thank you so much John, appreciate the detailed explanations.
7 Views