Hi, I had been on 1.1.14 version of poetry and the requirements.txt didn't contain any optional or extras. Today I upgraded poetry and now requirements.txt contains extras and I am seeing the test cases run by pants are failing. Here's my pyproject.toml file -> from which requirements.txt is generated via

poetry export -f requirements.txt -o requirements.txt --without-hashes

python_requirement(
    name="setuptools",
    requirements=["setuptools"],
)

python_requirement(
    name="torch",
    requirements=["torch==1.10.0+cpu"],
    dependencies=[":setuptools"],
)

python_requirement(
    name="pytorch-lightning",
    requirements=["pytorch-lightning"],
    dependencies=[":torch"],
)

python_requirement(
    name="torchmetrics",
    requirements=["torchmetrics"],
    dependencies=[":torch"],
)

python_requirements(
    name="reqs",
    module_mapping={
        "python-constraint": ["constraint"],
        "google-api-core": ["google.api_core"],
        "grpcio": ["grpc"],
        "grpcio-health-checking": ["grpc_health"],
        "python-json-logger": ["pythonjsonlogger"],
    },
)

It looks like pants treats requirements.txt as a constraints file(

ERROR: Constraints cannot have extras

). Do I need to upgrade pants or make some changes in BUILD file to resolve the issue?

I think you missed to post pyproject.toml. In requirments.txt I have used extras and pants never complained about it

fastapi[all]~=0.18.0

How is the generated requirements.txt looks like?

Ah, thanks for notifying @refined-addition-53644. Updated the pyproject.toml link above. Adding here as well: https://github.com/brary/Sample-Pyproject/blob/main/pyproject.toml

@modern-monkey-78364 it looks like that requirements file has: 1. A pip flag at the top 2. No extras (

<project name>[<extra name>]...

), but lots of markers (

<project name><specifier>; <marker>

). Can you provide the full command line and output of an example failure along with the Pants version you're using?

Hi @enough-analyst-54434, thanks for taking a look. I was on pants version: 2.10 where I faced the issue, now I have upgraded to 2.14 and the issue is still there. There is a dependency with extras in pyproject.toml ->

pymongo = {extras = ["srv"], version = "^4.3.2"}

which after poetry upgrade, translated to

pymongo[srv]==4.3.2 ; python_version >= "3.7" and python_version < "4.0"

in requirements.txt file. To create requirements.txt I used:

poetry export -f requirements.txt -o requirements.txt --without-hashes

I was testing a unit test:

./pants test src/backend/project/service/metadata/test_add_metadata.py

And I got this error:

16:18:16.08 [ERROR] 1 Exception encountered:

ProcessExecutionFailure: Process 'Building requirements.pex with 17 requirements: astunparse<2.0.0,>=1.6.3, flask<3.0.0,>=2.0.2, google-api-core<2.0.0,>=1.22.4, marshmallow-sqlalchemy<0.24.0,>=0.23.1, opentelemetry-api<0.18.0,>=0.17b0, opentelemetry-exporter-opencensus<0.18.0,>=0.17b0, opentelemetry-exporter-prometheus<0.18.0,>=0.17b0, opentelemetry-instrumentation-grpc<0.18.0,>=0.17b0, opentelemetry-instrumentation-sqlalchemy<0.18.0,>=0.17b0, opentelemetry-instrumentation<0.18.0,>=0.17b0, opentelemetry-sdk<0.18.0,>=0.17b0, prometheus-client<0.15.0,>=0.14.1, protobuf<4.0.0,>=3.12.2, psycopg2-binary<3.0.0,>=2.8.5, pytest<7.0.0,>=6.0.1, setuptools, sqlalchemy<2.0.0,>=1.3.18' failed with exit code 1.

stdout:

stderr:

pid 133712 -> /.kubetop/home/navinderbrar/.cache/pants/named_caches/pex_root/venvs/57eae4c648bfb80a08e394e056b3ca1a88a06759/c68d021d9c61b1d493ab658b3a6b2c85f19a20a2/bin/python -sE /.kubetop/home/navinderbrar/.cache/pants/named_caches/pex_root/venvs/57eae4c648bfb80a08e394e056b3ca1a88a06759/c68d021d9c61b1d493ab658b3a6b2c85f19a20a2/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /.kubetop/home/navinderbrar/.cache/pants/named_caches/pex_root/pip_cache --log /tmp/pants-sandbox-cF6juK/.tmp/pex-pip-log.kb1jyhcx/pip.log download --dest /.kubetop/home/navinderbrar/.cache/pants/named_caches/pex_root/downloads/resolver_download.s4xpl8oe/kubetop.home.navinderbrar..pyenv.versions.3.7.15.bin.python3.7 --constraint __constraints.txt astunparse<2.0.0,>=1.6.3 flask<3.0.0,>=2.0.2 google-api-core<2.0.0,>=1.22.4 marshmallow-sqlalchemy<0.24.0,>=0.23.1 opentelemetry-api<0.18.0,>=0.17b0 opentelemetry-exporter-opencensus<0.18.0,>=0.17b0 opentelemetry-exporter-prometheus<0.18.0,>=0.17b0 opentelemetry-instrumentation-grpc<0.18.0,>=0.17b0 opentelemetry-instrumentation-sqlalchemy<0.18.0,>=0.17b0 opentelemetry-instrumentation<0.18.0,>=0.17b0 opentelemetry-sdk<0.18.0,>=0.17b0 prometheus-client<0.15.0,>=0.14.1 protobuf<4.0.0,>=3.12.2 psycopg2-binary<3.0.0,>=2.8.5 pytest<7.0.0,>=6.0.1 setuptools sqlalchemy<2.0.0,>=1.3.18 --index-url <https://pypi.org/simple/> --extra-index-url <https://download.pytorch.org/whl/cpu> --retries 5 --timeout 15 exited with 1 and STDERR:

DEPRECATION: Constraints are only allowed to take the form of a package name and a version specifier. Other forms were originally permitted as an accident of the implementation, but were undocumented. The new implementation of the resolver no longer supports these forms. A possible replacement is replacing the constraint with a requirement.. You can find discussion regarding this at <https://github.com/pypa/pip/issues/8210>.

ERROR: Constraints cannot have extras

Can you please re-run but add the flag

--keep-sandboxes=on_failure

? That will print a log line about keeping a

/tmp/...

directory and then we can go look at the contents of the

__constraints.txt

file inside that temp dir.

This is the content of that __constraints.txt file: https://github.com/brary/Sample-Pyproject/blob/main/constraints.txt

Aha - OK, Pip just has an inaccurate error message then I guess. There are 0 extras in that file, only markers.

Yeah, that's correct @enough-analyst-54434. Not using

[python] enable_resolves = true

and I am planning to migrate, just caught in the middle since it's a big project and already had been using poetry.

Ok, gotcha. Yeah, well besides editing out the markers manually after export you have no options except enabling resolves and using the exported requirements as the lock (instead of as constraints) or finishing the migration completely and switching to Pex lock files, which is the default.

Thanks @enough-analyst-54434, I have few doubts. 1. What's the need behind removing markers? 2. If I use

pip freeze > requirements.txt

instead of

poetry export -f requirements.txt -o requirements.txt --without-hashes

would't that remove the markers instead of doing it manually?

Backing up, you only need to remove environment markers for use of the requirements file as a constraints file. If you stop doing that and use it as a requirements file, all is good. This requires some new pants configuration, namely

[python] enable_resolves = true

+ pointing That subsystem at your requirements.txt as the lock file to use.

I have updated the pants.toml as below:

[python]
# The default interpreter compatibility for code in this repo. Individual targets can ovverride
#  this with the `compatibility` field. See
#  <https://pants.readme.io/docs/python-interpreter-compatibility>.
interpreter_constraints = ["==3.7.*"]
# Use a lockfile. See <https://pants.readme.io/docs/python-third-party-dependencies>.
enable_resolves = true
resolves = "requirements.txt"
resolves_generate_lockfiles = false

This is not resolving, even though I have it created manually.

Exception message: Error computing value for --resolves in scope 'python' (may also be from PANTS_* environment variables).

Caused by:

Invalid dict value: requirements.txt

Another issue which I faced when I tried to run pants test after poetry+pants upgrade was:

19:58:22.14 [ERROR] 1 Exception encountered:

`Exception: Unmatched glob from python_requirements(address="//:reqs", description=None, module_mapping=FrozenDict({'python-constraint': ('constraint',), 'google-api-core': ('google.api_core',), 'grpcio': ('grpc',), 'grpcio-health-checking': ('grpc_health',), 'python-json-logger': ('pythonjsonlogger',)}), overrides=None, source=requirements.txt, tags=None, type_stubs_module_mapping=FrozenDict({}))'s field `source`: "requirements.txt"` This above error stopped, when I changed

python_requirements

in my BUILD file to

poetry_requirements

--> Is this the right solution?

I think that's right. The error looks like it was simply placing a

python_requirements()

default target in a

BUILD

file that was not a sibling of a file named `requirements.txt`; so I think that could be fixed. That said, using

poetry_requirements()

in a `BUILD`that is a sibling to `pyproject.toml`is fine too, even if the lock is an exported one like you have.

Thank you so much @enough-analyst-54434, this conversation has been super helpful . 1. Is there a doc on completely switching over from poetry to pants, which I can follow from migration? 2. I have currently few BUILD files somewhat like these:

python_sources()

python_tests(
    name="tests",
)

There are certain tests which I guess were not being run because they were not mentioned in BUILD files(as you see in above code) with 2.10 pants version, but after the upgrade now to 2.14, my test cases are bring run and failing. Can I skip running these tests, which are not explicitly mentioned in BUILD files till I fix those test cases?

There is no such guide that I know of. IIUC the mapping from Poetry to Pants is: + poetry build -> ./pants package + poetry shell -> ./pants repl + poetry install ~> ./pants {fmt,lint,check,test,...} + poetry lock -> ./pants generate-lockfiles + poetry export ~> ./pants export Pants does not have add / remove etc fetaures, it expects you to add requirements yourself; i.e., manually research available requests versions and make a choice and then write

requests==2.0.0

or what have you. Mind you, this is not a one to one mapping. With tools like Poetry / tox, etc - its easier to set up venvs for ad-hoc tools. In fact Poetry / tox, etc only support ad-hoc tools. You have to pick these and set them up (pytest, etc). Pants has a set of plugins to run common tools for

fmt

,

lint

, `check`and

test

operations (goals), you just enable those plugins. It's easy to tweak tool versions for those plugins but its much less easy still to add an arbitrary tool not already supported. You can write a plugin, but that's a bit much compared to how easy it is using Poetry / tox, etc to add a tool. That said, you gain alot once done in Pants - free parallelism, incrementalism and caching amongst others. There are folks working hard though on making adding ad-hoc tool use quite a bit easier as we speak; so the tradeoffs there will probaly shift ~Pants 2.17 I'd guess.

Thank you so much John, appreciate the detailed explanations.