https://pantsbuild.org/ logo
Powered by
c

crooked-country-1937

12/18/2022, 1:56 PM
Hi! 👋 Is there way to specify interpreter constraints when 
./pants generate-lockfiles
for specific requirements.txt? need some guidance on how to setup codebases requiring multiple python versions. We have 2 big codebases. 1. 
py3.7
used in airflow, another 
py3.10
used in dockerized apps. To support these, I went with 2 requirements.txt. 
3rdparty/airflow/requirements.txt
and 
3rdparty/python/requirements.txt
. Added 
python_sources(resolve="airflow-default", interpreter_constraints=["CPython>=3.7.10, <3.8"],)
to all BUILD files for airflow codebases. Also changed my 
pants.toml
to this:
Copy code
[python]
interpreter_constraints = ["CPython>=3.7.10, <3.8", "==3.10.*"]
My local python is version 
3.10.8
The issue is 
./pants generate-lockfiles
when resolving packages for airflow seems to be looking for packages which support 3.10.8. Which causes resolution issues.
h

happy-kitchen-89482

12/18/2022, 5:03 PM
I think you want the resolves_to_interpreter_constraints option
Did that work?
c

crooked-country-1937

12/20/2022, 12:40 AM
Hey. thanks for following up. It did work for 3.10.8 stuff. Have to test airflow, will update once i go through it I completely missed the python section of the doc. Has some interesting options I may need in future. What does the pants community think of having this be documented in https://www.pantsbuild.org/docs/python-interpreter-compatibility? It is critical to for mixed interpreters to work.
Also, I had to do it slightly differently. Like this.
Copy code
[python]
# Default to 3.10.* so that pytest uses the correct version
interpreter_constraints = ["==3.10.*"]
# Enable the "resolves" mechanism, which turns on lockfiles for user code. See
# <https://www.pantsbuild.org/docs/python-third-party-dependencies>. This also adds the
# `generate-lockfiles` goal for Pants to generate the lockfile for you.
enable_resolves = true
default_resolve = "python-default"
# This will become the default in Pants 2.15.
tailor_pex_binary_targets = false

[python.resolves]
python-default = "3rdparty/python/default.lock"
airflow-default = "3rdparty/airflow/default.lock"

[python.resolves_to_constraints_file]
airflow-default = "3rdparty/airflow/airflow-constraints-3.7.txt"

[python.resolves_to_interpreter_constraints]
# for airflow specific codebase, use py3.7
airflow-default = ["CPython>=3.7.10, <3.8"]
python-default = ["==3.10.*"]
In general, pants requires a lot of copying of the constraint
h

happy-kitchen-89482

12/20/2022, 1:38 AM
We'll happily accept suggestions for documentation improvements! The docs have a "Suggest Edits" button you can use. Thanks!
c

crooked-country-1937

12/20/2022, 3:51 AM
Will do 🙂 Btw, it didn’t work. Got this error when running py3.7 code:
Copy code
InvalidFieldException: The target airflow/src/python/tunein/dags/hellodag.py has the `interpreter_constraints` ('CPython>=3.7.10, <3.8',), which are not a subset of the `interpreter_constraints` of some of its dependencies:

  * ('==3.10.*',): airflow/src/python/tunein/__init__.py

To fix this, you should likely adjust airflow/src/python/tunein/dags/hellodag.py's `interpreter_constraints` to match the narrowest range in the above list.
However, I had to make set. interpreter_constraints to ==3.10.* else 
./pants test ::
fails:
Copy code
ib/test/python/tunein/lib/samplelib_test.py:1: in <module>
    from tunein.lib.samplelib import func1, use_requests
E     File "lib/src/python/tunein/lib/samplelib.py", line 17
E       match point:
E                 ^
E   SyntaxError: invalid syntax
It looks like the answer to this is explicitly setting 
interpreter_constraints=["==3.10.*"]
in every build file. Is there a better way to do this?
I decided to stick to a single version for now and move on to other aspects of the build.
2 Views
Powered by