https://pantsbuild.org/ logo
Join Slack
Powered by
Hi all--I'm new to Pantsbuild and finding it a pow...
# general
s
Hi all--I'm new to Pantsbuild and finding it a powerful and helpful tool so far! I'm a bit stuck, however, attempting to do something I have no doubt is well-within the tool's use-case: I have a monorepo of Python packages, most of which are built and deployed as AWS Lambdas ("services"), and one of which is built as a wheel and distributed on PyPI (an SDK). The SDK is a dependency of some of the services. The services are all targeting a specific Python version ("==3.10.*"), while the SDK supports ">=3.9, <4". I'd like to test the SDK (i.e., run 
mypy
as well as 
pytest
-based tests) against all versions of Python that we support. In the current setup, we have only one 
resolve
with an 
interpreter_constraint
set to 
"==3.10.*"
. Everything about the package/deploy process currently works, but, as expected, tests are only running with Python 3.10. I've tried following the docs below to accomplish, ā˜ļø, but nothing that I do has ended without a configuration problem: ā€¢ https://www.pantsbuild.org/2.21/docs/python/overview/interpreter-compatibility#using-multiple-python-versions-in-the-same-project ā€¢ https://www.pantsbuild.org/2.21/docs/python/overview/lockfiles#multiple-lockfiles I'm wondering: is there specific documentation or an example of what I'm trying to accomplish?
a
What you're looking for is 
__defaults__
. You want to set a default for 
interpreter_constraints
(copy pasting from our stuff) 
(python_sources, python_test_utils, python_source)
something like 
["==3.9.*", "==3.10.*", "==3.11.*", "==3.12.*]
, for 
python_tests
you want parametrize of that, and your 
pex_binary
(that you use to deploy), to 
==3.10.*
so, hm. Maybe something like this:
Copy code
__defaults__(
    {
        (python_sources, python_test_utils, python_source): {
            "interpreter_constraints": ["==3.9.*", "==3.10.*", "==3.11.*", "==3.12.*"]
        },
        (python_tests): {
            "interpreter_constraints": parametrize(
                **{
                    "3_9": "==3.9.*",
                    "3_10": "==3.10.*",
                    "3_11": "==3.11.*",
                    "3_12": "==3.12.*",
                }
            )
        },
        (pex_binary): {"interpreter_constraints": ["==3.10.*"]},
    }
)
We wrote a small macro that does this dynamically, while we were upgrading, and we put it at the top of our BUILD files to specify what each library supports, like this:
Copy code
set_interpreter_constraints(supported_versions=["3_10", "3_11"], build_version="3_10")
Oh, right, you're using lockfiles.
You'll need to, let me see what the syntax was
Hm, actually, I don't know if this works with defaults, but
Maybe try this:
Copy code
__defaults__(
    {
        (python_sources, python_test_utils, python_source): {
            "interpreter_constraints": ["==3.9.*", "==3.10.*", "==3.11.*", "==3.12.*"]
        },
        (python_tests): {
            **parametrize("3_9", interpreter_constraints="==3.9.*", resolve="lock-3_9"),
            **parametrize("3_10", interpreter_constraints="==3.10.*", resolve="lock-3_10"),
            **parametrize("3_11", interpreter_constraints="==3.11.*", resolve="lock-3_11"),
            **parametrize("3_12", interpreter_constraints="==3.12.*", resolve="lock-3_12"),
        },
        (pex_binary): {"interpreter_constraints": ["==3.10.*"]},
    }
)
oh, right
Copy code
__defaults__(
    {
        (python_sources, python_test_utils, python_source): {
            "interpreter_constraints": ["==3.9.*", "==3.10.*", "==3.11.*", "==3.12.*"],
            "resolve": parametrize("lock-3_9", "lock-3_10", "lock-3_11", "lock-3_12")
        },
        (python_tests): {
            **parametrize("3_9", interpreter_constraints="==3.9.*", resolve="lock-3_9"),
            **parametrize("3_10", interpreter_constraints="==3.10.*", resolve="lock-3_10"),
            **parametrize("3_11", interpreter_constraints="==3.11.*", resolve="lock-3_11"),
            **parametrize("3_12", interpreter_constraints="==3.12.*", resolve="lock-3_12"),
        },
        (pex_binary): {"interpreter_constraints": ["==3.10.*"]},
    }
)
s
Thanks @ancient-france-42909 for the quick reply! Yep, we're using a lockfile (currently only one). I haven't tried this 
_defaults_
approach yet, but, when I tried parameterizing interpreter constraints more manually, I got a configuration error along the lines of (paraphrasing, from memory) "the default resolve 'python-default' doesn't match the interpreter_constraint 3.9". I can try this out though.
a
Yeah, that's why you use the **parametrize syntax.
If you scroll a bit down, you'll see an example:
Copy code
# Creates four targets:
#
#    example:tests@interpreter_constraints=py2,resolve=lock-a
#    example:tests@interpreter_constraints=py2,resolve=lock-b
#    example:tests@interpreter_constraints=py3,resolve=lock-a
#    example:tests@interpreter_constraints=py3,resolve=lock-b

python_test(
    name="tests",
    source="tests.py",
    interpreter_constraints=parametrize(py2=["==2.7.*"], py3=[">=3.6,<3.7"]),
    resolve=parametrize("lock-a", "lock-b"),
)
Erm, not that one, that's the broken one
Copy code
# Creates two targets:
#
#    example:tests@parametrize=py2
#    example:tests@parametrize=py3

python_test(
    name="tests",
    source="tests.py",
    **parametrize("py2", interpreter_constraints=["==2.7.*"], resolve="lock-a"),
    **parametrize("py3", interpreter_constraints=[">=3.6,<3.7"], resolve="lock-b"),
)
s
Yep, I did come across that in the docs--in that example, it matches one set of interpreter constraints to one "resolve" (presumably a resolve created for a matching Python version), and the other to another "resolve." Is that not necessary in my case?
a
Yup. You need to make a resolve per python 'constraint'.
s
Just for completeness, that means, pragmatically, that if I have one 
requirements.txt
file with all deps for all of the Python packages in my project, I would do something like this?
Copy code
python_requirements(
    name="3rdparty",
    source="requirements.txt",
    resolves=parametrize("lock-3_9", "lock-3_10", "lock-3_11", "lock-3_12")
)
Or is there another way I should be thinking about this?
a
Yes, that's what we do.
Or, well, we do absolutely ungodly things because we have multiple resolves, but the main one is like that.
s
Ha, OK, thank you! Will give this a try.
a
You'll also need to declare them in pants.toml in... 
python.resolves_to_interpreter_constraints
or whatever it's called
heh, I typed the whatever it's called before I looked it up šŸ˜„
šŸ˜ 1
5 Views
Open in Slack
PreviousNext
Powered by