https://pantsbuild.org/ logo
Join Slack
Powered by
I am sure I am going it in the worse possible way....
# general
s
I am sure I am going it in the worse possible way... but how do I do this better?
Copy code
def image_flavor(**kwargs):
    template = kwargs.pop("template", {})
    template.update(kwargs)
    docker_image(**template)

template=dict(
    name="base",
    source="Dockerfile",
    dependencies = [
        ":resources",
        ":reqs"
    ],
)

image_flavor(template=template, name="python-311", extra_build_args={"PYTHON_VERSION=3.11"})
image_flavor(template=template, name="python-312", extra_build_args={"PYTHON_VERSION=3.12"})
image_flavor(template=template, name="python")

image_flavor(template=template, name="cuda-python-311", source="Dockerfile.cuda", extra_build_args={"PYTHON_VERSION=3.11"})
image_flavor(template=template, name="cuda-python-312", source="Dockerfile.cuda", extra_build_args={"PYTHON_VERSION=3.12"})
image_flavor(template=template, name="cuda-python", source="Dockerfile.cuda")
ideally even the BASE_IMAGE becomes an extra_build_arg
e
What is the main goal here? it seems you don't want to just pass args to 
docker_image
directly because you want to pre-bake in some of them (such as dependencies). is that correct? You might consider something like:
Copy code
def app1_docker_image(**kwargs):
    # you could merge a template of defaults here
    # instead of repeated `kwargs.get` below if you prefer
    docker_image(
        name=kwargs.get("name", "base"),
        source=kwargs.get("source", "Dockerfile"),
        dependencies = [
            ":resources",
            ":reqs",
        ],
    )
then use it like:
Copy code
app1_docker_image(source="Dockerfile.cuda")
s
Thank you @elegant-florist-94385, what I want is something similar to the matrix approach on github_actions, which seems to be something parametrize should do, but can't get it to work. I want to get parametrize for 
extra_build_args
for both python (3.11, 3.12, 3.13) and cuda versions (for instance) something that would make it readable, and simple to expand/add new parameters
g
Copy code
docker_image=dict(
    name="base",
    source="Dockerfile",
    dependencies = [
        ":resources",
        ":reqs"
    ],
    **parametrize("python")
    **parametrize("python-311", extra_build_args=["PYTHON_VERSION=3.11"]),
    **parametrize("python-312", extra_build_args=["PYTHON_VERSION=3.12"]),
    **parametrize("cuda-python",  source="Dockerfile.cuda")
    **parametrize("cuda-python-311",  source="Dockerfile.cuda", extra_build_args=["PYTHON_VERSION=3.11"])
    **parametrize("cuda-python-312",  source="Dockerfile.cuda", extra_build_args=["PYTHON_VERSION=3.12"])
)
Could work. It does seem like doing an itertools.product equivalent macro to generate these parametrizations would be quite useful.
s
using the Dockerfile.cuda is a workaround, Ideally I would like to do something like a cartesian product between
Copy code
extra_build_args=["PYTHON_VERSION=3.11", "PYTHON_VERSION=3.12"])
extra_build_args=["BASE_IMAGE=ubuntu", "BASE_IMAGE=cuda_base_image"])
g
I am probably missing what's problematic with that approach. The cartesian product is trivial to implement with what's available in a build file, and you can just copy it from here: https://docs.python.org/3/library/itertools.html#itertools.product. Something like this... just did in notepad so might not be 100% valid.
Copy code
def product(*iterables):
    pools = [tuple(pool) for pool in iterables]

    result = [[]]
    for pool in pools:
        result = [x+[y] for x in result for y in pool]

    for prod in result:
        yield tuple(prod)

named_variants={f"{image}-{python}": [f"BASE_IMAGE={image}", "PYTHON={python}"] for image, python in product(["ubuntu", "cuda"], ["python3.10", "python3.11"])} 
docker_image(
    name="base",
    source="Dockerfile",
    dependencies = [
        ":resources",
        ":reqs"
    ],
    extra_build_args=parametrize(**named_variants)
)
s
thank you! Will test and try it out!
7 Views
Open in Slack
PreviousNext
Powered by