https://pantsbuild.org/ logo
Join Slack
Powered by
Hi everyone! Hope you are well! Apologies in advan...
# general
a
Hi everyone! Hope you are well! Apologies in advance for the long message I have an requirements.txt file with 3 libraries that I am trying to resolve on a pex file through pants package. The pex will try to resolve all libraries The 3 libs are
Copy code
albumentations
opencv-python==4.10.0.84
pillow
Now the issue comes from installing 
albumentations
. Their documentation for installing the library states the following
Copy code
pip will use the following logic to determine the required OpenCV distribution:

If your Python environment already contains opencv-python, opencv-contrib-python, opencv-contrib-python-headless or opencv-python-headless pip will use it.
If your Python environment doesn't contain any OpenCV distribution from step 1, pip will download opencv-python-headless.
What I expected to happen is to resolve opencv-python first and then resolve the library But instead what happens is that both wheels are added to the pex file which results in an error when I run the pex file or do 
_PEX_TOOLS=1_ python _/main.pex_
venv
 
--scope=deps
 
--compile
 
/bin/app
 --- What I have tried so far is to • Modify the requirements file and install each lib without any extra dependency
Copy code
# Libs for albumentations
albumentations==1.4.13 --install-option="--no-deps" --no-binary qudida,albumentations 
albucore --install-option="--no-deps"
eval-type-backport --install-option="--no-deps"
Unfortunately this doesn’t work
Copy code
"project_name": "albumentations",
          "requires_dists": [
            "numpy>=1.24",
            "opencv-python-headless>=4.9.0.80",
            "typing-extensions>=4.9.0; python_version < \"3.10\""
          ],
          "requires_python": ">=3.8",
          "version": "0.0.14"
• Added an explicit python dependency on the 3rdparty/python BUILD file but this doesn’t work either
Copy code
python_requirement(
    name="cpu_resolve_albumentations",
    requirements=["albumentations==1.4.13"],
    dependencies=[
        ":cpu#opencv-python"
    ],
    resolve="cpu-resolve"
)
-----
Copy code
python_requirements(
    name="cpu",
    source="requirements-cpu.txt",
    resolve="cpu-resolve",
    overrides={
        "albumentations": {"dependencies": [":cpu_resolve_albumentations"]},
        "opencv-python": {"dependencies": [":cpu#pillow"]},
    },
    module_mapping={"pillow": ["PIL"], "opencv-python": ["cv2"]}
)
✅ 1
b
Ah, libraries that dynamically select dependencies are no fun... Firstly, as you might already have discovered, it looks like the wheel (https://pypi.org/project/albumentations/1.4.14/#files) for 
albumentations
hardcodes a dependency on the headless version (
Requires-Dist: opencv-python-headless>=4.9.0.80
), explaining what appears in the lockfile. Secondly, I think pex locking will typically run in an isolated environment, that doesn't have anything installed already, conceptually similar to a single 
pip install opencv-python albumentations
. It looks like this lib is doing a dynamic look up of what's already installed (https://github.com/albumentations-team/albumentations/blob/b1ebad4c932235918d0aeccf51d070c3080e7516/setup.py#L46) that's expecting a sequence like 
pip install opencv-python; pip install albumentations
. Just to emphasise this, if I run the single install command above, I get both opencv libs installed:
Copy code
Successfully installed PyYAML-6.0.2 albucore-0.0.14 albumentations-1.4.14 annotated-types-0.7.0 eval-type-backport-0.2.0 imageio-2.35.1 lazy-loader-0.4 networkx-3.3 numpy-2.1.0 opencv-python-4.10.0.84 opencv-python-headless-4.10.0.84 packaging-24.1 pillow-10.4.0 pydantic-2.8.2 pydantic-core-2.20.1 scikit-image-0.24.0 scipy-1.14.1 tifffile-2024.8.28 typing-extensions-4.12.2
So, I think your options may be to do one or more of: 1. switch your code to 
opencv-python-headless
and not use 
opencv-python
at all 2. use the pex 
--exclude
flag (
pex --help | grep -A10 -e --exclude
) with https://www.pantsbuild.org/2.22/reference/targets/pex_binary#extra_build_args, e.g. 
extra_build_args=["--exclude=opencv-python-headless"]
3. work with 
albumentations
upstream to have its deps be more declarative/less dynamic (e.g. they have issues like https://github.com/albumentations-team/albumentations/issues/841)
a
Thanks a lot. The 2nd solution works but that means every pex_binary target will have to this 
extra_build_args
property. Is there any way to enforce that on global level, a relevant subsystem I can use?
b
a
Thank you. Worked like a charm. For anyone interested
Copy code
__defaults__({(pex_binary): dict(extra_build_args=["--exclude=opencv-python-headless"])})
👍 1
Open in Slack
PreviousNext
Powered by