https://pantsbuild.org/ logo
Join Slack
Powered by
Hey :wave:, not sure If I'm doing this correctly, ...
# general
r
Hey 👋, not sure If I'm doing this correctly, but I'm getting warnings about a deps conflict issues, my use case:
Copy code
python_requirement(
    name="tensorflow",
    requirements=["tensorflow==2.3.0"],
    modules=["tensorflow", "numpy", "scipy", "h5py"],
)
resources(name="config-files", sources=[...])
python_sources(
    name="lib",
    dependencies=[":c-extensions", ":config-files", "!//:numpy", ":tensorflow"],
)
and the warning:
Copy code
18:41:06.42 [WARN] The target my_target:../../../legacy imports `numpy`, but Pants cannot safely infer a dependency because more than one target owns this module, so it is ambiguous which to use: ['//:numpy', 'my_target:tensorflow'].

Please explicitly include the dependency you want in the `dependencies` field of my_target:../../../legacy, or ignore the ones you do not want by prefixing with `!` or `!!` so that one or no targets are left.

Alternatively, you can remove the ambiguity by deleting/changing some of the targets so that only 1 target owns this module. Refer to <https://www.pantsbuild.org/v2.9/docs/troubleshooting#import-errors-and-missing-dependencies>.
I've excluded the global 
numpy
dep, but it still inferred when running the 
test
goal for example
e
Does the tensorflow distribution actually contain all those top-level modules / packages?
Those look like modules / packages provided by its transitive dependencies.
I think you maybe just typed too much.
r
ah yes, I understood modules incorrectly then 😅
e
So, does this work?:
Copy code
python_requirement(
    name="tensorflow",
    requirements=["tensorflow==2.3.0"],
)
resources(name="config-files", sources=[...])
python_sources(
    name="lib",
    dependencies=[":c-extensions", ":config-files"],
)
I.E.: Why do you want to exclude numpy if tensorflow depends on it (transitively)?
For a bit too much info on tensorflow, here are its top-level modules and its dependencies:
Copy code
^jsirois@gill ~/Downloads $ zipinfo tensorflow-2.3.0-cp38-cp38-win_amd64.whl | tail
-rw-rw-rw-  2.0 fat     1332 b- defN 20-Jul-24 04:19 tensorflow/xla_aot_runtime_src/tensorflow/core/platform/default/dynamic_annotations.h
-rw-rw-rw-  2.0 fat     1269 b- defN 20-Jul-24 04:19 tensorflow/xla_aot_runtime_src/tensorflow/core/platform/default/integral_types.h
-rw-rw-rw-  2.0 fat     2399 b- defN 20-Jul-24 04:19 tensorflow/xla_aot_runtime_src/tensorflow/core/platform/windows/env_time.cc
-rw-rw-rw-  2.0 fat    11419 b- defN 20-Jul-24 04:19 tensorflow-2.3.0.dist-info/LICENSE
-rw-rw-rw-  2.0 fat     2648 b- defN 20-Jul-24 04:19 tensorflow-2.3.0.dist-info/METADATA
-rw-rw-rw-  2.0 fat      105 b- defN 20-Jul-24 04:19 tensorflow-2.3.0.dist-info/WHEEL
-rw-rw-rw-  2.0 fat      469 b- defN 20-Jul-24 04:19 tensorflow-2.3.0.dist-info/entry_points.txt
-rw-rw-rw-  2.0 fat       11 b- defN 20-Jul-24 04:19 tensorflow-2.3.0.dist-info/top_level.txt
?rw-rw-r--  2.0 fat   925731 b- defN 20-Jul-24 04:20 tensorflow-2.3.0.dist-info/RECORD
7502 files, 979933102 bytes uncompressed, 340951205 bytes compressed:  65.2%
^jsirois@gill ~/Downloads $ unzip -qc tensorflow-2.3.0-cp38-cp38-win_amd64.whl tensorflow-2.3.0.dist-info/top_level.txt
tensorflow
^jsirois@gill ~/Downloads $ unzip -qc tensorflow-2.3.0-cp38-cp38-win_amd64.whl tensorflow-2.3.0.dist-info/METADATA | grep Requires
Requires-Dist: absl-py (>=0.7.0)
Requires-Dist: astunparse (==1.6.3)
Requires-Dist: gast (==0.3.3)
Requires-Dist: google-pasta (>=0.1.8)
Requires-Dist: h5py (<2.11.0,>=2.10.0)
Requires-Dist: keras-preprocessing (<1.2,>=1.1.1)
Requires-Dist: numpy (<1.19.0,>=1.16.0)
Requires-Dist: opt-einsum (>=2.3.2)
Requires-Dist: protobuf (>=3.9.2)
Requires-Dist: tensorboard (<3,>=2.3.0)
Requires-Dist: tensorflow-estimator (<2.4.0,>=2.3.0)
Requires-Dist: termcolor (>=1.1.0)
Requires-Dist: wrapt (>=1.11.1)
Requires-Dist: wheel (>=0.26)
Requires-Dist: six (>=1.12.0)
Requires-Dist: scipy (==1.4.1)
Requires-Dist: grpcio (>=1.8.6)
r
I have a global numpy used by other modules @1.22.0, and tensorflow==2.3.0 only supports numpy < 1.19.0 so with that config I get:
Copy code
The conflict is caused by:
     The user requested numpy<2.0.0 and >=1.22.3
     tensorflow 2.3.0 depends on numpy<1.19.0 and >=1.16.0
 
 To fix this you could try to:
 1. loosen the range of package versions you've specified
 2. remove package versions to allow pip attempt to solve the dependency conflict
e
Aha. Here @hundreds-father-404 will probably want to guide you through multiple resolves. There are other ways to support conflicting dependencies like this in other words on the bleeding edge of Pants.
🙏 1
r
I've tested with:
Copy code
python_requirement(name="tensorflow", requirements=["tensorflow==2.3.0"])
python_requirement(name="numpy", requirements=["numpy==1.18.5"])
python_sources(
    name="lib",
    dependencies=[":c-extensions", ":config-files", ":numpy", ":tensorflow"],
)
But I still get the warning, because I import numpy in my tests, and dependency inference picks up both of them
To be more precise, the tests build correctly and pass using the above, but I get the warning specifically on the test files where 
numpy
is imported (not the rest), so it's probably a bug with dependency inference logging incorrect warnings ?
The following seems to avoid warnings:
Copy code
python_requirement(name="tensorflow", requirements=["tensorflow==2.3.0"])
python_requirement(name="numpy", requirements=["numpy==1.18.5"])
python_sources(
    name="lib",
    dependencies=[":c-extensions", ":config-files", ":numpy", ":tensorflow", "!//:numpy"],
)
e
That looks about right for the current stable state of the art.
r
I talked a bit too fast, apparently it was just a case of the built requirements being cached and reused, so the warnings didn't show 😓, with no-cache I can reproduce the warnings with the above config
h
Indeed! This sounds like a use case for "multiple resolves" (ie multiple lockfiles), which allows you to have conflicting versions of dependencies like NumPy. It has its soft launch in Pants 2.10, soft due to the limitations explained at https://www.pantsbuild.org/v2.10/docs/reference-python#section-enable-resolves. I'm still rewriting the general Python third-party docs on this, but that option and the next option explain how this new feature works
r
ah that's very cool indeed 🤩 , I'll wait for the stable release of 2.10 for the switch (I see it's around the corner)
h
Our top-voted issue in the January 2022 survey! (which we are blogging about shortly)
I'll wait for the stable release of 2.10 for the switch
If you are willing to create a branch that updates to 2.10, but not yet land it necessarily, that is super super helpful to us to have a more stable release
r
Yes, I have abranch running with pants 2.10, I'm note sure I understand this warnings thought when running 
./pants update-build-files --fix-python-macros
:
Copy code
* `poetry_requirements` in BUILD: add `name="poetry"
09:57:31.02 [ERROR] You must manually add the `name=` field to the following targets. This is not done automatically by the `update-build-files` goal.

  * `poetry_requirements` in BUILD: add `name="poetry"
09:57:31.02 [ERROR] You must manually add the `name=` field to the following targets. This is not done automatically by the `update-build-files` goal.

  * `poetry_requirements` in BUILD: add `name="poetry"
09:57:31.02 [ERROR] You must manually add the `name=` field to the following targets. This is not done automatically by the `update-build-files` goal.

  * `poetry_requirements` in BUILD: add `name="poetry"
Updated BUILD:
  - Update references to targets generated by `poetry_requirements`
poetry_requireents
doesn't have a field 
name
, no ?
Another question, If I understood correctly if I want to set 2 lock files resolves, (
default
and 
legacy
for example) I can't have the 
legacy
poetry_requirements
only containing the subset of dependencies that are different from the 
default
one, and for it to be resolved automatically with dep inference, example getting requests from 
default
pyproject.toml and 
numpy
from legacu pyproject.toml when building the 
python_sources
?
Yes, I have abranch running with pants 2.10, I'm note sure I understand this warnings thought when running 
./pants update-build-files --fix-python-macros
Fixed, it was just an issue of when to toggle 
use_deprecated_python_macros
, set to 
true
before running the 
update-build-files
and then set to 
false
before fixing the files based on the errors
👍 1
h
poetry_requireents doesn't have a field name , no ?
It does if you set 
use_deprecated_python_macros = false
because 
poetry_requirements
becomes a proper target, e.g. shows up in 
./pants help targets
I can't have the legacy poetry_requirements only containing the subset of dependencies that are different from the default one, and for it to be resolved automatically with dep inference
Pants 2.11 will make this easier through a new parametrize feature
Copy code
poetry_requirements(
   name="poetry",
   # All entries from this pyproject.toml will be usable by both resolves.
   resolve=parametrize("default", "legacy"),
)
Parametrize will create 2 
python_requirement
targets for each entry: one with the 
default
resolve and another with the 
legacy
resolve. In the meantime, you would create two distinct target generators here, which have have roughly the same effect:
Copy code
poetry_requirements(
   name="poetry-default",
   resolve="default",
)

poetry_requirements(
   name="poetry-legacy",
   resolve="legacy",
)
Does that make sense?
r
yeah makes perfect sense, I'm doing the same and working nice ^^ On another note we unfortunately had to disable the 
generate-lockfile
goal because we use some custom indexes apart from Pypi, switched to shell commands to generate lock files
Copy code
#!/bin/bash
set -eou pipefail

POETRY_LOCK_DIR=${POETRY_LOCK_DIR:-"."}

echo "Resoving dependencies in '$POETRY_LOCK_DIR'"
cd $POETRY_LOCK_DIR
poetry lock --no-update

echo "Exporting poetry.lock to a requirements.txt format ..."
poetry export --without-hashes -f requirements.txt -o lock.txt
echo "... Done"
Copy code
experimental_run_shell_command(
    name="lock",
    command="./monorepo/build-support/poetry/lock.sh",
)

experimental_run_shell_command(
    name="lock-legacy",
    command="POETRY_LOCK_DIR=./monorepo/legacy ./monorepo/build-support/poetry/lock.sh",
)
h
We're working on fixing that this week in 2.11 by switching to Pex for lockfile generation, very close to being enabled 🙂 In the meantime, did you see the flag 
[python].resolves_generate_lockfiles = false
so that you can still use 
generate-lockfiles
goal for tool lockfiles?
r
In the meantime, did you see the flag 
[python].resolves_generate_lockfiles = false
so that you can still use 
generate-lockfiles
goal for tool lockfiles?
Yep, that's the what I did ^^ I'm struggling a bit again with 3rd party deps, If I can have your help 🙏 I have the following BUILD file
Copy code
poetry_requirements(
    name="poetry",
    module_mapping={"javaobj-py3": ["javaobj"]},
    overrides={
        "fastapi": {"dependencies": [":poetry#itsdangerous"]},
        "tensorboard": {"dependencies": [":poetry#wheel", ":poetry#setuptools"]},
        "pytorch-lightning": {"dependencies": [":poetry#wheel", ":poetry#setuptools"]},
    },
)
When I try to run a pex binary that depends on the pytorch-lightning I get the error:
Copy code
17:50:01.57 [INFO] Completed: Installing lock.txt for the resolve `default`
17:50:02.57 [INFO] Completed: Extracting 3 requirements to build cli.pex from default_lockfile.pex: pytorch-lightning<2.0.0,>=1.5.10, setuptools==59.5.0, wheel<0.38.0,>=0.37.1
17:50:02.57 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Extracting 3 requirements to build cli.pex from default_lockfile.pex: pytorch-lightning<2.0.0,>=1.5.10, setuptools==59.5.0, wheel<0.38.0,>=0.37.1' failed with exit code 1.
stdout:

stderr:
Failed to resolve requirements from PEX environment @ /home/obendidi/.cache/pants/named_caches/pex_root/unzipped_pexes/257b2a9c82961e432f412c83fef2fb3ddbe8ed7b.
Needed cp38-cp38-manylinux_2_31_x86_64 compatible dependencies for:
 1: setuptools>=41.0.0
    Required by:
      FingerprintedDistribution(distribution=tensorboard 2.8.0 (/home/obendidi/.cache/pants/named_caches/pex_root/installed_wheels/8aac3016bf8745198e406b5a8ffad4f30454cc0b/tensorboard-2.8.0-py3-none-any.whl), fingerprint='6d11e0a4273464e01a0d11f770f96302ddf6f720')
    But this pex had no 'setuptools' distributions.
 2: wheel>=0.26
    Required by:
      FingerprintedDistribution(distribution=tensorboard 2.8.0 (/home/obendidi/.cache/pants/named_caches/pex_root/installed_wheels/8aac3016bf8745198e406b5a8ffad4f30454cc0b/tensorboard-2.8.0-py3-none-any.whl), fingerprint='6d11e0a4273464e01a0d11f770f96302ddf6f720')
    But this pex had no 'wheel' distributions.
 3: setuptools==59.5.0
    Required by:
      FingerprintedDistribution(distribution=pytorch-lightning 1.5.10 (/home/obendidi/.cache/pants/named_caches/pex_root/installed_wheels/1b178d9142a72ecbd1ca4ff6f71df50367efeed9/pytorch_lightning-1.5.10-py3-none-any.whl), fingerprint='d548245c67fb01318d04b12a747e35e3165f0f94')
    But this pex had no 'setuptools' distributions.
 4: wheel<0.38.0,>=0.37.1
    But this pex had no 'wheel' distributions.



Use `--no-process-cleanup` to preserve process chroots for inspection.
h
My first thought would be if 
default_lock.py
is missing setuptools & wheel in it? Note that this is failing after Pants has already installed default_lock.txt, and then now Pants is trying to extract a subset of the PEX so that you have finer-grained caching. When you install default_lock.txt. Pants/Pex does so naively - whatever you have it in, Pants goes with. No validation, particularly because this lockfile is being manually managed
r
yes the setuptools and wheel are not included in the poetry export: https://github.com/python-poetry/poetry/issues/1584 There is a fix, but it is only available on poetry 1.2.0 alpha release: https://github.com/python-poetry/poetry/pull/2826 I'll probably try to set them as 
python_requiremnt
directly to avoid this
h
I'll probably try to set them as python_requiremnt directly to avoid this
I don't think that will help you with manual lockfile management 😕 The best we can do is install the lock you give us. We don't install anything extra you ask because that would be violating the premise of a lock: that we only install the exact artifacts the lock contains I think that's actually good feedback for this question, it's hard to get manual lockfile management right https://pantsbuild.slack.com/archives/C0D7TNJHL/p1646844293306559
I think your options might be 1) manually add 
setuptools
and 
wheel
to the lock. These don't have transitive deps, so you only should need to add those. You will need 
--hash
for this to work properly, assuming you keep 
--hash
for everything else. 2) Use alpha Poetry 3) figure out if you don't need the 
setuptools
and 
wheel
deps, like if you can use a new version of 
pytorch-lightning
4) try out Pants 2.11 with Pex lockfile generation, hopefully released at the end of this week or next week
r
alpha poetry it is then, I'll try out pants 2.11 when the first rc release is out ^^
❤️ 1
3 Views
Open in Slack
PreviousNext
Powered by