rapid-crayon-8232
03/08/2022, 5:46 PMpython_requirement(
name="tensorflow",
requirements=["tensorflow==2.3.0"],
modules=["tensorflow", "numpy", "scipy", "h5py"],
)
resources(name="config-files", sources=[...])
python_sources(
name="lib",
dependencies=[":c-extensions", ":config-files", "!//:numpy", ":tensorflow"],
)
and the warning:
18:41:06.42 [WARN] The target my_target:../../../legacy imports `numpy`, but Pants cannot safely infer a dependency because more than one target owns this module, so it is ambiguous which to use: ['//:numpy', 'my_target:tensorflow'].
Please explicitly include the dependency you want in the `dependencies` field of my_target:../../../legacy, or ignore the ones you do not want by prefixing with `!` or `!!` so that one or no targets are left.
Alternatively, you can remove the ambiguity by deleting/changing some of the targets so that only 1 target owns this module. Refer to <https://www.pantsbuild.org/v2.9/docs/troubleshooting#import-errors-and-missing-dependencies>.
numpy
dep, but it still inferred when running the test
goal for exampleenough-analyst-54434
03/08/2022, 5:53 PMrapid-crayon-8232
03/08/2022, 5:56 PMenough-analyst-54434
03/08/2022, 5:58 PMpython_requirement(
name="tensorflow",
requirements=["tensorflow==2.3.0"],
)
resources(name="config-files", sources=[...])
python_sources(
name="lib",
dependencies=[":c-extensions", ":config-files"],
)
I.E.: Why do you want to exclude numpy if tensorflow depends on it (transitively)?^jsirois@gill ~/Downloads $ zipinfo tensorflow-2.3.0-cp38-cp38-win_amd64.whl | tail
-rw-rw-rw- 2.0 fat 1332 b- defN 20-Jul-24 04:19 tensorflow/xla_aot_runtime_src/tensorflow/core/platform/default/dynamic_annotations.h
-rw-rw-rw- 2.0 fat 1269 b- defN 20-Jul-24 04:19 tensorflow/xla_aot_runtime_src/tensorflow/core/platform/default/integral_types.h
-rw-rw-rw- 2.0 fat 2399 b- defN 20-Jul-24 04:19 tensorflow/xla_aot_runtime_src/tensorflow/core/platform/windows/env_time.cc
-rw-rw-rw- 2.0 fat 11419 b- defN 20-Jul-24 04:19 tensorflow-2.3.0.dist-info/LICENSE
-rw-rw-rw- 2.0 fat 2648 b- defN 20-Jul-24 04:19 tensorflow-2.3.0.dist-info/METADATA
-rw-rw-rw- 2.0 fat 105 b- defN 20-Jul-24 04:19 tensorflow-2.3.0.dist-info/WHEEL
-rw-rw-rw- 2.0 fat 469 b- defN 20-Jul-24 04:19 tensorflow-2.3.0.dist-info/entry_points.txt
-rw-rw-rw- 2.0 fat 11 b- defN 20-Jul-24 04:19 tensorflow-2.3.0.dist-info/top_level.txt
?rw-rw-r-- 2.0 fat 925731 b- defN 20-Jul-24 04:20 tensorflow-2.3.0.dist-info/RECORD
7502 files, 979933102 bytes uncompressed, 340951205 bytes compressed: 65.2%
^jsirois@gill ~/Downloads $ unzip -qc tensorflow-2.3.0-cp38-cp38-win_amd64.whl tensorflow-2.3.0.dist-info/top_level.txt
tensorflow
^jsirois@gill ~/Downloads $ unzip -qc tensorflow-2.3.0-cp38-cp38-win_amd64.whl tensorflow-2.3.0.dist-info/METADATA | grep Requires
Requires-Dist: absl-py (>=0.7.0)
Requires-Dist: astunparse (==1.6.3)
Requires-Dist: gast (==0.3.3)
Requires-Dist: google-pasta (>=0.1.8)
Requires-Dist: h5py (<2.11.0,>=2.10.0)
Requires-Dist: keras-preprocessing (<1.2,>=1.1.1)
Requires-Dist: numpy (<1.19.0,>=1.16.0)
Requires-Dist: opt-einsum (>=2.3.2)
Requires-Dist: protobuf (>=3.9.2)
Requires-Dist: tensorboard (<3,>=2.3.0)
Requires-Dist: tensorflow-estimator (<2.4.0,>=2.3.0)
Requires-Dist: termcolor (>=1.1.0)
Requires-Dist: wrapt (>=1.11.1)
Requires-Dist: wheel (>=0.26)
Requires-Dist: six (>=1.12.0)
Requires-Dist: scipy (==1.4.1)
Requires-Dist: grpcio (>=1.8.6)
rapid-crayon-8232
03/08/2022, 6:00 PMThe conflict is caused by:
The user requested numpy<2.0.0 and >=1.22.3
tensorflow 2.3.0 depends on numpy<1.19.0 and >=1.16.0
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
enough-analyst-54434
03/08/2022, 6:01 PMrapid-crayon-8232
03/08/2022, 6:01 PMpython_requirement(name="tensorflow", requirements=["tensorflow==2.3.0"])
python_requirement(name="numpy", requirements=["numpy==1.18.5"])
python_sources(
name="lib",
dependencies=[":c-extensions", ":config-files", ":numpy", ":tensorflow"],
)
But I still get the warning, because I import numpy in my tests, and dependency inference picks up both of themnumpy
is imported (not the rest), so it's probably a bug with dependency inference logging incorrect warnings ?python_requirement(name="tensorflow", requirements=["tensorflow==2.3.0"])
python_requirement(name="numpy", requirements=["numpy==1.18.5"])
python_sources(
name="lib",
dependencies=[":c-extensions", ":config-files", ":numpy", ":tensorflow", "!//:numpy"],
)
enough-analyst-54434
03/08/2022, 6:14 PMrapid-crayon-8232
03/08/2022, 6:21 PMhundreds-father-404
03/08/2022, 7:00 PMrapid-crayon-8232
03/08/2022, 7:50 PMhundreds-father-404
03/08/2022, 7:52 PMI'll wait for the stable release of 2.10 for the switchIf you are willing to create a branch that updates to 2.10, but not yet land it necessarily, that is super super helpful to us to have a more stable release
rapid-crayon-8232
03/09/2022, 9:00 AM./pants update-build-files --fix-python-macros
:
* `poetry_requirements` in BUILD: add `name="poetry"
09:57:31.02 [ERROR] You must manually add the `name=` field to the following targets. This is not done automatically by the `update-build-files` goal.
* `poetry_requirements` in BUILD: add `name="poetry"
09:57:31.02 [ERROR] You must manually add the `name=` field to the following targets. This is not done automatically by the `update-build-files` goal.
* `poetry_requirements` in BUILD: add `name="poetry"
09:57:31.02 [ERROR] You must manually add the `name=` field to the following targets. This is not done automatically by the `update-build-files` goal.
* `poetry_requirements` in BUILD: add `name="poetry"
Updated BUILD:
- Update references to targets generated by `poetry_requirements`
poetry_requireents
doesn't have a field name
, no ?default
and legacy
for example) I can't have the legacy
poetry_requirements
only containing the subset of dependencies that are different from the default
one, and for it to be resolved automatically with dep inference, example getting requests from default
pyproject.toml and numpy
from legacu pyproject.toml when building the python_sources
?Yes, I have abranch running with pants 2.10, I'm note sure I understand this warnings thought when running./pants update-build-files --fix-python-macros
Fixed, it was just an issue of when to toggle
use_deprecated_python_macros
, set to true
before running the update-build-files
and then set to false
before fixing the files based on the errorshundreds-father-404
03/09/2022, 4:29 PMpoetry_requireents doesn't have a field name , no ?It does if you set
use_deprecated_python_macros = false
because poetry_requirements
becomes a proper target, e.g. shows up in ./pants help targets
I can't have the legacy poetry_requirements only containing the subset of dependencies that are different from the default one, and for it to be resolved automatically with dep inferencePants 2.11 will make this easier through a new parametrize feature
poetry_requirements(
name="poetry",
# All entries from this pyproject.toml will be usable by both resolves.
resolve=parametrize("default", "legacy"),
)
Parametrize will create 2 python_requirement
targets for each entry: one with the default
resolve and another with the legacy
resolve.
In the meantime, you would create two distinct target generators here, which have have roughly the same effect:
poetry_requirements(
name="poetry-default",
resolve="default",
)
poetry_requirements(
name="poetry-legacy",
resolve="legacy",
)
Does that make sense?rapid-crayon-8232
03/09/2022, 4:41 PMgenerate-lockfile
goal because we use some custom indexes apart from Pypi, switched to shell commands to generate lock files
#!/bin/bash
set -eou pipefail
POETRY_LOCK_DIR=${POETRY_LOCK_DIR:-"."}
echo "Resoving dependencies in '$POETRY_LOCK_DIR'"
cd $POETRY_LOCK_DIR
poetry lock --no-update
echo "Exporting poetry.lock to a requirements.txt format ..."
poetry export --without-hashes -f requirements.txt -o lock.txt
echo "... Done"
experimental_run_shell_command(
name="lock",
command="./monorepo/build-support/poetry/lock.sh",
)
experimental_run_shell_command(
name="lock-legacy",
command="POETRY_LOCK_DIR=./monorepo/legacy ./monorepo/build-support/poetry/lock.sh",
)
hundreds-father-404
03/09/2022, 4:48 PM[python].resolves_generate_lockfiles = false
so that you can still use generate-lockfiles
goal for tool lockfiles?rapid-crayon-8232
03/09/2022, 5:08 PMIn the meantime, did you see the flagYep, that's the what I did ^^ I'm struggling a bit again with 3rd party deps, If I can have your help 🙏 I have the following BUILD fileso that you can still use[python].resolves_generate_lockfiles = false
goal for tool lockfiles?generate-lockfiles
poetry_requirements(
name="poetry",
module_mapping={"javaobj-py3": ["javaobj"]},
overrides={
"fastapi": {"dependencies": [":poetry#itsdangerous"]},
"tensorboard": {"dependencies": [":poetry#wheel", ":poetry#setuptools"]},
"pytorch-lightning": {"dependencies": [":poetry#wheel", ":poetry#setuptools"]},
},
)
When I try to run a pex binary that depends on the pytorch-lightning
I get the error:
17:50:01.57 [INFO] Completed: Installing lock.txt for the resolve `default`
17:50:02.57 [INFO] Completed: Extracting 3 requirements to build cli.pex from default_lockfile.pex: pytorch-lightning<2.0.0,>=1.5.10, setuptools==59.5.0, wheel<0.38.0,>=0.37.1
17:50:02.57 [ERROR] 1 Exception encountered:
ProcessExecutionFailure: Process 'Extracting 3 requirements to build cli.pex from default_lockfile.pex: pytorch-lightning<2.0.0,>=1.5.10, setuptools==59.5.0, wheel<0.38.0,>=0.37.1' failed with exit code 1.
stdout:
stderr:
Failed to resolve requirements from PEX environment @ /home/obendidi/.cache/pants/named_caches/pex_root/unzipped_pexes/257b2a9c82961e432f412c83fef2fb3ddbe8ed7b.
Needed cp38-cp38-manylinux_2_31_x86_64 compatible dependencies for:
1: setuptools>=41.0.0
Required by:
FingerprintedDistribution(distribution=tensorboard 2.8.0 (/home/obendidi/.cache/pants/named_caches/pex_root/installed_wheels/8aac3016bf8745198e406b5a8ffad4f30454cc0b/tensorboard-2.8.0-py3-none-any.whl), fingerprint='6d11e0a4273464e01a0d11f770f96302ddf6f720')
But this pex had no 'setuptools' distributions.
2: wheel>=0.26
Required by:
FingerprintedDistribution(distribution=tensorboard 2.8.0 (/home/obendidi/.cache/pants/named_caches/pex_root/installed_wheels/8aac3016bf8745198e406b5a8ffad4f30454cc0b/tensorboard-2.8.0-py3-none-any.whl), fingerprint='6d11e0a4273464e01a0d11f770f96302ddf6f720')
But this pex had no 'wheel' distributions.
3: setuptools==59.5.0
Required by:
FingerprintedDistribution(distribution=pytorch-lightning 1.5.10 (/home/obendidi/.cache/pants/named_caches/pex_root/installed_wheels/1b178d9142a72ecbd1ca4ff6f71df50367efeed9/pytorch_lightning-1.5.10-py3-none-any.whl), fingerprint='d548245c67fb01318d04b12a747e35e3165f0f94')
But this pex had no 'setuptools' distributions.
4: wheel<0.38.0,>=0.37.1
But this pex had no 'wheel' distributions.
Use `--no-process-cleanup` to preserve process chroots for inspection.
hundreds-father-404
03/09/2022, 5:14 PMdefault_lock.py
is missing setuptools & wheel in it?
Note that this is failing after Pants has already installed default_lock.txt, and then now Pants is trying to extract a subset of the PEX so that you have finer-grained caching. When you install default_lock.txt. Pants/Pex does so naively - whatever you have it in, Pants goes with. No validation, particularly because this lockfile is being manually managedrapid-crayon-8232
03/09/2022, 5:26 PMpython_requiremnt
directly to avoid thishundreds-father-404
03/09/2022, 5:29 PMI'll probably try to set them as python_requiremnt directly to avoid thisI don't think that will help you with manual lockfile management 😕 The best we can do is install the lock you give us. We don't install anything extra you ask because that would be violating the premise of a lock: that we only install the exact artifacts the lock contains I think that's actually good feedback for this question, it's hard to get manual lockfile management right https://pantsbuild.slack.com/archives/C0D7TNJHL/p1646844293306559
setuptools
and wheel
to the lock. These don't have transitive deps, so you only should need to add those. You will need --hash
for this to work properly, assuming you keep --hash
for everything else.
2) Use alpha Poetry
3) figure out if you don't need the setuptools
and wheel
deps, like if you can use a new version of pytorch-lightning
4) try out Pants 2.11 with Pex lockfile generation, hopefully released at the end of this week or next weekrapid-crayon-8232
03/09/2022, 5:37 PM