https://pantsbuild.org/ logo
#general
Title
# general
b

bland-father-19717

03/05/2022, 2:35 PM
Is there a way to add poetry_requirements to the dependencies of pex_binary? (For more information, see thread.)
The following BUILD file and pyproject.toml environment does not install
absl
when run or repl. BUILD
Copy code
poetry_requirements()

python_sources(
    sources=["**/*.py"],
)
pyproject.toml
Copy code
[tool.poetry.dependencies]
python = "^3.8"
numpy = "^1.22.1"
pytz = "^2021.3"
absl-py = "^1.0.0"
The following works correctly, but I would like a way to specify all python_requirements created by poetry_requirements as dependencies together if possible. BUILD
Copy code
poetry_requirements()

python_sources(
    sources=["**/*.py"],
    dependencies=[
        ":absl-py",
        ":numpy",
        ":pytz"
    ]
)
pyproject.toml
Copy code
[tool.poetry.dependencies]
python = "^3.8"
numpy = "^1.22.1"
pytz = "^2021.3"
absl-py = "^1.0.0"
h

hundreds-father-404

03/05/2022, 3:01 PM
There is if you upgrade to 2.10.0rc2 and follow the deprecation instructions to set
use_deprecated_python_macros = false
! (Don't set that immediately, read the deprecation instructions first). That change switches
poetry_requirements
to use the new "target generation" feature. One of the improvements thanks to that is that
poetry_requirements
becomes a normal target like anything else, meaning that you can address it. When you depend on it, it's an ~alias for depending on everything it generates See https://www.pantsbuild.org/docs/upgrade-tips
Gr, it's very unlikely that
update-build-files
will work properly to automate making this change though. We cherry-picked a fix this week but the release hasn't gone out yet and I'm blocked due to some temporary internal infrastructure stuff (I pinged other maintainers) If you're trying to do this today, you can use the
PANTS_SHA
feature mentioned in the
./pants
script with this commit: 84c0d295557fb97168ccc696cda60c9bf5e57497. That is what 2.10.0rc3 will be based on
b

bland-father-19717

03/05/2022, 3:24 PM
Thanks for the quick reply, Thats a great update! I will try it after 2.10.0 is released. looking forward to the 2.10.0 release. Thanks!
h

happy-kitchen-89482

03/06/2022, 1:01 AM
Curious why dependency inference isn't doing the right thing automatically here?
👍 1
b

bland-father-19717

03/06/2022, 10:18 AM
That was my concern too. How is the target source code for dependency inference determined? Is it subject to python_sources?
h

happy-kitchen-89482

03/06/2022, 5:10 PM
It works with your local code and with 3rd party code
In this case the problem is with 3rd party code
For example, if the code in your
python_sources
imports from
numpy
then that
:numpy
dep should be added automatically
Are you seeing any
WARN
logs in the output about multiple targets providing the same symbol or something like that?
Dep inference only works if there is no ambiguity
Do you have multiple
requirements.txt
for example?
b

bland-father-19717

03/07/2022, 1:39 AM
I created and added source code that will be reproduced in the repository.
./pants repl src/python/lib/lib_a
Copy code
[INFO] Completed: Building requirements.pex with 1 requirement: numpy==1.20.1
Python 3.8.12 (default, Feb  5 2022, 23:27:32) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import absl
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ModuleNotFoundError: No module named 'absl'
./pants run src/python/main/main_a:main
Copy code
File "xxx/pants-sample/.pants.d/tmp9ukio0z4/src/python/main/main_a/main.py", line 2, in <module>
    from lib.lib_a.utils.log import set_logging_basic_config
  File "xxx/pants-sample/.pants.d/tmp9ukio0z4/src/python/lib/lib_a/utils/log.py", line 3, in <module>
    import absl.logging
ModuleNotFoundError: No module named 'absl'
e

enough-analyst-54434

03/07/2022, 1:59 AM
That repo works for me @bland-father-19717 I can even trim back and it still works:
Copy code
$ git diff src/
diff --git a/src/python/main/main_a/BUILD b/src/python/main/main_a/BUILD
index bec269e..e96c61a 100644
--- a/src/python/main/main_a/BUILD
+++ b/src/python/main/main_a/BUILD
@@ -2,10 +2,9 @@ pex_binary(
     name="main",
     entry_point="main.py",
     dependencies=[
-        ":lib",
     ],
     platforms=["linux-x86_64-cp-38-cp38"]
 )
-python_sources(name="lib", dependencies=["src/python/lib/lib_a"])
+python_sources(name="lib")
 
 docker_image(name="docker", dependencies=["src/python/main/main_a:main"])
Then:
Copy code
$ ./pants run src/python/main/main_a:main
[0. 0. 0. 0. 0.]
👀 1
h

happy-kitchen-89482

03/07/2022, 2:08 AM
which pants version is this?
I suspect this is because you don't have https://github.com/pantsbuild/pants/pull/14604, which is only in 2.11.0.dev2 and up
e

enough-analyst-54434

03/07/2022, 2:14 AM
That example repo @bland-father-19717 linked is 2.9.0.
h

happy-kitchen-89482

03/07/2022, 2:17 AM
And it worked for you with 2.9.0 and not from source? That is surprising since it lacks the absl-py -> absl module mapping
e

enough-analyst-54434

03/07/2022, 2:17 AM
Yes, worked for me straight up.
I must've had the wrong branch?
b

bland-father-19717

03/07/2022, 2:20 AM
feature/add_lib_a_utils
h

happy-kitchen-89482

03/07/2022, 2:21 AM
Try adding this to the
poetry_requirements
target:
Copy code
poetry_requirements(
    module_mapping={'absl-py': ['absl']},
)
🎉 1
b

bland-father-19717

03/07/2022, 2:22 AM
@happy-kitchen-89482 Great, it worked.
h

happy-kitchen-89482

03/07/2022, 2:31 AM
Aha!
That will be in the defaults in 2.11, but until then that should tide you over
Pants assumes that a distribution named "foo" provides a top-level module named "foo". This guess is correct about 90% of the time. When it isn't we add an explicit mapping.
Over time we build up the default mapping for the remaining 10%, but absl only made it into that in 2.11
b

bland-father-19717

03/07/2022, 2:33 AM
Thanks! It is very tiring to have to manage Module mapping with Pants… 😇
e

enough-analyst-54434

03/07/2022, 2:40 AM
Hrm, I find that surprising, but I am used to having to list dependencies manually from back in the bad old days! It seems we could add a Pants goal to edit requirements (add them) that could download the requirement and inspect it's top-level.txt and auto configure this, but there are many details and devils there.
b

bland-father-19717

03/07/2022, 2:47 AM
As a user, it is very helpful to have the build tool manage the module mapping.
h

hundreds-father-404

03/07/2022, 3:21 AM
this was cherry-picked into the next 2.10 release candidate, so it will be in 2.10 stable. I'll finish the release tomorrow morning
h

happy-kitchen-89482

03/07/2022, 3:40 AM
Just to clarify, @bland-father-19717, are you saying that it's good that Pants does this, or that there is some issue that still makes this management tiring?
b

bland-father-19717

03/07/2022, 3:48 AM
@hundreds-father-404 Thanks! @happy-kitchen-89482 Sorry for the confusing wording. I think that it’s good that Pants does this. Thanks to the pants contributors for managing this!
👍 2
7 Views