Is there a way to add poetry requirements to the dependencie Pants #general

Join Slack

Is there a way to add poetry_requirements to the d...

# general

bland-father-19717

03/05/2022, 2:35 PM

Is there a way to add poetry_requirements to the dependencies of pex_binary? （For more information, see thread.）

bland-father-19717

03/05/2022, 2:38 PM

The following BUILD file and pyproject.toml environment does not install

absl

when run or repl. BUILD

Copy code

poetry_requirements()

python_sources(
    sources=["**/*.py"],
)

pyproject.toml

Copy code

[tool.poetry.dependencies]
python = "^3.8"
numpy = "^1.22.1"
pytz = "^2021.3"
absl-py = "^1.0.0"

bland-father-19717

03/05/2022, 2:40 PM

The following works correctly, but I would like a way to specify all python_requirements created by poetry_requirements as dependencies together if possible. BUILD

Copy code

poetry_requirements()

python_sources(
    sources=["**/*.py"],
    dependencies=[
        ":absl-py",
        ":numpy",
        ":pytz"
    ]
)

pyproject.toml

Copy code

[tool.poetry.dependencies]
python = "^3.8"
numpy = "^1.22.1"
pytz = "^2021.3"
absl-py = "^1.0.0"

hundreds-father-404

03/05/2022, 3:01 PM

There is if you upgrade to 2.10.0rc2 and follow the deprecation instructions to set

use_deprecated_python_macros = false

! (Don't set that immediately, read the deprecation instructions first). That change switches

poetry_requirements

to use the new "target generation" feature. One of the improvements thanks to that is that

poetry_requirements

becomes a normal target like anything else, meaning that you can address it. When you depend on it, it's an ~alias for depending on everything it generates See https://www.pantsbuild.org/docs/upgrade-tips

hundreds-father-404

03/05/2022, 3:04 PM

Gr, it's very unlikely that

update-build-files

will work properly to automate making this change though. We cherry-picked a fix this week but the release hasn't gone out yet and I'm blocked due to some temporary internal infrastructure stuff (I pinged other maintainers) If you're trying to do this today, you can use the

PANTS_SHA

feature mentioned in the

./pants

script with this commit: 84c0d295557fb97168ccc696cda60c9bf5e57497. That is what 2.10.0rc3 will be based on

bland-father-19717

03/05/2022, 3:24 PM

Thanks for the quick reply, Thats a great update! I will try it after 2.10.0 is released. looking forward to the 2.10.0 release. Thanks!

happy-kitchen-89482

03/06/2022, 1:01 AM

Curious why dependency inference isn't doing the right thing automatically here?

👍 1

bland-father-19717

03/06/2022, 10:18 AM

That was my concern too. How is the target source code for dependency inference determined? Is it subject to python_sources?

happy-kitchen-89482

03/06/2022, 5:10 PM

It works with your local code and with 3rd party code

happy-kitchen-89482

03/06/2022, 5:10 PM

In this case the problem is with 3rd party code

happy-kitchen-89482

03/06/2022, 5:11 PM

For example, if the code in your

python_sources

imports from

numpy

then that

:numpy

dep should be added automatically

happy-kitchen-89482

03/06/2022, 5:12 PM

Are you seeing any

WARN

logs in the output about multiple targets providing the same symbol or something like that?

happy-kitchen-89482

03/06/2022, 5:12 PM

Dep inference only works if there is no ambiguity

happy-kitchen-89482

03/06/2022, 5:12 PM

Do you have multiple

requirements.txt

for example?

bland-father-19717

03/07/2022, 1:39 AM

I created and added source code that will be reproduced in the repository.

./pants repl src/python/lib/lib_a

Copy code

[INFO] Completed: Building requirements.pex with 1 requirement: numpy==1.20.1
Python 3.8.12 (default, Feb  5 2022, 23:27:32) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import absl
Traceback (most recent call last):
  File "<console>", line 1, in <module>
ModuleNotFoundError: No module named 'absl'

./pants run src/python/main/main_a:main

Copy code

File "xxx/pants-sample/.pants.d/tmp9ukio0z4/src/python/main/main_a/main.py", line 2, in <module>
    from lib.lib_a.utils.log import set_logging_basic_config
  File "xxx/pants-sample/.pants.d/tmp9ukio0z4/src/python/lib/lib_a/utils/log.py", line 3, in <module>
    import absl.logging
ModuleNotFoundError: No module named 'absl'

enough-analyst-54434

03/07/2022, 1:59 AM

That repo works for me @bland-father-19717 I can even trim back and it still works:

Copy code

$ git diff src/
diff --git a/src/python/main/main_a/BUILD b/src/python/main/main_a/BUILD
index bec269e..e96c61a 100644
--- a/src/python/main/main_a/BUILD
+++ b/src/python/main/main_a/BUILD
@@ -2,10 +2,9 @@ pex_binary(
     name="main",
     entry_point="main.py",
     dependencies=[
-        ":lib",
     ],
     platforms=["linux-x86_64-cp-38-cp38"]
 )
-python_sources(name="lib", dependencies=["src/python/lib/lib_a"])
+python_sources(name="lib")
 
 docker_image(name="docker", dependencies=["src/python/main/main_a:main"])

Then:

Copy code

$ ./pants run src/python/main/main_a:main
[0. 0. 0. 0. 0.]

👀 1

happy-kitchen-89482

03/07/2022, 2:08 AM

which pants version is this?

happy-kitchen-89482

03/07/2022, 2:11 AM

I suspect this is because you don't have https://github.com/pantsbuild/pants/pull/14604, which is only in 2.11.0.dev2 and up

enough-analyst-54434

03/07/2022, 2:14 AM

That example repo @bland-father-19717 linked is 2.9.0.

happy-kitchen-89482

03/07/2022, 2:17 AM

And it worked for you with 2.9.0 and not from source? That is surprising since it lacks the absl-py -> absl module mapping

enough-analyst-54434

03/07/2022, 2:17 AM

Yes, worked for me straight up.

enough-analyst-54434

03/07/2022, 2:18 AM

I must've had the wrong branch?

bland-father-19717

03/07/2022, 2:20 AM

Probably yes. https://github.com/peachanG/pants-sample/tree/feature/add_lib_a_utils

bland-father-19717

03/07/2022, 2:20 AM

feature/add_lib_a_utils

happy-kitchen-89482

03/07/2022, 2:21 AM

Try adding this to the

poetry_requirements

target:

Copy code

poetry_requirements(
    module_mapping={'absl-py': ['absl']},
)

🎉 1

bland-father-19717

03/07/2022, 2:22 AM

@happy-kitchen-89482 Great, it worked.

happy-kitchen-89482

03/07/2022, 2:31 AM

Aha!

happy-kitchen-89482

03/07/2022, 2:31 AM

That will be in the defaults in 2.11, but until then that should tide you over

happy-kitchen-89482

03/07/2022, 2:32 AM

Pants assumes that a distribution named "foo" provides a top-level module named "foo". This guess is correct about 90% of the time. When it isn't we add an explicit mapping.

happy-kitchen-89482

03/07/2022, 2:32 AM

Over time we build up the default mapping for the remaining 10%, but absl only made it into that in 2.11

bland-father-19717

03/07/2022, 2:33 AM

Thanks! It is very tiring to have to manage Module mapping with Pants… 😇

enough-analyst-54434

03/07/2022, 2:40 AM

Hrm, I find that surprising, but I am used to having to list dependencies manually from back in the bad old days! It seems we could add a Pants goal to edit requirements (add them) that could download the requirement and inspect it's top-level.txt and auto configure this, but there are many details and devils there.

bland-father-19717

03/07/2022, 2:47 AM

As a user, it is very helpful to have the build tool manage the module mapping.

hundreds-father-404

03/07/2022, 3:21 AM

this was cherry-picked into the next 2.10 release candidate, so it will be in 2.10 stable. I'll finish the release tomorrow morning

happy-kitchen-89482

03/07/2022, 3:40 AM

Just to clarify, @bland-father-19717, are you saying that it's good that Pants does this, or that there is some issue that still makes this management tiring?

bland-father-19717

03/07/2022, 3:48 AM

@hundreds-father-404 Thanks! @happy-kitchen-89482 Sorry for the confusing wording. I think that it’s good that Pants does this. Thanks to the pants contributors for managing this!

👍 2

7 Views

Open in Slack

Previous Next