https://pantsbuild.org/ logo
#general
Title
# general
a

ambitious-actor-36781

07/13/2021, 6:04 AM
I've got a simple script (
//test/python/broker/mock_service.py
) that a test runs via
POpen
and then checks the client and service can talk to each other. So I've created a
pex_binary
target in pants, and added that to the
runtime_package_dependencies
field in my tests. but when the binary runs, it just errors with
ModuleNotFoundError: No module named 'broker.mock_service'
what am I missing? Is it something to do with the tests and the actual library sitting under same-named parent folders? (in this case,
broker
)
e

enough-analyst-54434

07/13/2021, 6:21 AM
Quite likely. 1st, can you read this and report whether it's news to you? If so, then your solution is probably in hand. If not and you grok Python's handling of namespace packages, I can help you dig more: https://packaging.python.org/guides/packaging-namespace-packages/#creating-a-namespace-package
a

ambitious-actor-36781

07/13/2021, 6:25 AM
ah, there is an
src/python/broker/___init___.py
with ... stuff in it
so it's a non-namespace package
e

enough-analyst-54434

07/13/2021, 6:27 AM
Yup - ok. Sounds like you're on the right track then?
a

ambitious-actor-36781

07/13/2021, 6:28 AM
yeah, guess I've got to move things; or rename something to something slightly different.
e

enough-analyst-54434

07/13/2021, 6:28 AM
Exactly.
This is a super-common Python gotcha - you are very much not alone being surprised by this.
a

ambitious-actor-36781

07/13/2021, 6:30 AM
I seem to recall
Please
just munging everything together to get past this limitation. But that was 5+ years ago, so I could be wrong
e

enough-analyst-54434

07/13/2021, 6:32 AM
It seems like Please could only get away with that if: 1. There was exactly one non-empty
__init__.py
. 2. They were a bit reckless an concatted all
__init__.py
?
a

ambitious-actor-36781

07/13/2021, 6:32 AM
Please was a hot mess that didn't even work OOTB for their own projects. So I'm going to go with 2.
e

enough-analyst-54434

07/13/2021, 6:32 AM
Hah
a

ambitious-actor-36781

07/13/2021, 6:42 AM
feels a bit cumbersome moving a single file to somewhere completely different, then defining a
python_library
and a
pex_binary
all to make a single file executable.
e

enough-analyst-54434

07/13/2021, 6:44 AM
To make sure we're on the same page: this is fundamentally a Python problem. Do you agree? Do you see something Pants could do that is unambiguous and always applicable to help you get past this Python quirk that we're not doing?
At least for the move a file part. I think all the Pants devs agree you shouldn't need to define a python_library in nearly all cases nor a pex_binary in some cases. Do you know about the
./pants tailor
goal by the way? That's a bridge to reduce manual boilerplate.
a

ambitious-actor-36781

07/13/2021, 6:50 AM
Not sure tbh. Because this single script doesn't need to be placed in
broker/mock_service.py
along with the
broker
library. It could be just
slslskdfjsdflsdf.py
But given how pants maps things based on the
root_patterns
it's just an inconvenient conflict that has arisen from separating
src/python/<module>
and
test/python/<module>
e

enough-analyst-54434

07/13/2021, 6:56 AM
Right, but that's what I'm contending is a Python problem. You have a
sys.path
/
PYTHONPATH
of
["src/python","test/python"]
and - Pants aside - Python requires proper namespace packages for you to split a package across both those
sys.path
entries.
a

ambitious-actor-36781

07/13/2021, 6:58 AM
I think everyone's to blame? • I wrote a shitty little script with a
#!/usr/bin/env python
and a
if ___name___ == "___main__":_
• Pants suggested
/src/python/<package>
and
/test/python/<package>
as filepaths • Pex wants everything installed as a package • and Python just made poor life choices.
e

enough-analyst-54434

07/13/2021, 7:00 AM
Pex wants everything installed as a package
I don't follow. Can you explain more?
a

ambitious-actor-36781

07/13/2021, 7:02 AM
Reading through the pex documentation, There's no way to turn a
if name == "main"
style script into a pex binary w/ dependencies.
the whole thing's fine I guess. Can't support every way of doing things.
e

enough-analyst-54434

07/13/2021, 7:05 AM
pex -D <dir containing script> -e <script module name>
should do it.
Copy code
(pex) ~ $ mkdir src
(pex) ~ $ vi src/script.py
(pex) ~ $ cat src/script.py
#!/usr/bin/env python

if __name__ == "__main__":
    print("Hello from script.")

(pex) ~ $ pex -D src/ -e script -o script.pex
(pex) ~ $ ./script.pex
Hello from script.
(pex) ~ $
I'd like to help if you can help me narrow down what Pants (or Pex) is missing. I feel like we must be close but I'm not understanding the missing link.
a

ambitious-actor-36781

07/13/2021, 8:31 AM
ok, so, preface: I understand why it is the way it is and I am (more or less) fine with it. I have created a repo on github that journals my journey of running a simple script via pants: https://github.com/xlevus/verbose-pants-script-example It is a story of three commits. Commit 1 - I copied in my code from legacy projects and it didn't work, because of the aforementioned namespace issue. Commit 2 - Moved the conflicting namespace to somewhere else. Only
Copy code
raise error(msg.format(mod_name, type(ex).__name__, ex)) from ex
ImportError: Error while finding module specification for 'examples.mock_script' (ModuleNotFoundError: No module named 'examples')
Commit 3 - Created a single-file
python_library
and add that as a dependency to the script and 🎉 it works. It feels excessive making to declarations for what amounts to a single-file script. But it's not goingto be a frequent occurrence. Easy to accept it as a constraint of the tool.
e

enough-analyst-54434

07/13/2021, 10:02 AM
From commit 1:
Copy code
jsirois@siroisdesign:~/verbose-pants-script-example$ git diff
diff --git a/src/python/a_lib/BUILD b/src/python/a_lib/BUILD
index dfff480..d80ecb5 100644
--- a/src/python/a_lib/BUILD
+++ b/src/python/a_lib/BUILD
@@ -1,4 +1 @@
-
-python_library(
-    name="a_lib",
-)
+python_library()
diff --git a/test/python/a_lib/BUILD b/test/python/a_lib/BUILD
index 740479d..408b173 100644
--- a/test/python/a_lib/BUILD
+++ b/test/python/a_lib/BUILD
@@ -1,7 +1,6 @@
+python_library()
+
 pex_binary(
     name="mock_script",
     entry_point="mock_script.py",
-    dependencies=[
-        "//src/python/a_lib"
-    ],
 )
jsirois@siroisdesign:~/verbose-pants-script-example$ ./pants package test/python/a_lib/::
09:59:19.07 [INFO] Initialization options changed: reinitializing scheduler...
09:59:19.55 [INFO] Scheduler initialized.
09:59:21.21 [INFO] Wrote dist/test.python.a_lib/mock_script.pex
jsirois@siroisdesign:~/verbose-pants-script-example$ dist/test.python.a_lib/mock_script.pex
Foo
jsirois@siroisdesign:~/verbose-pants-script-example$
That's running the
./pants package
goal. The resulting Pex works as expected. If you could clarify the exact command line you hoped would work at commit 1, that would help.
And thanks alot for the example repo, that very much helps tighten the conversation.