https://pantsbuild.org/ logo
Join Slack
Powered by
I've got a simple script (`//test/python/broker/mo...
# general
a
I've got a simple script (
//test/python/broker/mock_service.py
) that a test runs via 
POpen
and then checks the client and service can talk to each other. So I've created a 
pex_binary
target in pants, and added that to the 
runtime_package_dependencies
field in my tests. but when the binary runs, it just errors with 
ModuleNotFoundError: No module named 'broker.mock_service'
what am I missing? Is it something to do with the tests and the actual library sitting under same-named parent folders? (in this case, 
broker
)
e
Quite likely. 1st, can you read this and report whether it's news to you? If so, then your solution is probably in hand. If not and you grok Python's handling of namespace packages, I can help you dig more: https://packaging.python.org/guides/packaging-namespace-packages/#creating-a-namespace-package
a
ah, there is an 
src/python/broker/___init___.py
with ... stuff in it
so it's a non-namespace package
e
Yup - ok. Sounds like you're on the right track then?
a
yeah, guess I've got to move things; or rename something to something slightly different.
e
Exactly.
This is a super-common Python gotcha - you are very much not alone being surprised by this.
a
I seem to recall 
Please
just munging everything together to get past this limitation. But that was 5+ years ago, so I could be wrong
e
It seems like Please could only get away with that if: 1. There was exactly one non-empty 
__init__.py
. 2. They were a bit reckless an concatted all 
__init__.py
?
a
Please was a hot mess that didn't even work OOTB for their own projects. So I'm going to go with 2.
e
Hah
a
feels a bit cumbersome moving a single file to somewhere completely different, then defining a 
python_library
and a 
pex_binary
all to make a single file executable.
e
To make sure we're on the same page: this is fundamentally a Python problem. Do you agree? Do you see something Pants could do that is unambiguous and always applicable to help you get past this Python quirk that we're not doing?
At least for the move a file part. I think all the Pants devs agree you shouldn't need to define a python_library in nearly all cases nor a pex_binary in some cases. Do you know about the 
./pants tailor
goal by the way? That's a bridge to reduce manual boilerplate.
a
Not sure tbh. Because this single script doesn't need to be placed in 
broker/mock_service.py
along with the 
broker
library. It could be just 
slslskdfjsdflsdf.py
But given how pants maps things based on the 
root_patterns
it's just an inconvenient conflict that has arisen from separating 
src/python/<module>
and 
test/python/<module>
e
Right, but that's what I'm contending is a Python problem. You have a 
sys.path
/ 
PYTHONPATH
of 
["src/python","test/python"]
and - Pants aside - Python requires proper namespace packages for you to split a package across both those 
sys.path
entries.
a
I think everyone's to blame? • I wrote a shitty little script with a 
#!/usr/bin/env python
and a 
if ___name___ == "___main__":_
• Pants suggested 
/src/python/<package>
and 
/test/python/<package>
as filepaths • Pex wants everything installed as a package • and Python just made poor life choices.
e
Pex wants everything installed as a package
I don't follow. Can you explain more?
a
Reading through the pex documentation, There's no way to turn a 
if name == "main"
style script into a pex binary w/ dependencies.
the whole thing's fine I guess. Can't support every way of doing things.
e
pex -D <dir containing script> -e <script module name>
should do it.
Copy code
(pex) ~ $ mkdir src
(pex) ~ $ vi src/script.py
(pex) ~ $ cat src/script.py
#!/usr/bin/env python

if __name__ == "__main__":
    print("Hello from script.")

(pex) ~ $ pex -D src/ -e script -o script.pex
(pex) ~ $ ./script.pex
Hello from script.
(pex) ~ $
I'd like to help if you can help me narrow down what Pants (or Pex) is missing. I feel like we must be close but I'm not understanding the missing link.
a
ok, so, preface: I understand why it is the way it is and I am (more or less) fine with it. I have created a repo on github that journals my journey of running a simple script via pants: https://github.com/xlevus/verbose-pants-script-example It is a story of three commits. Commit 1 - I copied in my code from legacy projects and it didn't work, because of the aforementioned namespace issue. Commit 2 - Moved the conflicting namespace to somewhere else. Only
Copy code
raise error(msg.format(mod_name, type(ex).__name__, ex)) from ex
ImportError: Error while finding module specification for 'examples.mock_script' (ModuleNotFoundError: No module named 'examples')
Commit 3 - Created a single-file 
python_library
and add that as a dependency to the script and 🎉 it works. It feels excessive making to declarations for what amounts to a single-file script. But it's not goingto be a frequent occurrence. Easy to accept it as a constraint of the tool.
e
From commit 1:
Copy code
jsirois@siroisdesign:~/verbose-pants-script-example$ git diff
diff --git a/src/python/a_lib/BUILD b/src/python/a_lib/BUILD
index dfff480..d80ecb5 100644
--- a/src/python/a_lib/BUILD
+++ b/src/python/a_lib/BUILD
@@ -1,4 +1 @@
-
-python_library(
-    name="a_lib",
-)
+python_library()
diff --git a/test/python/a_lib/BUILD b/test/python/a_lib/BUILD
index 740479d..408b173 100644
--- a/test/python/a_lib/BUILD
+++ b/test/python/a_lib/BUILD
@@ -1,7 +1,6 @@
+python_library()
+
 pex_binary(
     name="mock_script",
     entry_point="mock_script.py",
-    dependencies=[
-        "//src/python/a_lib"
-    ],
 )
jsirois@siroisdesign:~/verbose-pants-script-example$ ./pants package test/python/a_lib/::
09:59:19.07 [INFO] Initialization options changed: reinitializing scheduler...
09:59:19.55 [INFO] Scheduler initialized.
09:59:21.21 [INFO] Wrote dist/test.python.a_lib/mock_script.pex
jsirois@siroisdesign:~/verbose-pants-script-example$ dist/test.python.a_lib/mock_script.pex
Foo
jsirois@siroisdesign:~/verbose-pants-script-example$
That's running the 
./pants package
goal. The resulting Pex works as expected. If you could clarify the exact command line you hoped would work at commit 1, that would help.
And thanks alot for the example repo, that very much helps tighten the conversation.
Open in Slack
PreviousNext
Powered by