For fun dependency resolution reasons thanks for pipping `pi Pants #general

For "fun" dependency resolution reasons (thanks fo...

curved-manchester-66006

12/21/2022, 4:21 AM

For "fun" dependency resolution reasons (thanks for pipping

pip_version

thru!) this is on

2.16.0.dev3

so feel free to defer. My question is basically a gut check on "this looks like source roots or other common onboarding confusion" or "probably something with the dev release, check back in January". I'm getting variants of this error message:

Copy code

./pants run scripts/hello.py 
Traceback (most recent call last):
...
ImportError: No module named scripts.hello

🧵

✅ 1

curved-manchester-66006

12/21/2022, 4:23 AM

Copy code

$ ./pants roots
.
$ cat scripts/hello.py 
import modules.critical

print('hello')
$ cat modules/hello.py 
import modules.critical

print('hello')
$ ./pants run modules/hello.py 
hello
$ ./pants run modules/critical.py
$ ./pants run scripts/hello.py 
Traceback (most recent call last):
  File "/tmp/pants-sandbox-uOaYuI/./.cache/pex_root/venvs/27fadc29d37cd72af28911c9a5a9bb68222b5d90/4f398b5364288224a985fd546e447fdd048b6796/pex", line 248, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/usr/local/lib/python3.7/runpy.py", line 201, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/usr/local/lib/python3.7/runpy.py", line 136, in _get_module_details
    raise error("No module named %s" % mod_name)
ImportError: No module named scripts.hello

Where

modules.critical

is a real class with various 1st and 3rd party dependencies. However:

Copy code

$ cat modules/bar.py 
bar='bar'
$ cat scripts/hello2.py 
import modules.bar

print('hello')
$ ./pants run scripts/hello2.py 
hello

I have not been able to find a repo case between the trivial example and the real code that blows up.

curved-television-6568

12/21/2022, 3:15 PM

strange (thanks for trying out the dev releases, that helps find issues earlier!)

curved-television-6568

12/21/2022, 3:18 PM

couple of Q’s that comes to mind: if you comment out the

import modules.critical

scripts/hello.py

does it work then? do you miss any

__init__.py

files? does the

./pants dependencies --transitive scripts/hello.py

look sane? if you run with

--keep-sandboxes=always

and inspect/run the

__run.sh

from there (also check what files you have in the sandbox may be instructive)

curved-manchester-66006

12/21/2022, 3:54 PM

f you comment out the
import modules.critical
in
scripts/hello.py
does it work then?

Yes.

Copy code

$ cat scripts/hello.py 
#import modules.critical

print('hello')
$ ./pants run scripts/hello.py 
hello

do you miss any
__init__.py
files?

Copy code

$ ls modules/__init__.py 
modules/__init__.py
$ ls scripts/__init__.py 
scripts/__init__.py

I get the same behavior with and without a

__init__.py

at the repo root (eg

$ touch __init__.py

)

does the

./pants dependencies --transitive scripts/hello.py

look sane?

It's not obviously wrong to me, but it is also > 100 lines long. Like I said I haven't been able to get this to a minimal reproduction.

if you run with
--keep-sandboxes=always
and inspect/run the
__run.sh
from there (also check what files you have in the sandbox may be instructive)

I'd totally believe this is something silly with

__init__..py

files or whatnot, but I'm still not seeing it. It seems to work when I run it interactively.

Copy code

/tmp/pants-sandbox-jxnlqA $ ./scripts.pex_pex_shim.sh
Traceback (most recent call last):
  File "/tmp/pants-sandbox-jxnlqA/.cache/pex_root/venvs/27fadc29d37cd72af28911c9a5a9bb68222b5d90/4f398b5364288224a985fd546e447fdd048b6796/pex", line 248, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/usr/local/lib/python3.7/runpy.py", line 201, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/usr/local/lib/python3.7/runpy.py", line 136, in _get_module_details
    raise error("No module named %s" % mod_name)
ImportError: No module named scripts.hello

Copy code

/tmp/pants-sandbox-jxnlqA $ ./scripts.pex_bin_python_shim.sh
Python 3.7.15 (default, Dec 19 2022, 16:05:03) 
[GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import scripts.hello
/tmp/pants-sandbox-jxnlqA/.cache/pex_root/venvs/27fadc29d37cd72af28911c9a5a9bb68222b5d90/4f398b5364288224a985fd546e447fdd048b6796/lib/python3.7/site-packages/airflow/configuration.py:276: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if StrictVersion(sqlite3.sqlite_version) < StrictVersion(min_sqlite_version):
hello
>>>

(Thanks for taking a look!)

👍 1

curved-television-6568

12/21/2022, 4:39 PM

the sandbox scenario seems to be worth exploring, as you’ve got a working and a failing scenario from there.. I’d look at what the sys.path and relevant env vars are for each of those.. 🙂

curved-manchester-66006

12/21/2022, 9:34 PM

Thank you for your suggestion regarding the sandbox! Although I think I've now found a python packaging footgun and not a pants bug. The

sys.path

in the sanbox looks (from my dumb print statement debugging) to be:

['/home/ecsb/.cache/pants/named_caches/pex_root/venvs/27fadc29d37cd72af28911c9a5a9bb68222b5d90/4f398b5364288224a985fd546e447fdd048b6796', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/home/ecsb/.cache/pants/named_caches/pex_root/venvs/s/19a57b50/venv/lib/python3.7/site-packages']

Inside the first part there is.... a lot of stuff

Copy code

ls -1 /home/ecsb/.cache/pants/named_caches/pex_root/venvs/27fadc29d37cd72af28911c9a5a9bb68222b5d90/4f398b5364288224a985fd546e447fdd048b6796/lib/python3.7/site-packages/ | wc -l

Notable:

Copy code

ls  /home/ecsb/.cache/pants/named_caches/pex_root/venvs/27fadc29d37cd72af28911c9a5a9bb68222b5d90/4f398b5364288224a985fd546e447fdd048b6796/lib/python3.7/site-packages/scripts/
build_api_docs.py            check_for_technical_terms.py  __init__.py         validate_docs_snippet_line_numbers.py
build_glossary_page.py       check_type_hint_coverage.py   __pycache__
check_docstring_coverage.py  docs_link_checker.py          trace_docs_deps.py

^^^ Some transitive dep which also has a top level

scripts

module!!! Back to my original example, if I do

Copy code

$ cp -ra scripts crazy_scripts
$ ./pants   run crazy_scripts/hello.py 
16:32:25.50 [INFO] Initializing scheduler...
16:32:26.10 [INFO] Scheduler initialized.
hello

It works! I guess to tie it back to pants, this is something it could maybe warn about?

curved-television-6568

12/21/2022, 10:11 PM

AH! yeah, that would’ve been a nice service for sure 👍

3 Views

Open in Slack

Previous Next