Hi All, i have an issue I'm hoping you can help wi...
# general
s
Hi All, i have an issue I'm hoping you can help with. I have a test which implicitly depends on a 3rd party package (
ssh-python
which isn't inferred). I've manually added the dependency to the test, and when I run
pants dependencies --transitive <my-test.py>
i can see the package listed in the output, but when i run
pants test <my-test.py>
the test fails with a
ModuleNotFoundError: No module named 'ssh.channel'
oddly enough, if i export my pants resolve to a venv and run pytest within it, the same test succeeds. I suspect what's happening here is that another transitive dependency in my closure is exposing a
ssh
package which doesn't include a
ssh.channel
module, and during export of the venv, the correct
ssh
package including
ssh.channel
is overwriting the incorrect
ssh
package. I also suspect that the reason
ssh-python
isn't being inferred is because pants is identifying this other theoretical
ssh
-exposing package as a dependency instead of
ssh-python
My question is: how to i get pants to tell me which python import symbols map to which dependencies/targets?
t
Do you have a first-party package named
ssh
, or is it another inferred third-party dependency?
s
i don't have a first-party package named ssh. I thought it might be another 3rd-party dependency. I tried to sole the mystery by writing a new python file whose only content is
import ssh
and run
pants dependencies --transitive <my-file.py>
but this has led to an even bigger mystery: the output is blank! according to pants, this file which imports
ssh
has no dependencies!?
ok, nvm, this new testfile wasn't owned by any
python_source
target. once i added a target for it and re-ran the
pants dependencies
command, i got
Copy code
12:49:47.13 [WARN] Pants cannot infer owners for the following imports in the target src:abc:

  * ssh (line: 1)

If you do not expect an import to be inferable, add `# pants: no-infer-dep` to the import line. Otherwise, see <https://www.pantsbuild.org/2.29/docs/using-pants/troubleshooting-common-issues#import-errors-and-missing-dependencies> for common problems.
so i'm back to square one. I have no idea how to troubleshoot this issue
hmmm.... so i added
ssh-python: ['ssh']
to the module_mappings of my 3rdparty requirements target, and the ssh inference error from before is now gone. I can run my temp python file and it's able to import
ssh.channel
from the
ssh-python
package, but my regression test from above is still failing. i don't understand what's going on here. Can anyone give me any ideas for how to troubleshoot this?
t
Do you have an example repo?
b
I suspect what's happening here is that another transitive dependency in my closure is exposing a
ssh
package which doesn't include a
ssh.channel
module, and during export of the venv, the correct
ssh
package including
ssh.channel
is overwriting the incorrect
ssh
package.
Why surmise? How about just https://www.pantsbuild.org/2.29/docs/using-pants/troubleshooting-common-issues#debug-tip-inspect-the-sandbox-with---keep-sandboxes and actually look in the venv and see what's in there?
s
Thanks @brief-scientist-13682! i forgot about that option! I've got the sandbox, but I'm not really sure what I'm looking at here... I think i've figured out where the site packages get deployed during a run, but this site-packages folder matches the exported venv. if I run the
__run.sh
script, i get the
ssh.channel
import error, but if i directly run
<sandbox dir>/.cache/pex_root/venvs/1/<hash>/<other hash>/bin/python
, I'm able to import
ssh.channel
without error...
@thankful-stone-5860 sorry, i should have led with that. my pants monorepo can be found here: https://github.com/uoft-networking/tools/tree/pants and the test that's failing is
pants test src/uoft/core/tests/regression_tests.py
ok, progress! thanks to the documentation, I figured out how to trigger pdb on test failure and replicate the error. @thankful-stone-5860 you asked earlier if i have a package named
ssh
in my repo. I don't, but i do have a
src
root containing a pep420-namespace
uoft
with a
uoft.ssh
package in it. while in pdb, if i import and look at
sys.path
, the very first entry is
'/tmp/pants-sandbox-CMZ3Nd/src/uoft'
So i can now see exactly what the problem is, the pytest execution is seeing my
uoft.ssh
package as a
ssh
package and importing that. What i can't figure out is what the heck that path is doing there... it's not a source root, and i have no code in my codebase doing any kind of silly
sys.path
manipulation...
b
@stale-twilight-79248 well trod ground. You don't really understand your own pytest usage fully: https://pantsbuild.slack.com/archives/C046T6T9U/p1764445356139009?thread_ts=1764196876.676159&cid=C046T6T9U
Copy code
:; pants test src/uoft/core/tests/regression_tests.py
...
=========================== short test summary info ============================
FAILED src/uoft/core/tests/regression_tests.py::test_module_imports - ModuleNotFoundError: No module named 'ssh.channel'
======================== 1 failed, 2 warnings in 0.67s =========================
So, just
touch src/uoft/__init__.py
and:
Copy code
:; pants test src/uoft/core/tests/regression_tests.py
...
=========================== short test summary info ============================
FAILED src/uoft/core/tests/regression_tests.py::test_module_imports - ModuleNotFoundError: No module named 'uoft.scripts.stg_ipam_dev.__main__'
========================= 1 failed, 1 warning in 0.75s =========================
And, FWICT that is a legitimate error pointing out a stale test.
s
oh for crying out loud! Thank you so much @brief-scientist-13682, you know, the sad part is, i should have known about this. i did attempt to arrange my monorepo as a set of pep420-namespaced packages once, years ago, and i remember now encountering this very problem and fixing it with a pytest.ini config (which is long since gone), but i had forgotten all about it
thanks again!