https://pantsbuild.org/ logo
Join Slack
Powered by
I'm encountering a weird python module resolution ...
# general
v
I'm encountering a weird python module resolution issue. I have a package called 
A
under the directory 
company_name/lib/A
that depends on a third party package called 
B
and B depends on a package also called 
A
(but not the same as my package under company_name/lib/A). In testing company_name/lib/A, when B tries to import from A, it resolves to my own package. Why would this happen if my package is nested under company_name/lib? I'm using a docker environment. My source root is just '/'. I'd expect `import company_name.lib.A`to resolve to my package and 
import A
to resolve to the third party package.
Could it be that in testing pants copies the package files to a flat directory structure and loses 
company_name.lib
? Shouldn't we keep the directory structure to prevent this situation?
k
Short answer is that you will save yourself alot of time be renaming your local package A. Both for now and for later. Python has to select between which one to use. If you expect 
import company_name.lib import A
to work it might be that your pants.toml is incorrect. The root patterns decide what is threaten as as a first party. It might be that you have 
lib
as a root pattern.
v
yeah, i've renamed to save the trouble but curious to know why it's an issue. My root pattern is 
root_patterns = ['/', '/idl']
so company_name is at the root of the repo
k
Do you have any python_distribution? In you build file?. I'm shooting from the hip here, but if you export environment through pants and have "py_editable_in_resolve" it will install it in your env. But only under this circumstance afaik.
v
Nope, my BUILD file just looks like this:
Copy code
python_sources()

python_tests(
    name="tests",
    environment="python"
)
h
That is odd. What does 
pants roots
show?
(the package structure is not flattened out on disk)
v
Copy code
pants roots
.
idl
h
To debug this you can run with 
--keep-sandboxes=always
, which will print out the paths to all the sandboxes, find the one for the relevant process (pytest?), and look at the directory structure inside it, and at the command line in 
__run.sh
šŸ‘ 1
But this is pretty surprising, you should find 
company_name/lib/A
in there and unless 
company_name/lib
is on the sys.path, there should be no way for 
B
to 
import A
and get that code instead of what was intended
v
Right, that's what I thought. Here's the directory structure
Copy code
tree -L 3   
.
ā”œā”€ā”€ __run.sh
ā”œā”€ā”€ extra-output
ā”œā”€ā”€ idl
ā”‚   ā””ā”€ā”€ jazmo
ā”‚       ā””ā”€ā”€ contracts
ā”œā”€ā”€ jazmo
ā”‚   ā””ā”€ā”€ lib
ā”‚       ā””ā”€ā”€ postgrest
ā”œā”€ā”€ jazmo.lib.postgrest.postgrest_test.py.tests.xml
ā”œā”€ā”€ local_dists.pex
ā”‚   ā”œā”€ā”€ PEX-INFO
ā”‚   ā”œā”€ā”€ __main__.py
ā”‚   ā”œā”€ā”€ __pex__
ā”‚   ā”‚   ā””ā”€ā”€ __init__.py
ā”‚   ā””ā”€ā”€ pex -> __main__.py
ā”œā”€ā”€ pytest.pex
ā”‚   ā”œā”€ā”€ PEX-INFO
ā”‚   ā”œā”€ā”€ __main__.py
ā”‚   ā”œā”€ā”€ __pex__
ā”‚   ā”‚   ā””ā”€ā”€ __init__.py
ā”‚   ā””ā”€ā”€ pex -> __main__.py
ā”œā”€ā”€ pytest_runner.pex
ā”‚   ā”œā”€ā”€ PEX-INFO
ā”‚   ā”œā”€ā”€ __main__.py
ā”‚   ā”œā”€ā”€ __pex__
ā”‚   ā”‚   ā””ā”€ā”€ __init__.py
ā”‚   ā””ā”€ā”€ pex -> __main__.py
ā”œā”€ā”€ pytest_runner.pex_bin_python_shim.sh
ā”œā”€ā”€ pytest_runner.pex_pex_shim.sh
ā””ā”€ā”€ requirements.pex
    ā”œā”€ā”€ PEX-INFO
    ā”œā”€ā”€ __main__.py
    ā”œā”€ā”€ __pex__
    ā”‚   ā””ā”€ā”€ __init__.py
    ā””ā”€ā”€ pex -> __main__.py
and the __run.sh file
Copy code
#!/usr/bin/env bash
# This script replicates the Docker-based process execution performed by Pants.
# It starts a container and executes the process within it.

set -euo pipefail

# Ensure required Docker volume exists
if ! docker volume inspect pants-named-caches-baa3e18b7d02 >/dev/null 2>&1; then
    echo "Creating Docker volume: pants-named-caches-baa3e18b7d02"
    docker volume create pants-named-caches-baa3e18b7d02
fi

# Start the container if it's not already running
CONTAINER_ID=$(docker run -d \
    --init \
    --tty \
    -v /private/var/folders/lf/tr4gz0k514lf165m8gcg73yw0000gn/T:/pants-sandbox \
    -v pants-named-caches-baa3e18b7d02:/pants-named-caches \
    -v /private/var/folders/lf/tr4gz0k514lf165m8gcg73yw0000gn/T/immutable_inputsQA8s2Z:/pants-immutable-inputs \
    $'sha256:8186c9704e598774b42228bd4257330ff02f26b88c068238ab9367f977bb2b80' \
    /bin/sh)

echo "Started container: $CONTAINER_ID"

# Ensure container cleanup on script exit
cleanup() {
    echo "Stopping and removing container: $CONTAINER_ID"
    docker stop "$CONTAINER_ID" >/dev/null 2>&1 || true
    docker rm "$CONTAINER_ID" >/dev/null 2>&1 || true
}
trap cleanup EXIT

# Execute the command in the container
echo "Executing command in container..."
docker exec \
    -w /pants-sandbox/pants-sandbox-u0CCTc \
    -e PEX_EXTRA_SYS_PATH=$'.:idl' \
    "$CONTAINER_ID" \
    ./pytest_runner.pex_pex_shim.sh $'--color=yes' $'--junit-xml=jazmo.lib.postgrest.postgrest_test.py.tests.xml' -o $'junit_family=xunit2' jazmo/lib/postgrest/postgrest_test.py
and I get this error from importing the 
supabase
package
Copy code
==================================== ERRORS ====================================
____________ ERROR collecting jazmo/lib/postgrest/postgrest_test.py ____________
ImportError while importing test module 'jazmo/lib/postgrest/postgrest_test.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.13/importlib/__init__.py:88: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
jazmo/lib/postgrest/postgrest_test.py:17: in <module>
    from supabase import AuthRetryableError, Client, create_client
/pants-named-caches/pex_root/venvs/1/s/89ebac87/venv/lib/python3.13/site-packages/supabase/__init__.py:1: in <module>
    from postgrest import APIError as PostgrestAPIError
E   ImportError: cannot import name 'APIError' from 'postgrest' (jazmo/lib/postgrest/__init__.py)
As you can see, 
site-packages/supabase/__init__.py
is importing 
postgrest
, which is being directed to 
jazmo/lib/postgrest
h
Do you have a repo you can share that reproduces this? Or can you easily make a toy one?
v
i'll try to make a toy one
@happy-kitchen-89482 Here's a minimal repo https://github.com/mfairley/pants-debug
Also, printing sys.path reveals that 
company/lib
is indeed first in the search path. But why should it even be there if it's not a source root?
b
v
Ah ha, that explains it. Pytest prepends the test file directory by default. Should we disable this for pants testing by default?
b
There is a price to pay for things working by default as demonstrated here. It allows you to not know the tools you use.
v
yeah, in this case Pants is meant to handle paths for us so it doesn't seem to make sense for pytest to modify sys.path further
h
Ah, I did not know (or at least did not remember) that pytest did this. Thanks @brief-scientist-13682! @victorious-dress-47449 It looks like you can work around this by creating the missing intermediate 
company/__init__.py
and 
company/lib/__init__.py
, as pytest uses the presence of those to infer the package root.
Copy code
...
āœ• company/lib/postgrest/main_test.py:tests failed in 1.57s.
$ touch company/lib/__init__.py
$ touch company/__init__.py
$ pants test company/lib/postgrest/main_test.py
15:19:58.90 [INFO] Completed: Scheduling: Run Pytest for company/lib/postgrest/main_test.py:tests
15:19:58.90 [INFO] Completed: Run Pytest - company/lib/postgrest/main_test.py:tests - succeeded.

āœ“ company/lib/postgrest/main_test.py:tests succeeded in 2.99s.
waldorf:[/tmp/pants-debug][main]$
šŸ‘ 1
v
Got it, although not really ideal to put these 
__init__.py
files around the place nor have the path be different from the pants source roots
h
Well, with those 
__init__.py
the sys.path would not be different, right?
But yes, not ideal to have to add them
šŸ‘ 1
šŸ‘ 1
So if you find a value for 
--import-mode
that works for you, you can set that in your pants.toml
5 Views
Open in Slack
PreviousNext
Powered by