@witty-crayon-22786 Any idea?

To be a bit more specific, my code is using the Python API from QGIS which can only be installed from the debian package. I thought, why not just run the tests in the https://hub.docker.com/r/qgis/qgis image then. But as pants runs the tests in a sandboxed environment and did not infer the dependencies (as they are only available in the target docker environment). Hence, I am getting an import error. What is the best way to handle such case? Install the debian package where pants runs and add its location as a source root? Or can I tell pants to not run in a sandbox in the docker environment? When I package the application code as PEX and add it to the docker image I am not facing any import issues. Ideas @happy-kitchen-89482 @curved-television-6568?

Possibly @hundreds-father-404 will have ideas here?

But normally pants/pex goes to great lengths to isolate you from random things that happen to be installed in the ambient python environment, to ensure repeatability

Pex has ways to let you poke holes in that. I'm not sure if those are wired through to Pants.

But I feel like this has come up before, so maybe there is a way

When I package the application code as PEX and add it to the docker image I am not facing any import issues.

that is interesting! I would have expected the same issue.

When you build that pex are you specifying qgis as a requirement?

Would an option be installing it with

apt-get install python3-qgis

, look for where it is installed and then add it as a local requirement?

It will install it to

/usr/lib/python3/dist-packages/qgis

, can I add this as a dependency?

Could not find any flag that is related to building the PEX file, but one could try this env in the docker environment https://pex.readthedocs.io/en/v2.1.119/api/vars.html#PEX_INHERIT_PATH

Another option might be to add /usr/lib/python3/dist-packages/qgis to the source roots, but they can only be in the pants root dir

For testing purposes, I have just copied the two relevant modules to a new folder

pyqgis

in the root of my repo, added it to

root_patterns

and run

./pants tailor

on it. Still, my tests throws an import error:

__ ERROR collecting services/qgis/tests/test_add_spatial_analysis_project.py ___
ImportError while importing test module '/tmp/pants-sandbox-W4B4LD/services/qgis/tests/test_add_spatial_analysis_project.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
services/qgis/tests/test_add_spatial_analysis_project.py:20: in <module>
    from api import add_spatial_analysis_project
services/qgis/api/add_spatial_analysis_project.py:12: in <module>
    from qgis.core import QgsApplication, QgsProject, QgsVectorLayer, QgsCoordinateReferenceSystem, QgsRasterLayer, QgsPalLayerSettings, QgsTextFormat, QgsVectorLayerSimpleLabeling, QgsLayerTreeGroup
pyqgis/qgis/__init__.py:78: in <module>
    from qgis.PyQt import QtCore
pyqgis/qgis/PyQt/QtCore.py:24: in <module>
    from PyQt5.QtCore import *
E   ModuleNotFoundError: No module named 'PyQt5.QtCore'

When I check the sandbox, it should be there:

vscode ➜ /tmp/pants-sandbox-W4B4LD $ ls -la pyqgis/PyQt5/
total 1536
drwxr-xr-x 3 vscode vscode   4096 Dec 21 22:56 .
drwxr-xr-x 4 vscode vscode   4096 Dec 21 22:56 ..
-rw-r--r-- 1 vscode vscode 156355 Dec 21 22:56 Qsci.pyi
-rw-r--r-- 1 vscode vscode 407093 Dec 21 22:56 QtCore.pyi
-rw-r--r-- 1 vscode vscode 422646 Dec 21 22:56 QtGui.pyi
-rw-r--r-- 1 vscode vscode  20230 Dec 21 22:56 QtPrintSupport.pyi
-rw-r--r-- 1 vscode vscode 509967 Dec 21 22:56 QtWidgets.pyi
-rw-r--r-- 1 vscode vscode  27660 Dec 21 22:56 QtXml.pyi
-rw-r--r-- 1 vscode vscode    960 Dec 21 22:56 __init__.py
drwxr-xr-x 2 vscode vscode   4096 Dec 21 22:56 __pycache__

What is missing here?

With

python_tests(
    environment="linux_docker",
    extra_env_vars=["PEX_INHERIT_PATH=prefer"]
)

I am getting:

__ ERROR collecting services/qgis/tests/test_add_spatial_analysis_project.py ___
ImportError while importing test module '/pants-sandbox/pants-sandbox-5JV0Aj/services/qgis/tests/test_add_spatial_analysis_project.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
services/qgis/tests/test_add_spatial_analysis_project.py:20: in <module>
    from api import add_spatial_analysis_project
services/qgis/api/add_spatial_analysis_project.py:12: in <module>
    from qgis.core import QgsApplication, QgsProject, QgsVectorLayer, QgsCoordinateReferenceSystem, QgsRasterLayer, QgsPalLayerSettings, QgsTextFormat, QgsVectorLayerSimpleLabeling, QgsLayerTreeGroup
E   ModuleNotFoundError: No module named 'qgis'
=============================== warnings summary ===============================
services/qgis/api/add_spatial_analysis_project.py:65
  /pants-sandbox/pants-sandbox-5JV0Aj/services/qgis/api/add_spatial_analysis_project.py:65: DeprecationWarning: invalid escape sequence \d
    regex_pattern = f'qgis/({AWS_OBJECT_PREFIX} analysis [-\d\w]+\.zip)'

services/qgis/api/add_spatial_analysis_project.py:79
  /pants-sandbox/pants-sandbox-5JV0Aj/services/qgis/api/add_spatial_analysis_project.py:79: DeprecationWarning: invalid escape sequence \d
    regex_pattern = f'qgis/{AWS_OBJECT_PREFIX} analysis ([-\d\w]+)\.zip'

-- Docs: <https://docs.pytest.org/en/stable/how-to/capture-warnings.html>
- generated xml file: /pants-sandbox/pants-sandbox-5JV0Aj/services.qgis.tests.test_add_spatial_analysis_project.py.xml -
=========================== short test summary info ============================
ERROR services/qgis/tests/test_add_spatial_analysis_project.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
========================= 2 warnings, 1 error in 0.27s =========================

Ignoring the following environment variables in Pex venv mode:
PEX_INHERIT_PATH=prefer



✕ services/qgis/tests/test_add_spatial_analysis_project.py failed in 0.72s.

Looks like this env is just getting ignored

@bitter-ability-32190 Any idea? You seem familiar with PEX.

Overall, I am wondering what is the best Pants solution to address my use case. I was hoping that specifying the docker environment for my test would address it. Is that an intended use case? Another option, which I am not familiar with, seems to build some artifact (the QGIS dependencies) within the docker container and pass that to the test with

runtime_package_dependencies

. Is that something that makes sense?

I'll defer to others to make the call on whether we should make the change and enshrine this behavior in a test, or if we want to continue in a maximum-hermecity way

What is your thought on the alternative? https://pantsbuild.slack.com/archives/C046T6T9U/p1672647000653119?thread_ts=1671486987.564319&cid=C046T6T9U