https://pantsbuild.org/ logo
#general
Title
# general
t

thousands-plumber-33255

12/19/2022, 9:56 PM
Hello all! I am trying to use the new environments feature to run a test in a docker container. The problem is that my function code is relying on a python module that is not part of the pants context, i.e. not added as a third party package that is inferred by pants, but is only part of that docker image. If I run the test it fails due to an import error. I guess that pants is not looking for over python modules in that environment but runs in its sandbox. How can I incorporate those modules during the test run?
@witty-crayon-22786 Any idea?
To be a bit more specific, my code is using the Python API from QGIS which can only be installed from the debian package. I thought, why not just run the tests in the https://hub.docker.com/r/qgis/qgis image then. But as pants runs the tests in a sandboxed environment and did not infer the dependencies (as they are only available in the target docker environment). Hence, I am getting an import error. What is the best way to handle such case? Install the debian package where pants runs and add its location as a source root? Or can I tell pants to not run in a sandbox in the docker environment? When I package the application code as PEX and add it to the docker image I am not facing any import issues. Ideas @happy-kitchen-89482 @curved-television-6568?
c

curved-television-6568

12/21/2022, 3:09 PM
I’m not sure I follow exactly.. adding an explicit dependency would that work? Also, the “works as PEX” have me slightly confused 😉 (haven’t read up on the full thread here though)
h

happy-kitchen-89482

12/21/2022, 3:51 PM
@witty-crayon-22786 is out on vacation, so I'd suggest pinging him again after January 3, if this isn't resolved by then.
Possibly @hundreds-father-404 will have ideas here?
But normally pants/pex goes to great lengths to isolate you from random things that happen to be installed in the ambient python environment, to ensure repeatability
Pex has ways to let you poke holes in that. I'm not sure if those are wired through to Pants.
But I feel like this has come up before, so maybe there is a way
When I package the application code as PEX and add it to the docker image I am not facing any import issues.
that is interesting! I would have expected the same issue.
When you build that pex are you specifying qgis as a requirement?
t

thousands-plumber-33255

12/21/2022, 7:41 PM
I think I can answer both your (similiar) question: No, I have not added qgis as an explicit dependency/requirement. The only thing I have is that my code imports the qgis module, Pants complains about that it cannot infer this but when I build that Docker image with the PEX it does not complain about the import. I would really appreciate help here, because I cannot think of a very straight forward Pants way here. I would prefer adding it as an explicit dependency, but would also be okay to run the code with pants in the docker environment.
Would an option be installing it with
apt-get install python3-qgis
, look for where it is installed and then add it as a local requirement?
It will install it to
/usr/lib/python3/dist-packages/qgis
, can I add this as a dependency?
c

curved-television-6568

12/21/2022, 8:53 PM
I don’t think you can.. as dependencies only work for targets in your project, so has to live within that. Oh, unless there’s some way to setup a python_requirement target along with some find links to a whl file.. I’m just fishing I have no idea, sorry.
h

happy-kitchen-89482

12/21/2022, 9:12 PM
I think there should be a way to pass a relevant flag to Pex? Not sure
t

thousands-plumber-33255

12/21/2022, 10:15 PM
I was looking for a whl file, but didn't find any.
Could not find any flag that is related to building the PEX file, but one could try this env in the docker environment https://pex.readthedocs.io/en/v2.1.119/api/vars.html#PEX_INHERIT_PATH
Another option might be to add /usr/lib/python3/dist-packages/qgis to the source roots, but they can only be in the pants root dir
For testing purposes, I have just copied the two relevant modules to a new folder
pyqgis
in the root of my repo, added it to
root_patterns
and run
./pants tailor
on it. Still, my tests throws an import error:
Copy code
__ ERROR collecting services/qgis/tests/test_add_spatial_analysis_project.py ___
ImportError while importing test module '/tmp/pants-sandbox-W4B4LD/services/qgis/tests/test_add_spatial_analysis_project.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
services/qgis/tests/test_add_spatial_analysis_project.py:20: in <module>
    from api import add_spatial_analysis_project
services/qgis/api/add_spatial_analysis_project.py:12: in <module>
    from qgis.core import QgsApplication, QgsProject, QgsVectorLayer, QgsCoordinateReferenceSystem, QgsRasterLayer, QgsPalLayerSettings, QgsTextFormat, QgsVectorLayerSimpleLabeling, QgsLayerTreeGroup
pyqgis/qgis/__init__.py:78: in <module>
    from qgis.PyQt import QtCore
pyqgis/qgis/PyQt/QtCore.py:24: in <module>
    from PyQt5.QtCore import *
E   ModuleNotFoundError: No module named 'PyQt5.QtCore'
When I check the sandbox, it should be there:
Copy code
vscode ➜ /tmp/pants-sandbox-W4B4LD $ ls -la pyqgis/PyQt5/
total 1536
drwxr-xr-x 3 vscode vscode   4096 Dec 21 22:56 .
drwxr-xr-x 4 vscode vscode   4096 Dec 21 22:56 ..
-rw-r--r-- 1 vscode vscode 156355 Dec 21 22:56 Qsci.pyi
-rw-r--r-- 1 vscode vscode 407093 Dec 21 22:56 QtCore.pyi
-rw-r--r-- 1 vscode vscode 422646 Dec 21 22:56 QtGui.pyi
-rw-r--r-- 1 vscode vscode  20230 Dec 21 22:56 QtPrintSupport.pyi
-rw-r--r-- 1 vscode vscode 509967 Dec 21 22:56 QtWidgets.pyi
-rw-r--r-- 1 vscode vscode  27660 Dec 21 22:56 QtXml.pyi
-rw-r--r-- 1 vscode vscode    960 Dec 21 22:56 __init__.py
drwxr-xr-x 2 vscode vscode   4096 Dec 21 22:56 __pycache__
What is missing here?
With
Copy code
python_tests(
    environment="linux_docker",
    extra_env_vars=["PEX_INHERIT_PATH=prefer"]
)
I am getting:
Copy code
__ ERROR collecting services/qgis/tests/test_add_spatial_analysis_project.py ___
ImportError while importing test module '/pants-sandbox/pants-sandbox-5JV0Aj/services/qgis/tests/test_add_spatial_analysis_project.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
services/qgis/tests/test_add_spatial_analysis_project.py:20: in <module>
    from api import add_spatial_analysis_project
services/qgis/api/add_spatial_analysis_project.py:12: in <module>
    from qgis.core import QgsApplication, QgsProject, QgsVectorLayer, QgsCoordinateReferenceSystem, QgsRasterLayer, QgsPalLayerSettings, QgsTextFormat, QgsVectorLayerSimpleLabeling, QgsLayerTreeGroup
E   ModuleNotFoundError: No module named 'qgis'
=============================== warnings summary ===============================
services/qgis/api/add_spatial_analysis_project.py:65
  /pants-sandbox/pants-sandbox-5JV0Aj/services/qgis/api/add_spatial_analysis_project.py:65: DeprecationWarning: invalid escape sequence \d
    regex_pattern = f'qgis/({AWS_OBJECT_PREFIX} analysis [-\d\w]+\.zip)'

services/qgis/api/add_spatial_analysis_project.py:79
  /pants-sandbox/pants-sandbox-5JV0Aj/services/qgis/api/add_spatial_analysis_project.py:79: DeprecationWarning: invalid escape sequence \d
    regex_pattern = f'qgis/{AWS_OBJECT_PREFIX} analysis ([-\d\w]+)\.zip'

-- Docs: <https://docs.pytest.org/en/stable/how-to/capture-warnings.html>
- generated xml file: /pants-sandbox/pants-sandbox-5JV0Aj/services.qgis.tests.test_add_spatial_analysis_project.py.xml -
=========================== short test summary info ============================
ERROR services/qgis/tests/test_add_spatial_analysis_project.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
========================= 2 warnings, 1 error in 0.27s =========================

Ignoring the following environment variables in Pex venv mode:
PEX_INHERIT_PATH=prefer



✕ services/qgis/tests/test_add_spatial_analysis_project.py failed in 0.72s.
Looks like this env is just getting ignored
@bitter-ability-32190 Any idea? You seem familiar with PEX.
b

bitter-ability-32190

12/31/2022, 1:10 PM
I think your breadcrumb is in the error... "Ignoring the following environment variables in Pex venv mode"
t

thousands-plumber-33255

01/02/2023, 8:02 AM
Yes, as @happy-kitchen-89482 suggested I was looking for al flag to pass to PEX which I did with
PEX_INHERIT_PATH=prefer
. But it did not take any effect due to that error. Can this be solved somehow? I could not find anything to not run it in the venv mode.
h

happy-kitchen-89482

01/02/2023, 8:05 AM
How are you building the pex?
t

thousands-plumber-33255

01/02/2023, 8:07 AM
Just running
./pants test services/qgis/tests/test_add_spatial_analysis_project.py
Overall, I am wondering what is the best Pants solution to address my use case. I was hoping that specifying the docker environment for my test would address it. Is that an intended use case? Another option, which I am not familiar with, seems to build some artifact (the QGIS dependencies) within the docker container and pass that to the test with
runtime_package_dependencies
. Is that something that makes sense?
b

bitter-ability-32190

01/02/2023, 12:20 PM
At the end of the day Pants focuses on hermeticity, and so ignoring the hosts environment is paramount (as much as it can). Now that we can also run in a predefined container, one could argue that could be relaxed. But the truth is it hasn't yet, and may not be, so it's a bumpy road
t

thousands-plumber-33255

01/02/2023, 1:43 PM
That makes sense. To conclude for my specific use case (consider a python package that is installed by a debian package), this is currently not possible with the environment feature?
b

bitter-ability-32190

01/02/2023, 1:59 PM
Not for tests it seems. It's honestly more of happenstance though, than a hard stance. We're using a venv Pex because we can tack on a speed hack for instant startup on Nth runs. But then the venv Pex is ignoring the setting. Since we're executing in a docker image, there will never be an Nth run so you could argue we shouldn't do this. In which case your setting would likely work.
I'll defer to others to make the call on whether we should make the change and enshrine this behavior in a test, or if we want to continue in a maximum-hermecity way
t

thousands-plumber-33255

01/02/2023, 2:02 PM
Ok thanks, would be great to get an update once decided :)
b

bitter-ability-32190

01/02/2023, 2:06 PM
Honestly, if I was you I'd just play with the import machinery so that I could import the module with a semi-hardcoded path (Im guessing the module is in the system pythons
dist-packages
)
t

thousands-plumber-33255

01/02/2023, 2:12 PM
Do you mean the python import machinery or Pants? Something like
Copy code
sys.path.append("/usr/lib/python3/dist-packages/qgis")
?
b

bitter-ability-32190

01/02/2023, 3:29 PM
That, or something similar
👀 1