https://pantsbuild.org/ logo
Join Slack
Powered by
Sorry for the spamming ! I have a more complex sit...
# general
f
Sorry for the spamming ! I have a more complex situation. The python project I work on has conda dependencies. I tried to create a fresh conda env, install the binary (non python) dependencies and then run 
pants tests
. Unfortunately, these binaries are not detected. Any idea of how to solve this issue ?
e
Pants strives for reproducibility. To achieve this, it isolates build steps from your local environment as much as possible. For Python this means Pants resolves your dependencies (via Pip using Pex). Any dependencies you install by hand in your local environment are purposefully ignored.
So this could tread on new ground for Pants.
We don't currently have a Conda integration / official support.
👍 1
h
Where are those non-Python binaries located? For example, are they on your 
$PATH
?
e
Keep in mind, beyond the Conda Python binary (which is compatible) it has its own 
conda install
that is most definitely != 
pip install
/ doesn't use PyPI.
h
Iiuc tho, they're talking about non-Python deps: here, Conda is acting similar to Homebrew, apt-get etc
e
The wording is certainly confusing since the sentence before that says "The python project". @freezing-photographer-88553 we'll probably need a good deal more detail. By non python deps do you mean - say - ruby deps or python native code deps?
👍 1
Or shared library deps? (i.e. 
LD_LIBRARY_PATH
territory)
f
The binaries are located in the conda environment directory:
Copy code
./anaconda3/envs/PantsBuild/lib/libspatialindex_c.dylib
I have a python mono repo with 2 projects and 5 libraries. These 2 projects depends on 3 libraries each. These projects have different python dependencies and binary dependencies, even python versions. I need to create different virtual env for each project and libraries for testing. Since the `pex`` environments does not support non-python package dependencies, I have to find another tool to do it. as an example: The project depends on a python pkg called 
geopandas
 . This pkg makes calls to 
libspatialindex
 which is a C++ library. To setup the project with the geopandas dependency I have 2 options: Option 1:
Copy code
apt-get install gdal libspatialindex rtree ...
pip install geopandas
Option 2:
Copy code
conda install geopandas rtree -c conda-forge
The second option is way better because: 1) these libs are installed in a virtual env 2) the versions are resolved across python and C++ libraries 3) it is platform agnostic (same command for MacOs and Linux)
However, when using conda, tests run from pants are not detecting these C++ libraries.
e
Yeah, they shouldn't be per the hermetic build goal of Pants. This will require feature work from what I can tell to support. I can't think of useful hacks that won't burn you in other ways.
f
Ho I see. But then how do people make the non-python libraries available ?
I would expect that this is a common use case
Is it possible to add a command to the script that runs the test ?
h
Is it possible to add a command to the script that runs the test ?
We do have a plugin hook to set up tests before Pytest executes, like starting databases. But I'm not sure that would help: 1) It wouldn't help with other goals like 
package
and 
run
, which will still be missing the deps 2) I'm not confident we could get that plugin hook to wire things up properly into the test runner
Option 1: apt-get install gdal libspatialindex rtree ...
@enough-analyst-54434 any reason you think this would not work here? That's how users normally get native things like databases to build: make sure you have the correct non-Python deps installed and exposed to Pants FYI @freezing-photographer-88553, when Pex installs something, it is using pip. So if you can get 
pip install
to work, it should be possible for Pants to install things. (Ack that you prefer to keep using conda)
e
Generally you'd make native libraries available by either enforcing uniform images or else using something like LD_LIBRARY_PATH and exposing it via `[subprocess-environment] env_vars.add = ["LD_LIBRARY_PATH"]`(See: https://www.pantsbuild.org/docs/reference-subprocess-environment#section-env-vars).
@hundreds-father-404 the issue here is Conda is hermetic. It installs things in non-standard locations (Conda environments) and exposes these things to user applications.
@freezing-photographer-88553 you might be able to hack this together using 
LD_LIBRARY_PATH
but I expect this will be tricky and brittle.
h
Ack, I'm suggesting that Thomas not use Conda and use option 1 of 
apt-get
. They prefer option 2 working, but if we can't get it to, then at least we can do option 1
e
Ah, well people who use Conda use it for reasons. I was guessing "Just don't use Conda" is not a valid answer.
f
I am not able to use option 1 because different project have incompatible non python libraries.
So I have to use conda 🙂
👍 1
e
Yeah.
f
These libraries cannot be installed by pip
e
So, this will require feature work @freezing-photographer-88553. You're the 1st conda user we've knowingly hit to my knowledge.
f
Ha
This is a struggle accross all platforms. Peoetry, pip and pants don't support it. Conda does but it is not a tool for managing a mono-repo !
e
We did have lots of discussions about this walking back and forth to lunch though in 2018! We knew we'd hit the issue eventually.
👍 1
f
I am the issue
I see
e
So, to be blunt, we're Conda ignorant. It would be great to have a knowledgeable Conda user help get this ball rolling, but that's a big task to take on too.
f
I am happy to have a call but I cannot provide much more. I have limited time constraints. And unfortunately, I might have to drop the migration of my mono repo to Pants If I don't find a solution
e
Yeah, this may be bad timing unfortunately.
f
We can organise a call this week for a debrief on the use case
I expect this is a usual use case.
e
I expect so too. @happy-kitchen-89482 or @busy-vase-39202 may be interested in pursuing this with you.
h
Thanks for the detailed info @freezing-photographer-88553! It would be really helpful if you could open a feature request ticket, with as much detail as possible about the problem, and what your ideal solution would look like. Then we can ask others who may have conda needs to chime in. Thanks!
b
By the way @freezing-photographer-88553, asking a lot of questions here is totally welcome. No apologies needed. We're glad the community can be helpful for you, plus we glean valued insights from users' questions and we appreciate how new tickets help the project to continue improving.
➕ 1
💯 1
2 Views
Open in Slack
PreviousNext
Powered by