Sorry for the spamming I have a more complex situation The p

Question

Sorry for the spamming I have a more complex situation The python project I work on has conda dependencies I tried to create a fresh conda env install the binary non python dependencies and then run `pants tests` Unfortunately these binaries are not detected Any idea of how to solve this issue

enough-analyst-54434 · Answer

Pants strives for reproducibility To achieve this it isolates build steps from your local environment as much as possible For Python this means Pants resolves your dependencies via Pip using Pex Any dependencies you install by hand in your local environment are purposefully ignored

enough-analyst-54434 · Answer

So this could tread on new ground for Pants

enough-analyst-54434 · Answer

We don t currently have a Conda integration official support

hundreds-father-404 · Answer

Where are those non Python binaries located For example are they on your `$PATH`

enough-analyst-54434 · Answer

Keep in mind beyond the Conda Python binary which is compatible it has its own `conda install` that is most definitely = `pip install` doesn t use PyPI

hundreds-father-404 · Answer

Iiuc tho they re talking about non Python deps here Conda is acting similar to Homebrew apt get etc

enough-analyst-54434 · Answer

The wording is certainly confusing since the sentence before that says The python project < freezing photographer 88553> we ll probably need a good deal more detail By non python deps do you mean say ruby deps or python native code deps

enough-analyst-54434 · Answer

Or shared library deps i e `LD LIBRARY PATH` territory

freezing-photographer-88553 · Answer

The binaries are located in the conda environment directory ``` anaconda3 envs PantsBuild lib libspatialindex c dylib```

freezing-photographer-88553 · Answer

I have a python mono repo with 2 projects and 5 libraries These 2 projects depends on 3 libraries each These projects have different python dependencies and binary dependencies even python versions I need to create different virtual env for each project and libraries for testing Since the `pex`` environments does not support non python package dependencies I have to find another tool to do it as an example The project depends on a python pkg called `geopandas` This pkg makes calls to `libspatialindex` which is a C++ library To setup the project with the geopandas dependency I have 2 options Option 1 ```apt get install gdal libspatialindex rtree pip install geopandas ``` Option 2 ```conda install geopandas rtree c conda forge``` The second option is way better because 1 these libs are installed in a virtual env 2 the versions are resolved across python and C++ libraries 3 it is platform agnostic same command for MacOs and Linux

freezing-photographer-88553 · Answer

However when using conda tests run from pants are not detecting these C++ libraries

enough-analyst-54434 · Answer

Yeah they shouldn t be per the hermetic build goal of Pants This will require feature work from what I can tell to support I can t think of useful hacks that won t burn you in other ways

freezing-photographer-88553 · Answer

Ho I see But then how do people make the non python libraries available

freezing-photographer-88553 · Answer

I would expect that this is a common use case

freezing-photographer-88553 · Answer

Is it possible to add a command to the script that runs the test

hundreds-father-404 · Answer

gt Is it possible to add a command to the script that runs the test We do have a plugin hook to set up tests before Pytest executes like starting databases But I m not sure that would help 1 It wouldn t help with other goals like `package` and `run` which will still be missing the deps 2 I m not confident we could get that plugin hook to wire things up properly into the test runner gt Option 1 apt get install gdal libspatialindex rtree < enough analyst 54434> any reason you think this would not work here That s how users normally get native things like databases to build make sure you have the correct non Python deps installed and exposed to Pants FYI < freezing photographer 88553> when Pex installs something it is using pip So if you can get `pip install` to work it should be possible for Pants to install things Ack that you prefer to keep using conda

enough-analyst-54434 · Answer

Generally you d make native libraries available by either enforcing uniform images or else using something like LD LIBRARY PATH and exposing it via ` subprocess environment env vars add = LD LIBRARY PATH ` See <https www pantsbuild org docs reference subprocess environment section env vars>

enough-analyst-54434 · Answer

< hundreds father 404> the issue here is Conda is hermetic It installs things in non standard locations Conda environments and exposes these things to user applications

enough-analyst-54434 · Answer

< freezing photographer 88553> you might be able to hack this together using `LD LIBRARY PATH` but I expect this will be tricky and brittle

hundreds-father-404 · Answer

Ack I m suggesting that Thomas not use Conda and use option 1 of `apt get` They prefer option 2 working but if we can t get it to then at least we can do option 1

enough-analyst-54434 · Answer

Ah well people who use Conda use it for reasons I was guessing Just don t use Conda is not a valid answer

freezing-photographer-88553 · Answer

I am not able to use option 1 because different project have incompatible non python libraries

freezing-photographer-88553 · Answer

So I have to use conda slightly smiling face

enough-analyst-54434 · Answer

Yeah

freezing-photographer-88553 · Answer

These libraries cannot be installed by pip

enough-analyst-54434 · Answer

So this will require feature work < freezing photographer 88553> You re the 1st conda user we ve knowingly hit to my knowledge

freezing-photographer-88553 · Answer

Ha

freezing-photographer-88553 · Answer

This is a struggle accross all platforms Peoetry pip and pants don t support it Conda does but it is not a tool for managing a mono repo

enough-analyst-54434 · Answer

We did have lots of discussions about this walking back and forth to lunch though in 2018 We knew we d hit the issue eventually

freezing-photographer-88553 · Answer

I am the issue

freezing-photographer-88553 · Answer

I see

enough-analyst-54434 · Answer

So to be blunt we re Conda ignorant It would be great to have a knowledgeable Conda user help get this ball rolling but that s a big task to take on too

freezing-photographer-88553 · Answer

I am happy to have a call but I cannot provide much more I have limited time constraints And unfortunately I might have to drop the migration of my mono repo to Pants If I don t find a solution

enough-analyst-54434 · Answer

Yeah this may be bad timing unfortunately

freezing-photographer-88553 · Answer

We can organise a call this week for a debrief on the use case

freezing-photographer-88553 · Answer

I expect this is a usual use case

enough-analyst-54434 · Answer

I expect so too < happy kitchen 89482> or < busy vase 39202> may be interested in pursuing this with you

happy-kitchen-89482 · Answer

Thanks for the detailed info < freezing photographer 88553> It would be really helpful if you could open a feature request with as much detail as possible about the problem and what your ideal solution would look like Then we can ask others who may have conda needs to chime in Thanks

busy-vase-39202 · Answer

By the way < freezing photographer 88553> asking a lot of questions here is totally welcome No apologies needed We re glad the community can be helpful for you plus we glean valued insights from users questions and we appreciate how new tickets help the project to continue improving