What advice would you have for handling dependenci...
# general
f
What advice would you have for handling dependencies that only exist in conda but not pypi?
Context: I’m trying to migrate from (multiple) conda projects to pants. One example would be non python packages like cuda, for which different projects may want different (conflicting) versions? For python dependencies, e.g. torch (in fact different torch versions are the reason for this), the documentation is fairly clear on how to have different resolves. Another example would be actual python packages that are only published on conda (e.g. https://github.com/oxpig/ANARCI)? Have you ever encountered this and do you have any advice?
f
Interesting @few-airline-50721 - I am currently doing the same.
h
Good question. I’m not intimately familiar with conda, would like to hear if any community members have figured this one out.
My instincts tell me that working with conda would require a plugin to be developed
Well, before I go that far, it would be useful to know a little bit about how you use conda today?
For example, a conda environment might roughly correspond to a Pants resolve? Because different environments can have conflicting versions of things?
Is it, roughly, “build an environment, which happens to contain a python interpreter, and then use that interpreter for all the pants stuff”?
that might not be too hard to implement, apart from the issue of PEX scrubbing the sys.path
f
Most projects that we plan to put in our monorepo define their own environment using a conda environment file (somewhat like a requirements file, or yes like a resolve). You can think of it like a more powered virtual env. It’s pretty standard in ml/science, and many packages are far easier to install using conda than otherwise. Notably: pytorch, which you can easily install with cuda. Yes that is originally exactly what I had in mind: can we swap out pex for a conda env
or pass in the right paths to pex
I tried
Copy code
env_vars.add = ["PEX_INHERIT_PATH=prefer", ]
but this doesn’t work when running tests since pex runs in venv mode which doesn’t respect this feature. Before I went too deep this route I wanted to see if there were already known solutions
h
Well, you’d need to have the venv pex be built using the conda interpreter, then it might work
Trying to think how you could force Pants to use a conda interpreter
I think those identify as
CPython
, right?
👍 1
(to use a conda interpreter for your code, not for running itself on, that is entirely orthogonal)
😍 1
I’ll play with it
f
@happy-kitchen-89482 if you can point me in some directions I would also be happy to dive deep and try to figure it out.
h
Thanks! I will start a discussion on github discussions
Feel free to chime in there, as a start?