Hi all! I am trying to find a pre-compiled `.so` f...
# general
a
Hi all! I am trying to find a pre-compiled
.so
file as a dependency for one of the python_sources. The closest I could fins is https://www.pantsbuild.org/docs/adhoc-tool#using-externally-managed-tools `system_binary`but it seems it will only work with the ad_hook. Is there a way to do that? The binary currently lives in user's `.pyenv`and should be same for everyone. Any hint would be appreciated!
One thing that I have tried is
resources(
name="spylizard",
sources=["spylizard.so"],
)
python_sources(
overrides={
"solver_spylizard.py": {
"dependencies": [
":spylizard",
],
}
}
)
But this does not solve the problem: ModuleNotFoundError: No module named 'spylizard' When the binary is in .pyenv, and script is executed with python, module is imported.
h
What do you mean by "the binary lives in users
.pyenv
? I take that to mean the python distribution that provides that .so (presumably
spylizard
) was installed using
pip
in one of the
pyenv
interpreters on the system?
If the
spylizard
package provides
spylizard.so
as part of its installation, then you want to add
spylizard
as a 3rd-party requirement in your code (e.g., in requirements.txt) and generate it into your lockfile if you're using lockfiles. You can't rely on it existing in an interpreter's site-packages, because Pants (via Pex) takes great pains to scrub those from its working set.
There is work under way to support "provided dependencies" - that exist in the interpreter's site-packages and are not installed by Pants, but this is not supported yet.
a
The binary was actually just build with cmake and copied over to .pyenv
I can't really add it to requirements.txt, since it is not pip installable. I also can't build a wheel for it.
h
So you manually build it and copy it to... where exactly?
why .pyenv?
Interesting, hmm
I checked to see if https://pypi.org/project/spylizard/ does what you want but it appears to be a broken work in progress, so let's ignore that.
One thought is to treat this is another use case of @flat-zoo-31952’s work on provided dependencies.
a
Yeah, to ~/.local/lib/python3.10/site-packages/ . So there isn't a way to make pants understand it at the moment?
h
Unfortunately I don't see a way. Pants is all about hermetic environments, and this violates that. Josh Reed is working on a proper way to poke holes in that hermeticity for similar cases - where the Python interpreter provides some requirements and Pants just has to be told not to worry about that...
I'll think on this one for a bit
The "proper" way would be to run the steps in https://github.com/halbux/sparselizard-users/tree/main/api/python in a series of adhoc_tool or shell_command invocations, but even then I don't see how we could install it into the site-packages afterwards
Without a plugin
this may require a custom plugin
But I will chew on this for a bit, it's an interesting case
a
Ok, thanks! for the 'provided dependencies' feature, when do you think it will be available? Since we are using lots of open-source libraries like that, I am afraid we will run into more problems of the same kind...
f
work under way
This is charitable. I'd say it's more some throwaway branches and vague proposals
I don't really have free time to devote to this kind of work, sadly. There was just an idea for adding a union rule to python dep inference so that we can non-pip types of 3rd party deps
Then you'd have to write a plugin for finding your provided dep (probably would be left as a plugin hook, since this could vary so much depending on your use case)
And then I think newer version of PEX can be told to just ignore some of these deps and run them from the environment
If you're interested in contributing, I could walk you through this in greater detail, I just don't foresee having much time to sit down and code it myself
h
There is work towards this on the pex side of things: https://github.com/pantsbuild/pex/issues/2097
a
Thanks both! I will keep an eye on the PR and new features. For now, I think we will have to manage this without pants :(
f
This is exactly where we're at too 😢 We use RPM-provided dependencies extensively, which is what motivated me to look into this in the first place. But we miss out on Pants' caching, retries, sharding, etc. It's really something I'd like us fix.