Is it possible to declare 3rd party dependencies i...
# general
n
Is it possible to declare 3rd party dependencies in a way analogous to sitecustomize.py? We have several internal lib dependencies not on our PyPI proxy, and we would prefer not to have to explicitly bring them into the repo as that would cause a lot of bloat and be tedious to maintain. In our current setup we make these packages importable in our main virtual environment (and known/indexable by our IDE) by updating sitecustomize.py with a sys.path.append("resolved/path/to/dependency") statement (we have a library that does the abs path resolution of various AFS release links). Is there a canonical way to do that in Pants such that A) Pants will infer the dependency in any build sources B) package the dependency as it does a PyPI requirement? Even if we placed the dependency in ./3rdparty, what distribution format does Pants expect in order to package it or will it just copy whatever you have in the top-level directory (i.e., assume it is already "installed")?
e
Not today. I couldn't find an existing issue, but we have had discussions about this idea in general. In the JVM world these would be dependencies with "provided" scope; I.E.: I depend on this thing, but don't fetch it, it will be provided by a prepared environment configured out of band.
Our machinery for Python supports the runtime side of this (PEX_EXTRA_SYS_PATH) but there would be other work needed to fully support dependencies of this sort. If you don't mind filing a feature request issue, that would be welcome. I think this may be a more common use case than we imagined.
n
OK sounds good. I just tried using a symlink in 3rdparty to the AFS dependency. Works fine importing venv/IDE is ok with it, but when I add "3rdparty/" to sources and run
./pants roots
Pants emits an error Failed to read link "path/to/link." So it seems the only option at the moment is to copy the entire binary into source control?
e
There is a way to refer to local wheels or sdists outside the repo and have Pants work with those. There is not a way to refer to local sys.path entries (directories or zips) outside the repo though,
It sounds like you have the latter, which is currently a no go. If you also have the former I can explain more on how to set that up.
n
The former should be ok too, because where we add to sys.path in sitecustomize.py we would do whatever is required in this case (in other words, we know where those artifacts are in AFS and can do whatever Pants needs to integrate them as project dependencies and make sure they are packaged in pex binary/test and other builds). Actually, the artifacts are already in "installed" format (i.e., what you would see in your virtual environment after python setup.py install), but inverting that into a bdist_wheel or sdist is not a problem.
e
Ok. If you can arrange all needed sdists and wheels to be in a flat directory mounted at a fixed path, you can add that path to `[python-repos] repos`: https://www.pantsbuild.org/docs/reference-python-repos
You can either use a file: URL or a bare absolute path in that list.
n
ah ok, i think this is what we wanted!
so in our workflow that adds an AFS mounted package not on our Artifactory, in addition to putting a sys.path entry in sitecustomize.py, we can also put an entry in pants.toml like so ^