Got a question on using find_links. We are current...
# general
b
Got a question on using find_links. We are currently building a series of binary wheels using
pip install --use-pep517 <some wheel>
. These builds are executed in an x86 docker environment. The wheels after being built are then collected into a directory which we hope to use later on in the build process for our actual python code. Since these wheels are built in a docker environment dynamically, I was wondering if there was a way to add them as resources to a
python_distribution
target. We can't currently publish them to a pypi repo, so having them available via a find_links or other mechanism is the only option we have.
f
Typically in a pants situation you avoid this but not even building intermediate wheels at all and just build a pex with everything you need in it
But if these wheels are built outside of pants but internally to the company then you might need a way to consume them in pants. Can you describe your situation in a bit more detail?
b
Yes. 1. We have a series of open source libraries we depend on that do not have a build for our x86 CentOS7/Python 38 runtime. 2. We would like to do the build using pants of these bundles using the
pip install --use-pep517
and some other flags to make sure they get built properly for our OS/Python distro 3. Once pants is done doing the build we'd like to do a copy of the built binaries and copy them to a location where they can be consumed by the python_distribution. 4. After this is done we'd like to have the .whl files collected and put into a location where they can be used as a python repo (using
find_links
for instance) 5. Using pex distros isn't available as we need these installed as wheel files for execution on a traditional virtualenv deployment. In fact the entire app gets bundled into an rpm and deployed that way. To confound matters further, the system we are targeting is the CentOS7/Python 38/x86 runtime, so doing builds from our MacOS machines isn't an option. Because of this I have set up a
docker_environment
to do the builds. This works with the custom scripts we need to setup and build the wheel files. But the output doesn't appear to be available anywhere in the pants root or sub-folders. Additionally if we wanted to do the distro using setup.py we need to make sure that it's compiled specifically for the x86 CentOS/Python38 combo. Specifically there doesn't appear to be a way to set an
environment
property on the
python_distribution
target.
@flat-zoo-31952 Is there a time where I can walk you through what I've got? I think we have a unique need and this may lead to me being able to contribute back. Perhaps a slack huddle?
f
I'm kinda learning about the
python_distribution
stuff in Pants myself as well as environments, not sure how much I can help in that context. Some thoughts: 1. The output from you
docker_environment
builds should be available somehow. What target are you using that references that
environment
to perform the build if
python_distribution
doesn't support it? 2. For consuming the wheels with Pants from a directory you can look at your options for local requirements 3. You can use
python_distribution
for building your first party code, and declare a requirement on the other wheels you've built. There's something fundamentally non-Pantsy about having that
find_links
directory there as a stateful cache that Pants isn't really managing. So yeah, I think this could be improved. It looks like you need a way to perform delcare your sdist -> wheel transformations as environment-aware Pants targets of some kind, and then be able to consume those as both requirements for your first party code, and ship them as distributions with
package
goals. Is that correct?
Definitely recommend reposting this in #C01CQHVDMMW, because this is probably beyond the scope of what is possible without custom target types.
b
so what you described is pretty much what I'm hoping for.
I'm just in a position where we also deploy using mod_wsgi for the app so I don't know if PEX distros support that.
f
I don't know anything at all about mod_wsgi unfortunately
b
yeah, things are not looking good for the good guys!