pantsbuild #pex

Hello everyone. Big fan of pex and pants! Thanks for all your effort, it's been a great tool for us. Is this the right place to ask a question about pex generation during pantsbuild? I've been looking into an issue without success for the last 3 days, hoping somebody with expertise can point me in the right direction.

swift-noon-29276

03/28/2017, 5:09 PM

I'll just ask, and if I'm making a fool of myself, you can tell me to buzz off and/or ignore me 🙂 Our python app has some native dependencies, and when pants builds the pex, it spends a LONG time on a few of these dependencies. I presume it is compiling them. (This is for a linux x86_64 system).

Copy code

pex: Translating /tmp/tmpzjxTaW/lxml-3.7.2.tar.gz into distribution :: Packaging lxml :: Installing /tmp/tmpTZ_Pha
pex: Translating /tmp/tmpzjxTaW/lxml-3.7.2.tar.gz into distribution: 82936.6ms
pex:   Packaging lxml: 82567.2ms
pex:     Installing /tmp/tmpTZ_Pha: 82563.4ms

swift-noon-29276

03/28/2017, 5:10 PM

I presume this is an instance of https://github.com/pantsbuild/pex/issues/314 relating to manylinux vs linux-x86_64, but I can't find figure out how I would implement the solution suggested in that issue during a pants build.

swift-noon-29276

03/28/2017, 5:10 PM

Might anybody be able to point me in the right direction?

ancient-breakfast-45303

03/28/2017, 6:01 PM

so one solution would be to create .whl

ancient-breakfast-45303

03/28/2017, 6:01 PM

and pull that

swift-noon-29276

03/28/2017, 6:47 PM

Ah, create the .whl for the linux-x86_64 platform, and pull that one? That makes sense. Do you happen to know what pants option I would set to tell it to look in a local dir for prebuilt .whls?

rough-minister-58256

03/28/2017, 6:51 PM

—python-repos-repos=”[‘file:///tmp/wheels/’]”

etc

rough-minister-58256

03/28/2017, 6:51 PM

./pants help-advanced python-repos

swift-noon-29276

03/28/2017, 6:59 PM

Awesome, thank you so much!

rough-minister-58256

03/28/2017, 7:09 PM

you can also stash the wheels on an HTTP frontend and point to those with http://

rough-minister-58256

03/28/2017, 7:09 PM

(note: the trailing / in URIs is important in this context)

rough-minister-58256

03/28/2017, 7:10 PM

the http interface is just an apache dir listing (a “find-links” repo) - pants/pex will parse out the html links and resolve against what’s available there.

rough-minister-58256

03/28/2017, 7:11 PM

if you’re in a position to use git-lfs, some ppl also create a .wheelhouse dir in their repo and resolve against that too with file://

swift-noon-29276

03/28/2017, 7:32 PM

I was thinking about possible putting it into an NFS mount too. I presume that would work the same as local.

rough-minister-58256

03/28/2017, 7:46 PM

should, yeah

rough-minister-58256

03/28/2017, 7:46 PM

the benefit of something like git-lfs is that the state of the binaries stays consistent commit over commit, which leads to more reproducible builds.

aloof-painter-75660

03/30/2017, 6:51 PM

I've been trying to find information about accessing non-py files from within a built pex, and haven't been able to find much. So my app has translation files, as well as some template files to be used with Jinja, and if I build with pants the absolute path (e.g.

os.path.join(os.path.abspath(os.path.dirname(__file__)), TEMPLATE_DIR)))

) the path always starts with the

.pex

file as if it were a directory. Resulting in something like:

/app/server.pex/data/something

, but

server.pex

is not a directory and any

open()

call to a file with this directory prefix will cause an IOError.

fast-nail-55400

03/30/2017, 6:53 PM

can you use the

pkg_resources

module?

fast-nail-55400

03/30/2017, 6:53 PM

(assuming these files are sitting with your .py files inside the pex)

aloof-painter-75660

03/30/2017, 6:54 PM

Yes, they are next to my .py files

aloof-painter-75660

03/30/2017, 6:55 PM

I do not have a lot of experience using pkg_resources, but I did use it recently, so I'll check if I can get that working -- I assume that goes through sys.path and uses some python internals to access the files (and the file was imported using zipimport or something similar?)

fast-nail-55400

03/30/2017, 6:58 PM

I used it in the pants bash-completion task to access a template

fast-nail-55400

03/30/2017, 6:58 PM

I will guess that looks inside the pants pex

fast-nail-55400

03/30/2017, 6:58 PM

see

src/python/pants/core_tasks/bash_completion.py