# The max pip verbosity is -vvv and for pex it's -vvvvvvvvv

excellent 😄

Pex 2.0.0 is released: + https://pypi.org/project/pex/2.0.0/ + https://github.com/pantsbuild/pex/releases/tag/v2.0.0 Posted in #releases

if my command is

pex . click requests s3fs pandas pyarrow boto numpy -e -o dist/wm.pex --disable-cache --python=python3.6

, it should use python3.6 to build the pex and thus try to get all the libraries that are compatible with 3.6, right?

ah looks like 2.0 introduced some change in behavior from my code

Pex 2.0.1 is released: + https://pypi.org/project/pex/2.0.1/ + https://github.com/pantsbuild/pex/releases/tag/v2.0.1 NB: This is a hotfix release for a CLI breakage when using `-i`/`--index`/`--index-url`. There will be another less-hotfix release tomorrow of 2.0.2 with an API level fix for

'current'

platform handling and a runtime fix for current interpreter selection biasing. Posted in #releases

Pex 2.0.2 is released: + https://pypi.org/project/pex/2.0.2/ + https://github.com/pantsbuild/pex/releases/tag/v2.0.2 I'll switch back to an upgrade for Pants which revealed the bugs fixed in 2.0.2 / requires the fixes. Posted in #releases

In the vein of https://github.com/pantsbuild/pex/issues/789 (discussing solutions to ship around pex files without all of their requirements stuffed inside) => I've created a doc discussing the alternative solutions we've come up with, which also splits up the problem of "dehydrated" pex files into: 1. How to create "dehydrated" pex files (without requirements)? 2. How to re-hydrate those requirements to run the pex (there are many solutions to this part)? You should have comment access (https://docs.google.com/document/d/1B_g0Ofs8aQsJtrePPR1PCtSAKgBG1o59AhS_NwfFnbI/edit?usp=sharing) -- please let me know if I can clarify the problem or the solutions. This comes from a workstream at Twitter where we're trying to reduce redeploy times for pex files by not shipping around >1GB files with all the 3rdparty requirements attached, mainly for large ML applications. I think this approach is somewhat novel, but I'd love to know if we're missing something obvious.

@enough-analyst-54434 @happy-kitchen-89482 @hundreds-father-404 @rough-minister-58256: @average-vr-56795 has proposed a really interesting way to decouple the "ipex" or "dehydrated pex" from the rest of the pex architecture, by introducing a customizable bootstrap hook: see https://docs.google.com/document/d/1B_g0Ofs8aQsJtrePPR1PCtSAKgBG1o59AhS_NwfFnbI/edit?disco=AAAADyHorIY in the above doc. I'd love to get your thoughts on whether that seems like an effective way to decouple this specialized use case (building/uploading super massive pex files) from the rest of the pex architecture. I was really excited when I saw this comment today, and would love to help develop the idea further if it seems promising. A full review of the doc isn't needed right now if you don't have bandwidth -- just a description of the concerns we should address in architecting this proposal -- we can try to fill in the blanks. cc @witty-crayon-22786 @early-needle-54791

What are solutions people use to distribute their

pex

files? Other than bundling

.pex

files in OS specific packaging solutions (e.g. Homebrew, debs, RPMs, etc) is there any existing tooling for deploying a

pex

file onto a machine? I'm thinking something like

pip

or

maven

.

@enough-analyst-54434 Looking at https://github.com/jsirois/pex/pull/1 - I’m reading that as meaning we’d effectively deploy two pexes, right? One which is a pex of

ptex.py

which has

pex

installed as a requirement, and a second which is the

.ipex

file which will actually be run, right?

So a

.ipex

itself isn’t actually runnable without the

ptex

file running (at some point) first?

For the sketch, yes, but you're reading a bit to literally. You could invert things and have ptex copy itself to temp, reexec there and write over it's old location.

And no, you don't deploy two pexes in the sketch, you deploy ptex and that's all.

It happens to create an ipex and delegate to that, but as an invoker of the ptex, you are unaware.

In the sketch I have ptex kick out the ipex as a sibling on 1st run, but it could stuff it in the cache instead to hide it away / leverage a known writable location.

The thing I think I’m missing is where the source files in the ipex actually exist? From the sketch, only requirements are actually hydrated, and the

ipex

itself only actually contains a bootstrap script, PEX-INFO, and metadata around requirements? Or am I reading it wrong?

Would they just be zip entries in the tpex which we’d copy over when we generate the ipex?

I mean... How detailed a flesh out did I need to do? :) I have meetings here for a few hours but can add that to the example too. I'll make the thing slightly more pointed. If it's not clear, there is an external tool already (Pants) with the resolver configuration data, sources and resources data, etc...

I trust that there are many ways a tool could do this, I’m just trying to get a clear picture of how you think it would 🙂 One of the benefits of the approach of “we just add a configurable bootstrap script to the pex file” or similar is that we don’t need to re-design things like “how we actually generate the pex file from a list of source files and requirements”, and I don’t know off the top of my head how many other configuration knobs there are on pex files which we may also need to re-implement the configuration of. My instinct is that we’d just add the entire pex file as a zip entry into the tpex (zip file in a zip file - fun), which avoids needing to do that re-generation at runtime, but maybe you have cleaner ideas?

Yes, cleaner. Same exact mechanism as wheel cache, but with the already existing code hash.

At the end of all this it would be great if Twitter could devote long term resources to PEX. Lots of changes spanning back to 2015 have spoken of minimal knowledge. It would be good to get a deeper bench for cleaner evolution.

I agree that it would be great for Twitter to have a lot more context/experience here 🙂 I’m trying to build up some of that, but not knowing how the code hash works, can you expand a little?

will try to answer the above^ after reviewing the PR further this afternoon, and will ask john to check my work

I just pushed and the example now has user code. If you run it locally with

tox -epy37 -- -vvsk test_issues_789_demo

the presence of user code is made obvious by colors for isort: https://github.com/jsirois/pex/pull/1/files#diff-b5b57c9d4fc663f59ecaedb851c13bd8

https://github.com/pantsbuild/pex/issues/820 <= i found this issue to be super succinctly and helpfully described!

> ...the realization that Pex currently has two distinct caches for wheels. One for the buildtime resolve of the wheel and one for the runtime extraction of the wheel from zipped pex files. By unifying these caches, we save disk space on machines where we both build and run wheels. this was the kind of analysis and re-understanding of the problem we're trying to solve i was hoping to get from your involvement! thank you again! i'm now at least 1 full file into the PR, spurred by your recent push!

i also saw the PR to parallelize resolve and haven't yet reviewed but i was super glad because i'd seen parallelizing resolve induce a wonderful speedup when i was investigating my initial mad dash to make running pex truly instantaneous (which involved fingerprinting "components", aka informally-defined cacheable/parallelizable equivalents to pants targets). https://github.com/pantsbuild/pex/compare/master...cosmicexplorer:incremental-pex-production-with-fingerprinting?expand=1 actually has a much more useful readme at the top than i remember writing! it parallelizes adding source files to the pex in a way that i'm sure has none of the guarantees described in https://github.com/pantsbuild/pex/pull/819.

a somewhat-related note that may describe (i just realized) why i hopped on this during hackweek is that i am extremely mad about the company that's making a proprietary highly parallel scala compiler they're calling hydra, and i am trying to replicate their result with open source code, and i had just spent far too many evenings attempting to productionize an alternative experiment on top of https://github.com/twitter/rsc which would handily beat theirs (now made much easier by being able to lean on pants v2 the way i always wanted to)

so basically i have bottomless pools of activist fury to devote to the problem

If there is anyone willing to take a look at https://github.com/pantsbuild/pex/pull/819 I'd be grateful for more active reviewer eyeballs. Namely the producer/consumer concurrency limiter implemented here: https://github.com/pantsbuild/pex/blob/0fef27e144894aac91d4306bd4b51a929be07ea7/pex/jobs.py#L184-L243 If that bit of code is broken, you'd get inifinte hang and without that code you get ... nearly infinite hangs while your machine attempts to execute hundreds of concurrent wheel installs.