https://pantsbuild.org/ logo
Join SlackCommunities
Powered by
<https://github.com/pantsbuild/pex/issues/1357> th...
# general
a
https://github.com/pantsbuild/pex/issues/1357 this is weird and might be a regression? I cannot fathom why pex takes like multiple seconds to build a tiny pex
e
I asked on the issue, but is Pex slower than Pip here? If not, much creativity will be needed to not undo moving from a bespoke resolver to Pip. If so, sounds buggy / fixable.
a
ok it seems that pip is just a tad faster but pex seems wildly inconsistent
e
When you include -o and force PEX to zip up a PEX file it should always lose by that overhead, which can be multisecond for large zips.
But Pip vs Pex aside, I linked a few issues re lockfile support. That seems like one worthwhile reason to dip back into the resolution game. There is no solving in that case which is what I most want to avoid re-implementing.
a
I don't think this is serious, I just had a workflow where I would 
bidst_pex
and then copy the pex over, but then it was getting slow
is there a way to ask pex to never resolve, just collect a bunch of wheels?
e
No. You can resolve a PEX file and then later use --pex-repository to only resolve from that PEX file.
That's very fast.
Pants uses that mode to resolve a "lockfile PEX" for a global set of requirements once and then sub-resolve smaller PEXes very quickly without hitting the network at all.
a
got it, thanks John! this is is all very informative I guess for a speedy building I would need to have something like: 1. map a fully resolved 
requirements.txt
to a pex file of just deps. 2. change the 
bdist_command
to just resolve from that, passing in no 
requirements.txt
3. Combine the above into a single command, skipping step 1 if the file has not changed
I guess if I was dockerizing a build, as well caching the result from 1 would be great
e
Out of curiosity, what does using 
bdist_pex
make easier for you over just using 
pex
directly?
a
old habbits I guess
e
I've never really understood the use case.
a
I guess if I do the 'new thing' and not even have a 
setup.py
at all I should be using just 
pex
directly
I think a long time ago, I used 
bdist_pex
and never ever questioned it until today
probably a good time to stop since I think distuitls plugins are not 'cool' anymore
e
Ok. 
pex project/ ...
Should be equivalent (or 
.
If the project dir is where you're at already).
a
perfect!
e
Aha! Ok. I thought you knew that but had a reason.
I might kill bdist_pex in 3.0 if there are zero use cases it adds.
👍 1
a
you should, I don't think it adds value and I think the other use cases for bdist style stuff like 
bdist_rpm
are going away as well
👍 1
just to follow up here, I think that if you want to get rid of 
bdist_pex
you should remove it from the docs first
so just to discuss some more, after the data I added to the ticket.
using 
--pex-repository
with a pex of the requirements results in:
Copy code
pex.resolver.Untranslatable: Cannot resolve local projects from PEX repositories. Asked to resolve /Users/zmanji/code/git-squash from ./dist/requirements.pex.
using it with 
--requirements-pex ./dist/requirements.pex
seems to work fine however
this command seems to be pretty fast
Copy code
pex . --requirements-pex ./dist/requirements.pex --no-build --wheel --no-transitive --inherit-path=false --script git-squash -o dist/out.pex
is this what you were suggesting?
e
Nope. Like so: Create the repository:
Copy code
$ time pex . -r requirements.txt -c git-squash -o git-squash.pex

real	0m5.389s
user	0m4.663s
sys	0m0.524s
Use it:
Copy code
$ time pex git-squash==0.0.1 -r requirements.txt --pex-repository git-squash.pex -c git-squash -o git-squash.2.pex -c git-squash

real	0m0.339s
user	0m0.302s
sys	0m0.035s
You get the same pex in this case (a full resolve, not a subset):
Copy code
$ ./git-squash.2.pex -h
usage: git-squash.2.pex [-h] branch

positional arguments:
  branch      The upstream branch to squash commits of the current branch on to.

optional arguments:
  -h, --help  show this help message and exit
$ diff <(pex-tools git-squash.pex info -i4) <(pex-tools git-squash.2.pex info -i4)
$
But you could get a subset:
Copy code
$ time pex GitPython -o GitPython.pex

real	0m0.791s
user	0m0.652s
sys	0m0.093s
$ time pex GitPython -o GitPython.2.pex --pex-repository git-squash.pex 

real	0m0.317s
user	0m0.289s
sys	0m0.029s
$ diff <(pex-tools GitPython.pex info -i4) <(pex-tools GitPython.2.pex info -i4)
^jsirois@gill /tmp/git-squash (master *) $ ./GitPython.2.pex 
Python 3.9.5 (default, May 24 2021, 12:50:35) 
[GCC 11.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from git import Repo
>>> repo = Repo()
>>> repo.rev_parse("HEAD")
<git.Commit "505cf06c598409db9d119a521480e43de3fa8908">
>>> 
now exiting InteractiveConsole...
🎉 1
The 
--pex-repository
resolve speed is going to be pretty much constant. PEX file build times will become dominated by zip times.
And if you mess up, Pex will let you know:
Copy code
$ time pex GitPython isort -o GitPython.2.pex --pex-repository git-squash.pex 
A distribution for isort could not be resolved in this environment.

real	0m0.197s
user	0m0.177s
sys	0m0.021s
a
Thanks a lot John!
Open in Slack
PreviousNext
Powered by