abundant-autumn-67998
12/03/2022, 5:46 PMPEX_TOOLS=1
? We ship a set of dependencies and sometimes they invoke commands with subprocess
instead of a Python API. So I'm looking for an easy way to ship the console scripts for each distribution as well.enough-analyst-54434
12/03/2022, 5:57 PMPEX
env var), You can set PEX_SCRIPT=foo
in the environment and that alternate console script will be used. There is also the more generic busybox support provided by the conscript library, works with PEX but you can use it generically: https://pypi.org/project/conscript/PEX_SCRIPT
try pex --help-variables
.--venv prepend
pexes. On 1st run the PEX is turned into a noraml venv in the PEX_ROOT. That venv will have all console scripts installed in its bin/
dir and pre-pended to PATH.--venv
overhead and your deploy environments include /bin/sh
, throw in --sh-boot
. That drops ~50ms wasted re-execing into the pre-built venv via Python by doing it via sh
which takes ~1ms.abundant-autumn-67998
12/03/2022, 6:48 PMPEX_SCRIPT
may not work but the other two options look useful, specially the --venv prepend
- we could just run the pex once after deploying and the PATH will provide the scripts. Will try it out later.enough-analyst-54434
12/03/2022, 7:08 PM--pex-tools
and then run PEX_TOOLS=1 the.pex venv [many options] right/here
That will create a venv from the PEX right/here
and there will be a `right/here/pex`script you can use to run the venv just like the originating PEX would have run - but, of course, faster.abundant-autumn-67998
12/04/2022, 8:00 AM--venv=prepend
will "just work". Our code is split across two pex files. We run `source.pex`and specify the deps.pex
using PEX_PATH
. The console scripts are provided by deps.pex
. If I build source.pex
with --venv=prepend
, looks like all scripts from both pexes are put in the venv's bin directory added to the PATH.enough-analyst-54434
12/05/2022, 2:45 AMabundant-autumn-67998
12/07/2022, 10:06 PM--venv=prepend
and I just want to make sure I understand all the implications. At runtime we invoke the pex with subprocess
. It appears to unpack the dependencies into ~/.pex/unzipped_pexes/<hash>
. If we switch to doing --venv=prepend
builds, I believe the pex can still be run the same way but will unpack the deps into ~/.pex/venvs/<hash>/.../site-packages
and also produce surrounding folders such as bin
etc. We never directly look at ~/.pex
ever so I'm assuming this will require no change at the run site and only a build flag change. Is this correct?enough-analyst-54434
12/07/2022, 10:13 PM--venv
, --venv prepend
and --venv append
all create ~/.pex/venvs/...
and run from those. As part of the process of creating a venv on the 1st ever run, a ~/.pex/unzipped_pexes/...
entry happens to get made, but never is used again.
The latter part is correct, --venv *
nets you a bin/
that includes console scripts, they just won't be on the PATH
unless you use {append,prepend}
. In that not on the PATH
case, you'd need something like:
subprocess.run(args=[PurePath(sys.executable).parent / "console_script_name", ...], ...)
--venv
don't make that clear, (I just re-read them and of course they seem clear to the author!), then I'd welcome a PR fixing those up. At some point here in the new year my stack will clear and I'll be circling back to documenting the `pex3 lock`tools and everything else in prep to cut Pex 3.x. The readthedocs is very old and ~not updated since the Pex 1.x days before I took on maintainership.abundant-autumn-67998
12/07/2022, 11:48 PM--venv
docs are clear and it will give us the feature we want (console scripts). I was just trying to see if turning it on may break us in some unexpected way because of some assumption we have that only exists during runtime in the non-venv case. I think it's a safe change though and will try it out.scie-jump
looks very interesting. (Though I don't think we need it though in our current setup)enough-analyst-54434
12/08/2022, 12:18 AM--venv
(+ --venv-site-packages-copies
) gives you the closest thing to a fully normal hand made venv and should be the most compatible with the most code. The normal --venv
symlinks 3rdparty dists from the ~/.pex/installed_wheels/...
dirs into the venv `site-packages/`dir and this can confuse a few distributtions out there in the wild; so you may need to turn on copies, which of course just wastes space that might otherwise be saved by re-using via symlink.~/.pex
tree.abundant-autumn-67998
12/08/2022, 12:20 AMenough-analyst-54434
12/08/2022, 12:21 AMabundant-autumn-67998
12/08/2022, 12:23 AMenough-analyst-54434
12/08/2022, 12:24 AMif the entry point or number of processes changesI don't understand what that means.
abundant-autumn-67998
12/08/2022, 12:24 AMYeah, you're more likely to have already hit issues than you will be using --venvindeed - the console scripts was one issue
enough-analyst-54434
12/08/2022, 12:27 AM~/.pex/unzipped_pexes
on every run.
+ PEX --venv: 1 re-exec into ~/.pex/unzipped_pexes
oin 1st run to build venv then 1 re-exec into ~/.pex/venvs/...
, On 2+ run, just 1 re-exec straight into ~/.pex/venvs/...
unless it has been nuked.abundant-autumn-67998
12/08/2022, 12:27 AMPEX_TOOLS
into a short venv directory, the stacks traces will be nicer.enough-analyst-54434
12/08/2022, 12:28 AM--sh-boot
but still ugly paths.abundant-autumn-67998
12/08/2022, 12:29 AMvenv/.../pex
directly?enough-analyst-54434
12/08/2022, 12:29 AMabundant-autumn-67998
12/08/2022, 12:32 AMenough-analyst-54434
12/08/2022, 12:32 AM--layout packed
? DO you use that layout + rsync?abundant-autumn-67998
12/08/2022, 12:33 AMenough-analyst-54434
12/08/2022, 12:33 AMabundant-autumn-67998
12/08/2022, 12:33 AMenough-analyst-54434
12/08/2022, 12:33 AMabundant-autumn-67998
12/08/2022, 12:38 AMenough-analyst-54434
12/08/2022, 12:40 AMabundant-autumn-67998
12/08/2022, 12:42 AM