https://pantsbuild.org/ logo
#general
Title
# general
c

cool-yacht-37128

09/20/2022, 7:32 AM
Is it possible to set a PEX
--runtime-pex-root
for a
pex_binary
target? I don’t see any mention of this option on the docs page. On most hardened Linux systems (eg US STIG) the home dir is mounted non-executable (noexec) which prevents any wheels with .so libraries from executing.
1
Actually nevermind. I didn’t think this through.
even if I could set a --runtime-pex-root, any location I use would compromise the security of the OS. Since by design the no file systems are both user writable and executable.
To give context here is what I found: • On a hardened Linux system all normal user writable locations are mounted in
noexec
mode. Meaning even if you store a file there, you cannot run any executables from those locations. Not even if you are root • A .pex built with the default zipapp layout unpacks itself to ~/.pex • Where you get
failed to map segment from shared object
errors are when you have python modules which use
.so
library files. These are often memory mapped, and noexec prevents access to such files (more detail). So python packages that use compiled .so libraries cannot be run from user writable paths • The same issue doesn’t occur if you run the .pex via
sudo
because the pex is run as the root user, and the root user’s home dir is
/root
which is mounted with exec mode. Even if it was possible to direct PEX to which dir to use as a cache location, in practise on a hardened linux server there are no good paths that are user writable and can execute binaries by design. Creating a directory in /opt that is world writable just compromises the entire OS security. Thus it’s not possible at all to run .pexs by normal users on such hardened systems at all if they contain modules that container
.so
libraries My ugly solution: A shell script wrapper around the pex to check if the user is a superuser or not, and attempt to run the pex via sudo.
e

enough-analyst-54434

09/20/2022, 1:28 PM
Yeah, you're running into limitations of zipapps generally. I don't see any real workaround. You really need to ship a self-contained executable in that case (maybe investigate PyOxidizer et. al.: https://github.com/indygreg/PyOxidizer and Pants nascent support for it: https://www.pantsbuild.org/docs/pyoxidizer). See here for more on zipapp caveats in general: https://docs.python.org/3/library/zipapp.html#caveats
How do you currently "install" the PEX file on this type of machine? Depending on that, perhaps you could leverage pex-tools to install a venv from the PEX file instead. Basically if the PEX is built with
--include-tools
you can then run
PEX_TOOLS=1 the/pex venv install/here --compile --rm all
which will install the PEX at
install/here
with an executable
install/here/pex
script that acts just like the original PEX file (which is deleted by
--rm all
).
For your use case it would be useful if there was an
--alias my-app-exe-name
that created a relative symlink at
install/here/my-app-exe-name
that pointed at
install/here/pex
or even simply re-named it.
If that would be useful / solve this - please file a Pex issue at https://github.com/pantsbuild/Pex/issues/new
c

cool-yacht-37128

09/20/2022, 10:48 PM
I’m packaging the .pex file into a rpm, so the rpm system itself does the install. So I actually have a lot of flexibility at my disposal. It might be easiest if I skipped zipapp layout all together (rpm does it own compression), and just did a loose pex layout. That would allow rpm to also manage the clean up during upgrade/uninstall.
I’ll have a closer look at PyOxidizer though. That looks like it might solve some of my other issues (supporting aging distros with only old python versions availiable)
Thanks for your help and advice @enough-analyst-54434 You’ve been excellent as always
e

enough-analyst-54434

09/20/2022, 10:54 PM
One think to keep in mind on
--layout loose
PEXes (or packed) is these are slower and less compatible than
--venv
PEXes). The fastest is a PEX_TOOLS created venv - these run at native Python / venv speed and are the most compatible with arbitrary Python programs since these all expect to run in a venv.
The slower is up to ~O(100ms) of extra overhead on each run of the program over a venv PEX.
c

cool-yacht-37128

09/20/2022, 10:55 PM
Gotcha. Venv install is preferable, easy to do during rpm build time. Will consider that too!
e

enough-analyst-54434

09/20/2022, 10:57 PM
Great. I'm definitely interested in your solution if it does happen to use PEX in the end. That's a pretty novel use case as far as I know, but I have been thinking a good bit about end-user apps lately and rpms and Windows ~exes deserve some love.
c

cool-yacht-37128

09/20/2022, 11:07 PM
Single python interpreter binary that can run on six different operating systems (Linux, Mac, Windows, NetBSD, FreeBSD, OpenBSD)
e

enough-analyst-54434

09/20/2022, 11:23 PM
I have not.
I'm hoping to find some time to use parts of the PyOxidizer project to allow you to turn a PEX into a ~PAR that comes along with the Python interpreter embedded - but otherwise still extracts, etc before running.
c

cool-yacht-37128

09/20/2022, 11:56 PM
It’s a fun idea, looks like a lot of insanely low level work. I can’t see this becoming stable for a long time