https://pantsbuild.org/ logo
Join Slack
Powered by
Status of pyozidizer I am using `pyoxidizer` which...
# general
f
Status of pyozidizer I am using 
pyoxidizer
which only supports Python up to Python 3.10 which means I am unable to move on to Python 3.11 or 3.12. That's the only reason I need multiple Python resolves in our monorepo. This introduces complexity and it affects also other parts of my toolchain. There is an issue https://github.com/indygreg/PyOxidizer/issues/716 by @bitter-ability-32190 that is open since August 2023, but the mentioned PR is unmerged. The last commit to 
main
in https://github.com/indygreg/PyOxidizer is more than a year ago and the last 
pyoxidizer
release was on Dec 30, 2022 (v0.24). While I greatly appreciate any efforts that people put into open source projects (especially in their free time), I'm interested whether someone has insights into the maintenance status of 
pyoxidizer
.
w
I haven't looked into the status of 
pyoxidizer
specifically, but if you're looking for something to sorta package up a Python with your code - you could also investigate https://github.com/indygreg/python-build-standalone We use it in Pants, and you can build `scie`s with it as well (PBS + PEX + a bit of code to put the two into a zip, unpackage it all, and run it) - https://github.com/a-scie This is how scie-pants (https://github.com/pantsbuild/scie-pants) works under the hood basically
No idea if this will help with your workflow, but it's what I've moved my bundled code to (away from PyOxidizer - which was also pretty slow to run and had some sharp edges)
f
Thanks for the proposal. The approach is a technically feasible solution. I am concerned about start-up time for the end product and the pex startup time is quite slow. I guess it has to unpack the pex file before running it, right? Pyoxidizer in contrast seems pretty fast to me.
Just found this. It pre-extracts the pex in the docker image. Might be a good solution.
w
Ah, didn't realize that docker was the end goal - if so, yeah, that blog is probably a safe bet. Here is one of a few large threads we had on this topic in the past few days: https://pantsbuild.slack.com/archives/C046T6T9U/p1707419264786569
FWIW the 
scie
approach is what i use outside of dockerized solutions - it unpacks everything on first run (using basically the same commands)
Copy code
[[lift.files]]
name = ":apigateway-pex"

[[lift.commands]]
exe = "{scie.bindings.venv}/venv/bin/uvicorn"
args = ["apigateway.main:app", "--port", "7999"]
description = "The FastAPI executable."

[[lift.commands]]
name = "seed"
exe = "#{cpython:python}"
args = ["{:apigateway-pex}", "-m", "seeder.main"]
description = "The command to create a sqlite database from multiple input files"

[[lift.bindings]]
name = "venv"
description = "Installs the APIGateway into a venv and pre-compiles .pyc"
exe = "#{cpython:python}"
args = [
    "{:apigateway-pex}",
    "venv",
    "--bin-path",
    "prepend",
    "--compile",
    "--rm",
    "all",
    "{scie.bindings}/venv",
]
b
If you're using docker, just unpack the PEX in one layer, and download and extract Python Build Standalone in another
f
Thanks for the advice. I have both requirements, building a docker image that starts fast and runs a pants-built Python artifact (e.g. pex) as well as deploying something to developer machines that is self-contained (incl. Python) and does not use Docker. I will look into scie and how the end user experience of that scenario looks like. Btw, awesome community here!
❤️ 2
w
BTW: Proof of concept 
scie
plugin here: https://github.com/sureshjoshi/pants-plugins/tree/main/pants-plugins/experimental/scie I haven't touched it in a while, but it's what I am using in a few other projects. I have time allocated in March to ... better-ify it, so it's more generic for (e.g. multiple pexes)
b
w
There seem to be commits still trickling in, but no recent releases 🤷
b
Obviously that means it's perfect. 😅
🙃 1
w
SOLD!
f
Mmh, XAR requires SquashFS, so users need to have something preinstalled. That's a problem for our product. Also it only works on Linux. I need something that works on Linux, MacOS and also Windows (for the non-Docker part). Will 
scie
also work on Windows?
w
Only depending on whether pexes work on Windows. As far as my last check, that answer was still a no, until a bunch of unit tests go green. But, I did read some slack messages talking about pex and windows, making me think it worked - but I believe those might have been in WSL/cygwin environments? Actually, Pex + Windows is definitely a topic I need to refresh myself on - I don't want to speak out of turn until I review the repo and test it out on Windows...
f
w
f
Welcome to the jungle of Python cross-platform packaging and distribution.
b
Snakes everywhere 🤣
w
✈️
I was just thinking this to myself about C++ - it's so frustrating to have a complaint about something (non-first-party C++ package managers) when you basically first start programming a language, and then have that SAME complaint like 15 years later
e.g. I like 
cppfront
and some of the "typescript for C++" style ideas, but man - solve the whole problem (<--- hyper trivializing a very hard problem here). Dev experience is what helps with uptick. Bahh /rant
h
scie should work on Windows, I believe
w
scie
with Python pexes on Windows?
Ah re-reading, I guess Marco never originally specified pex'ing anything
f
I would go for pex if it's cross-platform (Lin,Mac,Win) and has a considerably fast start-up time.
In a nutshell I need a distributable Python package that contains all dependencies including the interpreter on all platforms. And it should start fast ideally, at least on subsequent calls.
w
aka. The dream 🙂
https://github.com/a-scie/jump Check out the 
scie
stuff and see if it could work for you. Unfortunately, I abandoned Windows development a long time ago. I still have some colleagues who insist on it, but end up using WSL - so I've never really had need to find a fully cross-platform, all-in solution
I'd be interested if there was another python bundled format that could maybe be run (ala pex) on Windows? Like 
shiv
or something? And then run that in a 
scie
?
Buuuut, if you had time and wanted to help get pex on windows over the line ........... 🙂
f
I fiddled with https://nuitka.net/ last night (while I should be sleeping) and had pretty impressive results. It correctly inferred all imports and picked up the dependencies from my current venv (Python 3.11). I could also build a standalone version that includes the interpreter. I think I will integrate this as an adhoc_tool for now.
The main dev https://github.com/kayhayen also has a nice business model around this. I feel there is quite some intelligence behind this compiler. Any new Python version needs a lot of investment on his side to get new constructs and concepts working when translating to C.
w
Cool. I tried nuitka a few times over the years, and it just never worked as I needed it to for anything non-trivial. It sounds like it’s time to revisit again! I had wanted to make a nuitka pants plugin, but yeah, it just kept falling over on some workflows I had
14 Views
Open in Slack
PreviousNext
Powered by