https://pantsbuild.org/ logo
Join Slack
Powered by
I've been working on a long brewing idea the past ...
# development
e
I've been working on a long brewing idea the past two weeks. TLDR: I wanted PAR (Google's internal Pex inspiration that also ships a Python interpreter) for Pex but wound up with more, for, I think, less. I'd like to see what you all think about this and whether or not it makes sense to use for Pants distribution. I'm certainly going to use it to implement PAR for Pex. 🧵
❤️ 1
So @ancient-vegetable-10556 did alot of work to get PyOxidizer working for the Pants binary. See here for gory details: https://github.com/pantsbuild/pants/pull/16484 He had to jump through several hoops due to the way PyOxidizer works. IIUC he got this all working save for perhaps plugins.
What I came up with attacks the problem from a higher level: + https://github.com/a-scie/jump + https://github.com/a-scie/jump/releases/tag/v0.1.11
There is reading to do there, but I like demonstrations, so here is me building a Pants binary and using it using this tool (N.B. That indygreg still has his fingerprints in there!): Assemble scie-pants:
Copy code
$ PY=python3.9 ./pants package src/python/pants/bin:pants
10:56:23.81 [INFO] Completed: Building local_dists.pex
10:56:33.39 [INFO] Completed: Building 18 requirements for src.python.pants.bin/pants.pex from the 3rdparty/python/user_reqs.lock resolve: PyYAML<7.0,>=6.0, ansicolors==1.1.8, chevron==0.14.0, fasteners==0.16.3, humbug==0.2.7, ijs... (264 characters truncated)
10:56:33.42 [INFO] Wrote dist/src.python.pants.bin/pants.pex
$ mkdir scie-pants
$ cp dist/src.python.pants.bin/pants.pex scie-pants/
$ curl -fL -O <https://github.com/indygreg/python-build-standalone/releases/download/20221002/cpython-3.9.14+20221002-x86_64-unknown-linux-gnu-install_only.tar.gz>
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 25.5M  100 25.5M    0     0  3897k      0  0:00:06  0:00:06 --:--:-- 5062k
$ mv cpython-3.9.14+20221002-x86_64-unknown-linux-gnu-install_only.tar.gz scie-pants/
$ curl -fL -O <https://github.com/a-scie/jump/releases/download/v0.1.10/scie-jump-linux-x86_64>
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 1521k  100 1521k    0     0  1300k      0  0:00:01  0:00:01 --:--:-- 2449k
$ chmod +x scie-jump-linux-x86_64
$ mv scie-jump-linux-x86_64 scie-pants/scie-jump
$ cd scie-pants/
scie-pants $ vi lift.json
$ cat lift.json
{
  "scie": {
    "lift": {
      "name": "pants",
      "boot": {
        "commands": {
          "": {
            "env": {
              "=PATH": "{cpython}/python/bin:{scie.env.PATH}"
            },
            "exe": "{cpython}/python/bin/python3.9",
            "args": [
              "{pants.pex}"
            ]
          }
        }
      },
      "files": [
        {
          "name": "cpython-3.9.14+20221002-x86_64-unknown-linux-gnu-install_only.tar.gz",
          "key": "cpython"
        },
        {
          "name": "pants.pex"
        }
      ]
    }
  }
}
scie-pants $ ./scie-jump
/home/jsirois/dev/pantsbuild/pants/scie-pants/lift.json: /home/jsirois/dev/pantsbuild/pants/scie-pants/pants
Inspect what we just built:
Copy code
scie-pants $ file pants
pants: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=dfa49ce2c9a49fd3e63fe07a43098dfef38bfc8b, for GNU/Linux 3.2.0, stripped
scie-pants $ zipinfo pants | tail
warning [pants]:  28319597 extra bytes at beginning or within zipfile
  (attempting to process anyway)
-rw-r--r--  2.0 unx     3668 b- defN 80-Jan-01 00:00 pants/util/osutil.py
-rw-r--r--  2.0 unx      947 b- defN 80-Jan-01 00:00 pants/util/resources.py
-rw-r--r--  2.0 unx    10759 b- defN 80-Jan-01 00:00 pants/util/strutil.py
-rw-r--r--  2.0 unx     4557 b- defN 80-Jan-01 00:00 pants/util/value_interpolation.py
drwxr-xr-x  2.0 unx        0 b- defN 80-Jan-01 00:00 pants/vcs/
-rw-r--r--  2.0 unx        0 b- defN 80-Jan-01 00:00 pants/vcs/__init__.py
-rw-r--r--  2.0 unx     6229 b- defN 80-Jan-01 00:00 pants/vcs/changed.py
-rw-r--r--  2.0 unx     6260 b- defN 80-Jan-01 00:00 pants/vcs/git.py
-rw-r--r--  2.0 unx      794 b- defN 80-Jan-01 00:00 pants/version.py
3316 files, 356390776 bytes uncompressed, 68422706 bytes compressed:  80.8%
scie-pants $ tail -1 pants | jq .
{
  "scie": {
    "lift": {
      "name": "pants",
      "base": "~/.nce",
      "files": [
        {
          "name": "cpython-3.9.14+20221002-x86_64-unknown-linux-gnu-install_only.tar.gz",
          "key": "cpython",
          "size": 26761725,
          "hash": "e63d0c00a499e0202ba7a0f53ce69fca6d30237af39af9bc3c76bce6c7bf14d7",
          "type": "tar.gz"
        },
        {
          "name": "pants.pex",
          "size": 69142851,
          "hash": "c8f1bd4a7cf6a2b4f80c366ac3c77d68bb73a72310ae9ed14f829b0fe16c9c2c",
          "type": "blob"
        }
      ],
      "boot": {
        "commands": {
          "": {
            "exe": "{cpython}/python/bin/python3.9",
            "args": [
              "{pants.pex}"
            ],
            "env": {
              "=PATH": "{cpython}/python/bin:{scie.env.PATH}"
            }
          }
        }
      }
    },
    "jump": {
      "size": 1557872,
      "version": "0.1.10"
    }
  }
}
Put on scie-pants:
Copy code
scie-pants $ cd ..
$ scie-pants/pants auth-acquire
11:41:30.91 [INFO] Starting: Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.24.0
11:41:33.21 [INFO] Completed: Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.24.0
11:41:34.35 [WARN] DEPRECATED: `pants.engine.environment.Environment` is scheduled to be removed in version 2.17.0.dev0.

Use `pants.engine.env_vars.EnvironmentVars`.
11:41:34.35 [WARN] DEPRECATED: `pants.engine.environment.EnvironmentRequest` is scheduled to be removed in version 2.17.0.dev0.

Use `pants.engine.env_vars.EnvironmentVarsRequest`.
11:41:34.36 [WARN] DEPRECATED: `pants.engine.environment.CompleteEnvironment` is scheduled to be removed in version 2.17.0.dev0.

Use `pants.engine.env_vars.CompleteEnvironmentVars`.
11:41:34.40 [INFO] Initializing scheduler...
11:41:34.46 [WARN] DEPRECATED: Setting `Goal.environment_behavior=EnvironmentBehavior.UNMIGRATED` for `Goal` `auth-acquire` is scheduled to be removed in version 2.17.0.dev0.

See <https://www.pantsbuild.org/v2.15/docs/plugin-upgrade-guide>

11:41:34.46 [WARN] DEPRECATED: Setting `Goal.environment_behavior=EnvironmentBehavior.UNMIGRATED` for `Goal` `auth-token-info` is scheduled to be removed in version 2.17.0.dev0.

See <https://www.pantsbuild.org/v2.15/docs/plugin-upgrade-guide>

11:41:34.46 [WARN] DEPRECATED: Setting `Goal.environment_behavior=EnvironmentBehavior.UNMIGRATED` for `Goal` `auth-token-check` is scheduled to be removed in version 2.17.0.dev0.

See <https://www.pantsbuild.org/v2.15/docs/plugin-upgrade-guide>

11:41:35.35 [INFO] Scheduler initialized.
Using a web browser navigate to: <https://app.toolchain.com/api/v1/token/auth/?headless=1&repo=pantsbuild%2Fpants>
Type or paste in the token exchange code: XXX
Enter token description [Gill-Windows]: Putting on my scie-pants
Access token acquired and stored.
The only asterisk is this diff was required on main just now to do this:
Copy code
$ git diff
diff --git a/pants-plugins/internal_plugins/test_lockfile_fixtures/lockfile_fixture.py b/pants-plugins/internal_plugins/test_lockfile_fixtures/lockfile_fixture.py
index 08ddce60f..e3a089339 100644
--- a/pants-plugins/internal_plugins/test_lockfile_fixtures/lockfile_fixture.py
+++ b/pants-plugins/internal_plugins/test_lockfile_fixtures/lockfile_fixture.py
@@ -7,7 +7,7 @@ from dataclasses import dataclass
 from pathlib import Path
 from typing import Iterable

-from _pytest.fixtures import FixtureRequest
+#from _pytest.fixtures import FixtureRequest

 from pants.jvm.resolve.common import ArtifactRequirement, ArtifactRequirements, Coordinate
 from pants.jvm.resolve.coursier_fetch import CoursierResolvedLockfile
diff --git a/src/python/pants/bin/BUILD b/src/python/pants/bin/BUILD
index f7a96c06a..8d82db797 100644
--- a/src/python/pants/bin/BUILD
+++ b/src/python/pants/bin/BUILD
@@ -56,6 +56,7 @@ target(
         "src/python/pants/backend/experimental/scala/debug_goals",
         "src/python/pants/backend/experimental/scala/lint/scalafmt",
         "src/python/pants/backend/experimental/terraform",
+        "src/python/pants/backend/explorer",
         "src/python/pants/backend/google_cloud_function/python",
         "src/python/pants/backend/plugin_development",
         "src/python/pants/backend/project_info",
I think that has nothing to do with the scie-pants approach, its just PEX exposing non-hermiticity in the dev deps.
Is this something folks are interested in pursuing? @wide-midnight-78598 you've obviously had experience with PyOxidizer. This is an alternative to that for Python apps and something new altogether perhaps for node.js and java (which has graal which does a lot better, but is specialized).
Feel free to ask questions - alot of dump here at once!
w
I have a lot of reading before I can form an opinion on this, but I like PyOxidizer as an alternative to what currently exists in the python packaging space (which I think is sorely lacking, imho). PyOx seemed like a breath of fresh air, and it absolutely is. Having said that, I've had to deal with a lot of the workarounds, and tweaks, and mods. So, it's not the out-of-the-box single-file app experience I would like, but it's as close as I've found so far. I've quietly been wondering on the side if there would be someway to work directly with a shipped python interpreter (specifically, IndyGreg's https://gregoryszorc.com/docs/pyoxidizer/main/pyoxy.html) - but I also admit this is so far beyond my day-to-day brain, so 🤷
So, I guess my focus would be on - how "single file executable"-y could this solution be, and are there any high level "this absolutely won't work for XYZ use cases"? Knowing those answers walking into this might provide some direction for better questions (from me at least)
h
I think this is potentially huge, both as a way to distribute Pants, and in the future as a format Pants can build for users. Doing the former first, even if we do it outside of Pants at first, would be a great way to kick the tires on those kinds of questions.
e
@wide-midnight-78598 I'm not sure what you mean exactly about "single file executable"-y. A scie is a single file executable. It just happens to then extract parts of itself to a CAS cache before executing. Since its a CAS, things like Python interpreters contained within sices are de-duped and re-used. Since you ship full distributions of CPython, a JDK or a node.js installation - or what have you - the corner cases seem non-existent. The only limitation is coming up with a packaged interpreter. For Python I chose Python Build Standalone by IndyGreg since he's amazing and I trusted these would work ~anywhere. For Node there was just the one provider - I did not try deno?. And for the JDK, well there are lots of choices, Picking Amazon's distribution for the examples was ~random.
Believe me - I played with PyOxy yaml apps a good bit before arriving here.
w
Ahh, okay okay, so - to clarify what I meant. When I PyOx some projects, it's required that under some use cases, some libraries are necessarily placed on the filesystem relative to your executable. While these could then be packaged into a self-extracting solution, it's just another step. I'm more referring to a fully packaged "blah.exe" (or whatever extension), and what happens under the hood with memory/caches, I'm ambivalent to that part.
Grand scheme, it's a small thing - but as a user, it feels disproportionately big.
e
I'm not really following. Any app you can concoct that uses an interpreter and your own code - that just turns into a scie made up of a tarball of the interpreter and a zip of your app all inside the scie.
Maybe if you had a repo you could point me at on this side-by-side thing?
w
Yep, once the kid goes to sleep - I'll send you a trivial pyox example I already have.
e
Sounds good. For comparison, presumably that pyox app also can't be PEXed then. This scheme will work with every PEX by definition.
👍 1
b
My only concern isn't anything technical. I think it's quite brilliant, but in a very vanilla way. My concern is how much of Pants (at least the Python bits) are gated by John's (powerhouse of a) brain. In terms of knowledge and understanding, but also in terms of review. Not saying we're shouldn't do it, I'd absolutely love to decouple from system Python. I'm just saying we need you to wear abus-proof vest at all times.
h
I share your concern, but AFAICT this is significantly simpler than Pex, and would be relatively easy to hack on. Pex, on the other hand… I would dearly love some volunteers to get up to speed on Pex and help John own it.
e
Pex is hard. It has to deal with the outside world and its standards and the cases where those are broken. Its >30k lines of code on top of that and has been developed over the course of 12 years and counting. The scie-jump is easy. It defines its own standard and is <3k lines of code and is mainly feature complete in 16 days of development.
I'd certainly love folks to join me and disabuse the project of its sophomore Rust code and help shave microseconds. After dealing with how slow Pants and Pex are in the Python its a breath of fresh air to have your code fully complete execution in <200us.
w
For comparison, presumably that pyox app also can't be PEXed then. This scheme will work with every PEX by definition.
Not to derail a very important conversation, but to finally get this point in (spent a bunch of time fighting with xcode's schrodinger's toolchain, which was somehow both installed and not installed). In https://github.com/sureshjoshi/pants-plugins/tree/23-nested-examples, running this command illustrates what I mean about a true single-file executable (from a user's perspective). 
./pants package examples/python/helloworld:helloworld-bin examples/python/hellonumpy:hellonumpy-bin
Once PyOxidized, the simple helloworld is a standalone executable, while the numpy example requires numpy's lib to be installed alongside the executable. This example is a couple versions old, so maybe the state-of-the-art is different. I guess what I'm referring to is what IndyGreg terms as "Packaging files instead of in-memory resources" (https://pyoxidizer.readthedocs.io/en/stable/pyoxidizer_packaging_additional_files.html?highlight=classified#packaging-files-in[…]in-memory-resources). Are these types of problems still a thing here?
e
I'll find out and report back. I suspect not.
👀 1
In nearly, but not quite, as long as it took PyOxidizer to finish (I ctrl-c'ed):
Copy code
$ git diff
diff --git a/examples/python/hellonumpy/BUILD.pants b/examples/python/hellonumpy/BUILD.pants
index 4577320..aec0703 100644
--- a/examples/python/hellonumpy/BUILD.pants
+++ b/examples/python/hellonumpy/BUILD.pants
@@ -10,9 +10,8 @@ python_distribution(
     ),
 )

-pyoxidizer_binary(
+pex_binary(
     name="hellonumpy-bin",
     entry_point="hellonumpy.main",
     dependencies=[":hellonumpy-dist"],
-    filesystem_resources=["numpy"],
 )
$ ./pants package examples/python/hellonumpy:hellonumpy-bin
19:32:06.91 [INFO] Completed: List contents of artifacts produced by examples/python/hellonumpy:hellonumpy-dist
19:32:09.74 [INFO] Completed: Building local_dists.pex with 1 requirement: hellonumpy_dist-0.0.1-py3-none-any.whl
19:32:28.58 [INFO] Completed: Building 1 requirement for examples.python.hellonumpy/hellonumpy-bin.pex from the build-support/python/default_lock.txt resolve: numpy
19:32:28.60 [INFO] Wrote dist/examples.python.hellonumpy/hellonumpy-bin.pex
$ mkdir -p scie/no-snow-job
$ cp dist/examples.python.hellonumpy/hellonumpy-bin.pex scie/no-snow-job/
$ cp ~/dev/a-scie/jump/examples/python/cpython-3.9.14+20221002-x86_64-unknown-linux-gnu-install_only.tar.gz scie/no-snow-job/
$ vi scie/no-snow-job/lift.json
$ curl -sSL ^C
$ curl -sSL <https://github.com/a-scie/jump/releases/download/v0.1.11/scie-jump-linux-x86_64> > scie/no-snow-job/scie-jump
$ cd scie/no-snow-job/
scie/no-snow-job $ chmod +x scie-jump
scie/no-snow-job $ ./scie-jump
/home/jsirois/dev/a-scie/pants-plugins/scie/no-snow-job/lift.json: /home/jsirois/dev/a-scie/pants-plugins/scie/no-snow-job/pants
scie/no-snow-job $ vi lift.json
scie/no-snow-job $ rm pants
scie/no-snow-job $ cd ../..
$ cat scie/no-snow-job/lift.json
{
  "scie": {
    "lift": {
      "name": "hellonumpy",
      "boot": {
        "commands": {
          "": {
            "env": {
              "=PATH": "{cpython}/python/bin:{scie.env.PATH}"
            },
            "exe": "{cpython}/python/bin/python3.9",
            "args": [
              "{hellonumpy-bin.pex}"
            ]
          }
        }
      },
      "files": [
        {
          "name": "cpython-3.9.14+20221002-x86_64-unknown-linux-gnu-install_only.tar.gz",
          "key": "cpython"
        },
        {
          "name": "hellonumpy-bin.pex"
        }
      ]
    }
  }
}

$ cd scie/no-snow-job/
scie/no-snow-job $ ./scie-jump
/home/jsirois/dev/a-scie/pants-plugins/scie/no-snow-job/lift.json: /home/jsirois/dev/a-scie/pants-plugins/scie/no-snow-job/hellonumpy
scie/no-snow-job $ file hellonumpy
hellonumpy: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=3150baa5e6c07628bc4307dd87974895dd7a3e62, for GNU/Linux 3.2.0, stripped
scie/no-snow-job $ ./hellonumpy
Launching HelloNumpy from __main__
Hello, data! [[1 2]
 [3 4]]
It truly is dead dumb simple.
🤯 1
And this has 0 to do with Python. That's the punch line.
w
🤯 🤯 🤯 🤯
I'm still waiting for another shoe to drop somewhere, but John, this is dope. I cloned 
jump
, and just threw in a use case that PyOx couldn't handle off the bat (noting that I haven't deep dived into the PyOx bug). Pex'd a 
kivy
app, then ran 
jump
on it, and it Just Works™️ Took maybe... 30 seconds of effort?
e
Excellent. Yeah no shoes. This is just basic science. The only gotcha's are pretty much on the tin: these things need a writable filesystem to extract to whereas a true always a single file binary like PyOxidizer can produce doesn't have this restriction. Of course PEX files have the same caveat and I know I've had customers there that have looked for ways around that limitation.
w
Is there a way it could be an in-mem filesystem? Is that a python limitation, or a tool limitation, or a time limitation?
Anecdotally, though, this already solves a lot of what drew me to PyOx. Being able to ship a python interpreter alongside an application - as a standalone, executable package. It's my Dad test. Can I write a tool or application that I can send my dad to use on his machine, and not have to debug or tweak stuff or be present for installation. For my servers, this also helps eliminate Dockers for the purpose of providing Python interpreters. In fact, I'd wager that I could drop in jump here: https://blog.pantsbuild.org/packaging-python-with-the-pyoxidizer-pants-plugin/ and have a similar analysis (at least, if this PyOx quote holds true):
PyOxidizer allows you to distribute your code as a single binary file, similar to Pex files. Unlike Pex, these binaries include a Python interpreter, often greatly simplifying distribution.
e
Well, in mem already is supported. Just specify a different 
scie.lift.base
. You can see the default of 
~/.nce
get filled in in the lower reified form created by the boot-pack here: https://github.com/a-scie/jump/tree/main/jump#readme
But the larger problem requires fuse be pre-installed on the host (for root-less mounting) or else root to mount arbitrary filesystems. And that's just Linux. No clue other OSes.
@wide-midnight-78598 I'm a little disappointed. I wanted to hear you were shipping node apps this way!
w
Lol, I don't ship node apps 🙂
e
Aha, the js is frontend. Gotcha.
w
Backends are Python or Bust for me (and soon, Python or Rust). Maybe C#, to be determined after a client meeting in a few weeks. TS + Svelte front-end for everything - last time I did a node backend, it was in 2015 or so, and I didn't like anything about it. However, this might have just opened up a new avenue that I was trying to PyOx with, which is standalone desktop apps using Python and web (ala Electron). I was literally looking at this Rust project yesterday to solve a similar problem (https://tauri.app/)
e
The last feature I'm aware of of any importance is in https://github.com/a-scie/jump/issues/7. I'm trying to knock that out today. Pants would need it for optimal installation. That may or may not be useful in the scenarios you have in mind. Basically commands that run once ever on a machine as prep for the main main.
w
Yep, saw that - which covered one of my questions PEX_TOOLS and the like
e
Explain more? You can already ship a busybox with multiple commands selected by SCIE_BOOT=x
In CI that's done here in the Pants / Python example: https://github.com/a-scie/jump/actions/runs/3415104455/jobs/5683860180#step:9:167
A proof the Pants scie is actually using it's embedded interpreter by calling an alternate 
inspect
command defined here: https://github.com/a-scie/jump/blob/0e359df388da7297f4d072605e8ad1f90d132cb9/examples/python/lift.macos-x86_64.json#L18-L28
w
Ah, okay, I may have mis-understood the ticket then. I thought post-install/pre/post-run script commands were unsupported. For example, I'm trying to replace this kinda Dockerfile (not the real file, which has a lot of other shenanigans going on):
Copy code
FROM nginx/unit:1.25.0-python3.9
COPY ./config.json /docker-entrypoint.d/config.json

COPY apigateway.pex /apigateway.pex
RUN PEX_TOOLS=1 /apigateway.pex venv /app
# ... run some one-off database init/migration type stuff here ...
WORKDIR /app
e
No, you have the ticket right. Exactly - it covers things just like that.
w
Ah okay, yeah - not a big deal, but I just thought of it because of that ^ project, where I'm re-writing the deployment files
e
Ok. Yeah - it is a big deal to me. That nets back 50ms in pantsd boot time.
Can I put you on code review for that change?
w
👍 Won't be much help for the code, as I'm a Rust noob - but I'll definitely test it out
e
Ok, that's still useful. Thanks - I'll take you up on that.
👍 1
a
Just to mirror things I’ve said elsewhere, this seems like a much more reliable solution to making a stand-alone Pants distribution than PyOx — scie using a standard Python interpreter importer with no weird custom interpreter means that there aren’t weird interactions and moving targets with Pex.
w
"weird custom interpreter"?
a
I keep writing “interpreter” when I mean “importer”
😆 1
PEX uses 
__file__
all over the place, and oxidized-importer famously does not set that variable, because it’s non-sensical
e
Ah right, that trick. I learned that from Pip.
It does the same
I think Pip learned it from setuptools.
a
Right, so even if PEX didn’t, the pip invoked by Pex would
e
It too does the same.
a
I don’t doubt that it’s utterly necessary for the use cases — they seemed like it at the time — but oxidized-importer doesn’t set it, because it takes the view that because the interned stdlib would never have 
__file__
set, then why should anything else?
which, as it turns out, is fundamentally wrong
e
Yeah.
The JVM got little right, but they got this right.
Resources were a 1st class concept from ~day one.
w
This ticket amuses me, re _file _ https://github.com/indygreg/PyOxidizer/issues/69
a
“NO, IT’S EVERYONE ELSE WHO IS WRONG”
e
Yeah, he's fighting an uphill battle there.
I fight that battle against Mac for folks who don't write Mac apps. Its a losing battle and I know it.
a
importlib-resources does indeed solve the problem, but again, there’s a reason why Pex needs it
e
Yeah, the problem is importlib-resources did not ship from day ~1 in the stdlib. That's what Java got right that Python did not.
I imagine Ruby is an even worse situation.
a
I always get the idea that Python accidented upon distributable applications and has been attempting to catch up ever since
e
Absolutely true.
Python was just Guido to start. Java, Go, Rust Swift were all corporations.
Its easy to get things right with ~unlimited time and manpower.
a
and clarity of purpose. If Guido cared about packaging as much as he did syntax…
e
Exactly.
I have a friend who makes his living off of Python education and he's certainly very glad Guido had the particular focus he did. My friend chose Python to teach / write about exactly because of the approachable syntax.
a
Indeed.
I used to teach Python for the same reason
w
@enough-analyst-54434 Any way to debug the problem I'm seeing? Created this helloworld example (https://pywebview.flowrl.com/) as a pex via pants, then jump'd it. I get logs saying that 
webview.start()
was called, but "nothing" happens. No webview launched, no windows launched - just kinda sits there. It was another example of something PyOx fails to build, because of 
__ file__
The webview debugger appears to suggest we have started/launched a webview (
[pywebview] Using Cocoa
), and my activity manager shows that the desired URL is part of the networking stack, so something is definitely happening, but nothing is popping up on screen. The 
pex
file works on its own, no problem
e
I don't have a Mac, but just take the jump out of the loop. 
SCIE=inspect ./your_scie
That will give you the components of the command line executed.
Execute it by hand.
Are you using the Python Build Standalone for your Python distribution? Perhaps those are hobbled in some way for UI stuff on a Mac?
If so, just build / bundle your own distribution.
Oh wait, maybe this is not Mac specific. Let me try to Pex this and report back.
w
Yeah, I did, but I'm not using the system python - so I would have assumed that the bundled interpreter would be "similar" to Homebrew's, unless homebrew bundles it - or if pywebview uses my system one regardless
Now, I question that assumption........
e
I have not really researched Python Build Standalone. It was an expedient choice. You could use pyenv to make yourself a distro.
just tar up ~/.pyenv/versions/X.Y.Z
w
For sure, but definitely a good angle for me to look into! Thanks!
e
But that won't distribute well probaly because it will have dynamic links
Basically though, the jump does nothing, so debugging is just running the command line it does, which is in the lift manifest in the scie.
w
Yeah, and there is nothing special going on there - I'll unpack the standalone interpreter tomorrow and run the pex against it directly. that should lead somewhere
Huh... Weird. Works with the downloaded python 🤯
e
What do you mean by downloaded python?
w
cpython-3.9.14+20221002-x86_64-apple-darwin-install_only.tar.gz
- unpacked that and ran it against the pex using the command line args, just as a sanity. Works fine, which I didn't expect.
e
PLease do the SCIE=inspect thing I suggested
w
Copy code
{
  "scie": {
    "lift": {
      "name": "hellowebview",
      "base": "~/.nce",
      "files": [
        {
          "name": "cpython-3.9.14+20221002-x86_64-apple-darwin-install_only.tar.gz",
          "key": "cpython",
          "size": 17138819,
          "hash": "b7d3a1f4b57e9350571ccee49c82f503133de0d113a2dbaebc8ccf108fb3fe1b",
          "type": "tar.gz"
        },
        {
          "name": "hellowebview-bin.pex",
          "size": 3262176,
          "hash": "a598744b565968520973431c93c32a34827e5dabf7e460053ecdb1c4d2316460",
          "type": "blob"
        }
      ],
      "boot": {
        "commands": {
          "": {
            "exe": "{cpython}/python/bin/python3.9",
            "args": [
              "{hellowebview-bin.pex}"
            ],
            "env": {
              "=PATH": "{cpython}/python/bin:{scie.env.PATH}"
            }
          }
        }
      }
    },
    "jump": {
      "size": 1433008,
      "version": "0.1.10"
    }
  }
}
e
Sorry
RUST_LOG=trace ./your_scie
That should print out the reified command.
w
Okay, concerningly, with the RUST_LOG, it worked (pop-up), and now the hellowebview pops up a GUI, where it didn't a few minutes ago
e
I don't know what to tell you.
That should affect nothing.
w
This feels like a coffee in the morning problem, and a dash of - wipe everything and start from scratch
e
Sounds good.
👍 1
image.png
No clue what 'gi' is, but that seems to stop rendering the url in the Window.
Packaging up a gui app is going to be hard, those almost always rely on system libs and those will not be shippable with the interpreter. The end user system will just have to have those installed. This is definitely a dumb simple tool and won't do anything with smarts for you. It won't know how to package up native libs.
w
So, the reason I like this one is that it specifically calls out to the system's web-centric tools. (edge/chromium, webkit, and qt/gtk on linux). If the user doesn't have a certain browser (and if I don't enable looking for it), then fair enough, it won't work Other option is to package chromium or similar. The Kivy GUI app I made also worked using jump. So, it's already far ahead from what I can tell. Have a lot more experimentation, and the desktop GUI side of what I personally do is super minimal (honestly, mostly just tools for myself) For this webview thing, my "supported" alternative is this: https://pywebview.flowrl.com/guide/freezing.html - which... nah...
e
I just hacked in the #7 boot binding support and used it for pants 2.14.0 to pre-install a fixed venv:
Copy code
$ hyperfine -w2 './pants -V' './scie-pants -V'
Benchmark 1: ./pants -V
  Time (mean ± σ):     688.2 ms ±  26.6 ms    [User: 562.5 ms, System: 25.7 ms]
  Range (min … max):   661.2 ms … 751.4 ms    10 runs

Benchmark 2: ./scie-pants -V
  Time (mean ± σ):     480.4 ms ±  19.2 ms    [User: 423.0 ms, System: 15.9 ms]
  Range (min … max):   466.6 ms … 532.3 ms    10 runs

Summary
  './scie-pants -V' ran
    1.43 ± 0.08 times faster than './pants -V'
So, apparently, the 
./pants
bash script has ~150ms of overhead. I confirmed this adding in a timing using 
date
inside the script in the 1st line and just before the exec at the bottom:
Copy code
$ ./pants -V
.145963213
2.14.0
So that's a surprising - to me - early win. I basically write off bash as very fast in my head, but 150ms is alot.
Of course the real comparison is this - getting the bash out of the way. Still a tiny bit faster and I'm not really sure why:
Copy code
$ hyperfine -w2 '~/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/bin/pants -V' './scie-pants -V'
Benchmark 1: ~/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/bin/pants -V
  Time (mean ± σ):     500.3 ms ±  30.2 ms    [User: 439.2 ms, System: 12.2 ms]
  Range (min … max):   483.2 ms … 584.7 ms    10 runs

Benchmark 2: ./scie-pants -V
  Time (mean ± σ):     455.3 ms ±   4.6 ms    [User: 398.3 ms, System: 17.2 ms]
  Range (min … max):   448.5 ms … 463.7 ms    10 runs

Summary
  './scie-pants -V' ran
    1.10 ± 0.07 times faster than '~/.cache/pants/setup/bootstrap-Linux-x86_64/2.14.0_py39/bin/pants -V'
w
Your Python installation has never been so simple. And fast: 
pyoxy
should be a few milliseconds faster to initialize a Python interpreter mostly because of 
oxidized_importer
and it avoiding filesystem overhead to look for and load 
.py[c]
files.
https://gregoryszorc.com/blog/2022/05/10/announcing-the-pyoxy-python-runner/ 🤷
Though I wouldn't expect a gap that big
Oh wait, we're using python-standalone, not the pyoxy version - correct?
e
Correct.
👍 1
h
150ms in 
pants
is surprising to me
w
150ms in 
pants
is surprising to me
Kinda tracks with what I saw when messing around with shell completions (trying to see if dynamic completions made sense). It's like 60ms just to run this 
python="$(determine_python_exe "${pants_version}")"
to return my homebrew python3.9 instance, 20ms to 
determine_pants_version
, and then 40 on 
bootstrap_pants
(trying to remember numbers off the top of my head) Mac mini, 6-core i5
And looking at my notes, I noted that all the bootstrap time on an already bootstrapped system was in: 
python_major_minor_version="$(get_python_major_minor_version "${python}")"
- which I thought could be cached from the 
determine_python_exe
step.
h
Oh yeah we’re running python extra times in there. God, all the more reason to bundle an interpreter.
➕ 1
w
i would be in favor of having 
scie
in the 
pantsbuild
org (although obviously John reserves the right to put it somewhere else), and in favor of trying to use it for Pants. as he pointed out: it’s not a ton of code, and what it does is massively simpler than what PEX needs to do. so i suspect that we could all come up to speed on it fairly quickly.
➕ 2
a
At the very least, I think Pants is a good test case, and having it supported as an experimental distribution of Pants would be a net good
➕ 1
e
Taking @bitter-ability-32190's concerns to heart nonetheless, it seems to me the person to package Pants as a scie, re-work the ./pants script to use it or create a new alternate script etc is exactly not me.
I'm happy to do it, have the time, etc. But the bus factor has been raised. The question is - are we serious about that concern?
b
I think, like anything, we weigh pros/cons. And it seems (from what you/others have said) that in this case it's a huge win for what amounts to not much code/cognitive addition. So feel free to simply note the concern and move forward?
e
Your the one to answer your question mark since you raised it. But it sounds like you have. I'll do so then. I do think your concern is warranted though and encourage folks to contribute. Its a small codebase with issues documented, etc.
✅ 1
h
I think packaging Pants via scie is a great way to kick the tires on it before ironing out the details of having Pants build these … what are we calling the resulting files?
a
scies
h
Ah, the puns
a
“scie” is the artefact, “scie-jump” is the tool
w
(acai) berries?
h
don’t get too far over your scies
e
Like Pants it is very literal. The binary is the self contained interpreted executable. The scie-jump launches the scie by exploding it into the ~/.nce making it a non-compact executable.
a
pants hasn’t been literal for 10 years 😉
e
This likely wont be in 10 years either.
🙃 2
I'm most chuffed by --single-lift-line. Again super literal, but every word of it true.
And interpretable 2 ways, the second being my favorite ski area all time:

https://perk.uno/go/gogo.jpg▾

Alrighty, the scie-pants is up for review finally: https://github.com/pantsbuild/scie-pants/pull/1 I added folks who've been most recently involved / expressed interest, but happy for anyone who cares to dig in to do so.
👍 1
2 Views
Open in Slack
PreviousNext
Powered by