https://pantsbuild.org/ logo
Join Slack
Powered by
My PyOxidizer branch now loads plugins that can be...
# development
a
My PyOxidizer branch now loads plugins that can be resolved by 
pip
. 
ox.py
is wild. https://github.com/pantsbuild/pants/pull/16484
❤️ 1
❗ 2
🤯 3
h
omg and those plugins can now talk properly to Pants?! exciting!
a
it’s in Pants
❤️ 1
THE PLUGIN WAS COMING FROM INSIDE PANTS
👀 1
b
You're crazy in the best of ways. This is so cool
Blog post! Blog post! 😈
💯 1
a
EVENTUALLY
⏳ 2
@wide-midnight-78598 If you get a chance, could you take a look at my branch again and see if it pulls in your plugins? I didn’t get a chance to test in-repo plugins, but it’s also not clear to me how they differ from pip-specified plugins
(for obvious reasons, plugin interpreter constraints are going to be a bad idea)
b
In repo plugins are no different than pants built-in plugins, since they're just chilling on the PYTHONPATH
a
May I direct you to all of the patching of the import machinery that I needed to do to make the PIP resolution work?
(so yeah, they should be the same, but I have no idea until we test the relevant paths)
w
@ancient-vegetable-10556 👍 Will do tonight
Aside - maybe this is a good time to upgrade to 0.22? https://github.com/indygreg/PyOxidizer/releases/tag/pyoxidizer%2F0.22.0
Okay, two notes: https://github.com/sureshjoshi/pants-plugins/blob/e9ffe21d0efbdc5b09b37c5e870a89ba6d663a64/pants.toml Running in a folder with pants.toml at 2.13.0.dev5 - I think this is both expected and good.
Copy code
Traceback (most recent call last):
  File "pants.engine.internals.options_parsing", line 48, in parse_options
    return _Options(options_bootstrapper, build_config)
  File "<string>", line 5, in __init__
  File "pants.engine.internals.options_parsing", line 36, in __post_init__
    assert self.options is not None
  File "pants.util.memo", line 123, in memoize
    result = func(*args, **kwargs)
  File "pants.engine.internals.options_parsing", line 40, in options
    return self.options_bootstrapper.full_options(self.build_config)
  File "pants.option.options_bootstrapper", line 282, in full_options
    raise BuildConfigurationError(
pants.base.exceptions.BuildConfigurationError: Version mismatch: Requested version was 2.13.0.dev5, our version is 2.14.0.dev5.
Running against my pants.toml after upgrading the version:
Copy code
Traceback (most recent call last):
  File "pants.bin.daemon_pants_runner", line 130, in single_daemonized_run
    scheduler, options_initializer = self._core.prepare(options_bootstrapper, complete_env)
  File "pants.pantsd.pants_daemon_core", line 129, in prepare
    build_config, options = self._options_initializer.build_config_and_options(
  File "pants.init.options_initializer", line 107, in build_config_and_options
    build_config = _initialize_build_configuration(
  File "pants.init.options_initializer", line 56, in _initialize_build_configuration
    return load_backends_and_plugins(
  File "pants.init.extension_loader", line 45, in load_backends_and_plugins
    load_build_configuration_from_source(bc_builder, backends)
  File "pants.init.extension_loader", line 127, in load_build_configuration_from_source
    load_backend(build_configuration, backend_package)
  File "pants.init.extension_loader", line 144, in load_backend
    raise BackendConfigurationError(f"Failed to load the {backend_module} backend: {ex!r}")
pants.base.exceptions.BackendConfigurationError: Failed to load the experimental.ansible.register backend: ModuleNotFoundError("No module named 'experimental'")
a
OK, I haven’t gone looking at loading in-repo backends yet
I guess that’ll be next.
w
nice! one thing to note is that this will likely collide with https://github.com/pantsbuild/pants/pull/16467 : was it necessary to use 
pip
directly rather than 
pex
? the former won’t have lockfiles.
a
@witty-crayon-22786 It all uses pex under the hood, which is important because we’re moving to a world where there’s a chance of not having a Python environment. I recommend reading 
ox.py
in a space where you feel safe weeping and/or cursing the void
👍 1
@wide-midnight-78598 I appear to be able to – at the very least – load the custom in-repo backends in your 
pants-plugins
branch
❤️ 2
💥 1
(as of changes pushed less than 5 mins ago)
again, please let me know if anything is missing, otherwise I’m going to throw the branch over the fence and get to work on the distribution side of things
🙌 1
w
I can test on the run-my-pants branch?
a
Yeah, that should now be up-to-date
w
Spicy - I'll try it out again tonight
b
Remind me the instructions
a
w
Note about possibly not needing the triplet
a
yeah, I don’t think you do
I’ll update that with something minimal that works momentarily
b
I'll try tomorrow if I get time
👍 1
w
Just compiling this all right now, but quick question - if this is the distribution mechanism, would the idea be that your 
pants.toml
specifies a version, and the pants script downloads that from github? Or, is the idea more of a versioned binary that would then define the version to use (which is chicken, and which is egg?)
h
My understanding is that you define in 
pants.toml
and 
./pants
script pulls it down for you. It's important to us that you can have multiple repos w/ different Pants versions. We don't want something like Bazel where it's a global binary for everything If we do have a global binary, which would be awesome, it would need to still support this per-repo config
w
Yep, okay, that's how I imagined it - just wanted to confirm. building this one specifically requires that I change my pants.toml, which is fine, but it got me wondering about the longer-term distribution strat
👍 1
Hmmmmm
Copy code
ERROR: pantsbuild.pants-2.14.0a0-cp38-cp38-macosx_12_0_x86_64.whl is not a supported wheel on this platform.
error[PYOXIDIZER_PYTHON_EXECUTABLE]: calling pip install

Caused by:
    command ["/private/var/folders/18/q1r7phps28nc9rx5j_0t3jmm0000gp/T/pants-sandbox-2zLVHa/.cache/pyoxidizer/python_distributions/python.4a39fa15024b/python/install/bin/python3.9", "-m", "pip", "--disable-pip-version-check", "install", "--target", "/tmp/pyoxidizer-pip-installnH1II9/install", "pantsbuild.pants-2.14.0a0-cp38-cp38-macosx_12_0_x86_64.whl"] exited with code 1
  --> ./pyoxidizer.bzl:21:30
   |
21 |     exe.add_python_resources(exe.pip_install(['pantsbuild.pants-2.14.0a0-cp38-cp38-macosx_12_0_x86_64.whl']))
   |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PythonExecutable.pip_install()
Sweet, I switched to a different computer and it picked up the ansible lint local plugin!
🙌 1
❤️ 2
h
I wonder what's up with the original error tho; maybe that the interpreter constraints in Chris's Readme are set to ==3.9?
w
It tried to grab 3.8, which was odd, and I don't have 3.8 on one of my machines. Never have, never will.
👍 1
I passed in 3.9 constraints, but I'm sure I just messed something up. Since it was just to test in-repo plugins, I just switched machines. Much faster than debuggin 🙂
a
Right, Pants itself needs to run with the same interpreter constraints as the version that you’re specifying with PyOxidizer
w
Does that mean we'll need a matrix of binary distributions per Pants version?
a
Well, no! We only need to pick whatever Python version the Pants team thinks is best
h
@NewestStableRelease
a
It’ll be stricter for people with custom backends or plugin users, but they’ll always be able to fallback on the external python distribution approach that we currently use
w
Ah, okay, sorry, I confused myself just now. I was building PyOx at the time, not running it 🤦‍♂️
a
but for the vast vast majority of people, we only need the one distribution of Pants and nobody should even need to know that there’s Python involved
h
but they’ll always be able to fallback on the external python distribution approach that we currently use
Well, TBD. I still propose that we don't need to give as much of a range as people think, specifically because of 
[python].resolves
. It robustly supports your own code being Py37 but Pants plugin being Py310, for example. I think it could be viable to only release for one version, which has several benefits like allowing us to use newest features & reducing risk we write code broken on a certain version
w
Will there be any limitations incurred by PyOx itself? https://pyoxidizer.readthedocs.io/en/stable/pyoxidizer_faq.html#faq-python-38
a
Yeah, we will have to distribute against a Python version that PyOx supports
w
Okay, so, let's say I'm an everyday normal user. And let's say that PyOx'd Pants is 3.10 I have in-repo plugins, and then my regular python production code. Will that PyOx'd 3.10 impose any constraints on an existing repo?
a
Only for your in-repo plugins
h
Will that PyOx'd 3.10 impose any constraints on an existing repo?
You would set up 
[python].resolves
for your pants-plugin, which we strongly encourage anyways. If you do that, zero impact to your own code
a
To be clear, the PyOxed pants binary still invokes an external Python environment for everything except things embedded into Pants itself. So if you run 
./pants test
, it will find a Python on your system that matches your pants interpreter constraints
👍 2
w
Okay, great - I'm jumping around between multiple clients code, my own in-repo plugins, and then mainline pants code, that I can barely keep track of what should be where and why
➕ 1
it will find a Python on your system that matches your pants interpreter constraints
Yes, okay, great, that's what I thought/hoped.
a
the difference is that Pants itself will no longer run against the python version specified by global interpreter constraints
(and transitively, plugin packages and your in-repo backends)
h
the difference is that Pants itself will no longer run against the python version specified by global interpreter constraints
Well, that was never a thing. What Pants itself runs on is solely determined by the 
./pants
bash script. It does not look at your global interpreter constraints at all Except when you are running Pants on Pants plugins like 
./pants test
, where Pants forks a new process like any other build. Then, indeed, for the forked process, it looks at interpreter constraints option. Which is where 
[python].resolves
comes into play
w
The mental model I'm going for is that Pants is a black box, and however it runs internally - good for it, no external dependencies that I need to think about.
➕ 1
a
Yes, except for those that you specify yourself
👍 1
and then hopefully there is nothing to think about
h
Agreed, and this PyOxy work makes that black box much more hermetic. Right now we leak by requiring you to have a Py37-39 interpreter
➕ 1
a
the external dependencies thing will end up being far more nuanced than it is now, but the people for whom that matters will now be restricted just to people who have a good understanding of Pants
➕ 1
as opposed to the current state of every-…-body
h
as long as we got config to copy-pasta for plugin authors for 
[python].resolves
, I think that's pretty feasible.
2 Views
Open in Slack
PreviousNext
Powered by