https://pantsbuild.org/ logo
#development
Title
# development
a

ancient-vegetable-10556

08/11/2022, 7:49 PM
My PyOxidizer branch now loads plugins that can be resolved by
pip
.
ox.py
is wild. https://github.com/pantsbuild/pants/pull/16484
❤️ 1
2
🤯 3
h

hundreds-father-404

08/11/2022, 7:52 PM
omg and those plugins can now talk properly to Pants?! exciting!
a

ancient-vegetable-10556

08/11/2022, 7:53 PM
it’s in Pants
❤️ 1
THE PLUGIN WAS COMING FROM INSIDE PANTS
👀 1
b

bitter-ability-32190

08/11/2022, 7:53 PM
You're crazy in the best of ways. This is so cool
Blog post! Blog post! 😈
💯 1
a

ancient-vegetable-10556

08/11/2022, 8:04 PM
EVENTUALLY
2
@wide-midnight-78598 If you get a chance, could you take a look at my branch again and see if it pulls in your plugins? I didn’t get a chance to test in-repo plugins, but it’s also not clear to me how they differ from pip-specified plugins
(for obvious reasons, plugin interpreter constraints are going to be a bad idea)
b

bitter-ability-32190

08/11/2022, 10:12 PM
In repo plugins are no different than pants built-in plugins, since they're just chilling on the PYTHONPATH
a

ancient-vegetable-10556

08/11/2022, 10:23 PM
May I direct you to all of the patching of the import machinery that I needed to do to make the PIP resolution work?
(so yeah, they should be the same, but I have no idea until we test the relevant paths)
w

wide-midnight-78598

08/11/2022, 10:45 PM
@ancient-vegetable-10556 👍 Will do tonight
Aside - maybe this is a good time to upgrade to 0.22? https://github.com/indygreg/PyOxidizer/releases/tag/pyoxidizer%2F0.22.0
Okay, two notes: https://github.com/sureshjoshi/pants-plugins/blob/e9ffe21d0efbdc5b09b37c5e870a89ba6d663a64/pants.toml Running in a folder with pants.toml at 2.13.0.dev5 - I think this is both expected and good.
Copy code
Traceback (most recent call last):
  File "pants.engine.internals.options_parsing", line 48, in parse_options
    return _Options(options_bootstrapper, build_config)
  File "<string>", line 5, in __init__
  File "pants.engine.internals.options_parsing", line 36, in __post_init__
    assert self.options is not None
  File "pants.util.memo", line 123, in memoize
    result = func(*args, **kwargs)
  File "pants.engine.internals.options_parsing", line 40, in options
    return self.options_bootstrapper.full_options(self.build_config)
  File "pants.option.options_bootstrapper", line 282, in full_options
    raise BuildConfigurationError(
pants.base.exceptions.BuildConfigurationError: Version mismatch: Requested version was 2.13.0.dev5, our version is 2.14.0.dev5.
Running against my pants.toml after upgrading the version:
Copy code
Traceback (most recent call last):
  File "pants.bin.daemon_pants_runner", line 130, in single_daemonized_run
    scheduler, options_initializer = self._core.prepare(options_bootstrapper, complete_env)
  File "pants.pantsd.pants_daemon_core", line 129, in prepare
    build_config, options = self._options_initializer.build_config_and_options(
  File "pants.init.options_initializer", line 107, in build_config_and_options
    build_config = _initialize_build_configuration(
  File "pants.init.options_initializer", line 56, in _initialize_build_configuration
    return load_backends_and_plugins(
  File "pants.init.extension_loader", line 45, in load_backends_and_plugins
    load_build_configuration_from_source(bc_builder, backends)
  File "pants.init.extension_loader", line 127, in load_build_configuration_from_source
    load_backend(build_configuration, backend_package)
  File "pants.init.extension_loader", line 144, in load_backend
    raise BackendConfigurationError(f"Failed to load the {backend_module} backend: {ex!r}")
pants.base.exceptions.BackendConfigurationError: Failed to load the experimental.ansible.register backend: ModuleNotFoundError("No module named 'experimental'")
a

ancient-vegetable-10556

08/12/2022, 4:48 PM
OK, I haven’t gone looking at loading in-repo backends yet
I guess that’ll be next.
w

witty-crayon-22786

08/15/2022, 7:04 PM
nice! one thing to note is that this will likely collide with https://github.com/pantsbuild/pants/pull/16467 : was it necessary to use
pip
directly rather than
pex
? the former won’t have lockfiles.
a

ancient-vegetable-10556

08/17/2022, 7:53 PM
@witty-crayon-22786 It all uses pex under the hood, which is important because we’re moving to a world where there’s a chance of not having a Python environment. I recommend reading
ox.py
in a space where you feel safe weeping and/or cursing the void
👍 1
@wide-midnight-78598 I appear to be able to – at the very least – load the custom in-repo backends in your
pants-plugins
branch
❤️ 2
💥 1
(as of changes pushed less than 5 mins ago)
again, please let me know if anything is missing, otherwise I’m going to throw the branch over the fence and get to work on the distribution side of things
🙌 1
w

wide-midnight-78598

08/25/2022, 9:32 PM
I can test on the run-my-pants branch?
a

ancient-vegetable-10556

08/25/2022, 9:32 PM
Yeah, that should now be up-to-date
w

wide-midnight-78598

08/25/2022, 9:33 PM
Spicy - I'll try it out again tonight
b

bitter-ability-32190

08/25/2022, 9:35 PM
Remind me the instructions
w

wide-midnight-78598

08/25/2022, 9:38 PM
Note about possibly not needing the triplet
a

ancient-vegetable-10556

08/25/2022, 9:38 PM
yeah, I don’t think you do
I’ll update that with something minimal that works momentarily
b

bitter-ability-32190

08/25/2022, 9:39 PM
I'll try tomorrow if I get time
👍 1
w

wide-midnight-78598

08/26/2022, 12:23 AM
Just compiling this all right now, but quick question - if this is the distribution mechanism, would the idea be that your
pants.toml
specifies a version, and the pants script downloads that from github? Or, is the idea more of a versioned binary that would then define the version to use (which is chicken, and which is egg?)
h

hundreds-father-404

08/26/2022, 12:25 AM
My understanding is that you define in
pants.toml
and
./pants
script pulls it down for you. It's important to us that you can have multiple repos w/ different Pants versions. We don't want something like Bazel where it's a global binary for everything If we do have a global binary, which would be awesome, it would need to still support this per-repo config
w

wide-midnight-78598

08/26/2022, 12:27 AM
Yep, okay, that's how I imagined it - just wanted to confirm. building this one specifically requires that I change my pants.toml, which is fine, but it got me wondering about the longer-term distribution strat
👍 1
Hmmmmm
Copy code
ERROR: pantsbuild.pants-2.14.0a0-cp38-cp38-macosx_12_0_x86_64.whl is not a supported wheel on this platform.
error[PYOXIDIZER_PYTHON_EXECUTABLE]: calling pip install

Caused by:
    command ["/private/var/folders/18/q1r7phps28nc9rx5j_0t3jmm0000gp/T/pants-sandbox-2zLVHa/.cache/pyoxidizer/python_distributions/python.4a39fa15024b/python/install/bin/python3.9", "-m", "pip", "--disable-pip-version-check", "install", "--target", "/tmp/pyoxidizer-pip-installnH1II9/install", "pantsbuild.pants-2.14.0a0-cp38-cp38-macosx_12_0_x86_64.whl"] exited with code 1
  --> ./pyoxidizer.bzl:21:30
   |
21 |     exe.add_python_resources(exe.pip_install(['pantsbuild.pants-2.14.0a0-cp38-cp38-macosx_12_0_x86_64.whl']))
   |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PythonExecutable.pip_install()
Sweet, I switched to a different computer and it picked up the ansible lint local plugin!
🙌 1
❤️ 2
h

hundreds-father-404

08/26/2022, 1:53 AM
I wonder what's up with the original error tho; maybe that the interpreter constraints in Chris's Readme are set to ==3.9?
w

wide-midnight-78598

08/26/2022, 1:54 AM
It tried to grab 3.8, which was odd, and I don't have 3.8 on one of my machines. Never have, never will.
👍 1
I passed in 3.9 constraints, but I'm sure I just messed something up. Since it was just to test in-repo plugins, I just switched machines. Much faster than debuggin 🙂
a

ancient-vegetable-10556

08/26/2022, 3:04 PM
Right, Pants itself needs to run with the same interpreter constraints as the version that you’re specifying with PyOxidizer
w

wide-midnight-78598

08/26/2022, 3:25 PM
Does that mean we'll need a matrix of binary distributions per Pants version?
a

ancient-vegetable-10556

08/26/2022, 3:25 PM
Well, no! We only need to pick whatever Python version the Pants team thinks is best
h

hundreds-father-404

08/26/2022, 3:25 PM
@NewestStableRelease
a

ancient-vegetable-10556

08/26/2022, 3:26 PM
It’ll be stricter for people with custom backends or plugin users, but they’ll always be able to fallback on the external python distribution approach that we currently use
w

wide-midnight-78598

08/26/2022, 3:26 PM
Ah, okay, sorry, I confused myself just now. I was building PyOx at the time, not running it 🤦‍♂️
a

ancient-vegetable-10556

08/26/2022, 3:27 PM
but for the vast vast majority of people, we only need the one distribution of Pants and nobody should even need to know that there’s Python involved
h

hundreds-father-404

08/26/2022, 3:28 PM
but they’ll always be able to fallback on the external python distribution approach that we currently use
Well, TBD. I still propose that we don't need to give as much of a range as people think, specifically because of
[python].resolves
. It robustly supports your own code being Py37 but Pants plugin being Py310, for example. I think it could be viable to only release for one version, which has several benefits like allowing us to use newest features & reducing risk we write code broken on a certain version
w

wide-midnight-78598

08/26/2022, 3:29 PM
Will there be any limitations incurred by PyOx itself? https://pyoxidizer.readthedocs.io/en/stable/pyoxidizer_faq.html#faq-python-38
a

ancient-vegetable-10556

08/26/2022, 3:29 PM
Yeah, we will have to distribute against a Python version that PyOx supports
w

wide-midnight-78598

08/26/2022, 3:31 PM
Okay, so, let's say I'm an everyday normal user. And let's say that PyOx'd Pants is 3.10 I have in-repo plugins, and then my regular python production code. Will that PyOx'd 3.10 impose any constraints on an existing repo?
a

ancient-vegetable-10556

08/26/2022, 3:31 PM
Only for your in-repo plugins
h

hundreds-father-404

08/26/2022, 3:31 PM
Will that PyOx'd 3.10 impose any constraints on an existing repo?
You would set up
[python].resolves
for your pants-plugin, which we strongly encourage anyways. If you do that, zero impact to your own code
a

ancient-vegetable-10556

08/26/2022, 3:33 PM
To be clear, the PyOxed pants binary still invokes an external Python environment for everything except things embedded into Pants itself. So if you run
./pants test
, it will find a Python on your system that matches your pants interpreter constraints
👍 2
w

wide-midnight-78598

08/26/2022, 3:33 PM
Okay, great - I'm jumping around between multiple clients code, my own in-repo plugins, and then mainline pants code, that I can barely keep track of what should be where and why
1
it will find a Python on your system that matches your pants interpreter constraints
Yes, okay, great, that's what I thought/hoped.
a

ancient-vegetable-10556

08/26/2022, 3:34 PM
the difference is that Pants itself will no longer run against the python version specified by global interpreter constraints
(and transitively, plugin packages and your in-repo backends)
h

hundreds-father-404

08/26/2022, 3:35 PM
the difference is that Pants itself will no longer run against the python version specified by global interpreter constraints
Well, that was never a thing. What Pants itself runs on is solely determined by the
./pants
bash script. It does not look at your global interpreter constraints at all Except when you are running Pants on Pants plugins like
./pants test
, where Pants forks a new process like any other build. Then, indeed, for the forked process, it looks at interpreter constraints option. Which is where
[python].resolves
comes into play
w

wide-midnight-78598

08/26/2022, 3:36 PM
The mental model I'm going for is that Pants is a black box, and however it runs internally - good for it, no external dependencies that I need to think about.
1
a

ancient-vegetable-10556

08/26/2022, 3:37 PM
Yes, except for those that you specify yourself
👍 1
and then hopefully there is nothing to think about
h

hundreds-father-404

08/26/2022, 3:37 PM
Agreed, and this PyOxy work makes that black box much more hermetic. Right now we leak by requiring you to have a Py37-39 interpreter
1
a

ancient-vegetable-10556

08/26/2022, 3:38 PM
the external dependencies thing will end up being far more nuanced than it is now, but the people for whom that matters will now be restricted just to people who have a good understanding of Pants
1
as opposed to the current state of every-…-body
h

hundreds-father-404

08/26/2022, 3:48 PM
as long as we got config to copy-pasta for plugin authors for
[python].resolves
, I think that's pretty feasible.