I m seeing odd behavior of Pants on `main` apparently not be Pants #development

I'm seeing odd behavior of Pants (on `main`) appar...

bitter-ability-32190

08/31/2023, 4:14 PM

I'm seeing odd behavior of Pants (on

main

) apparently not being a namespace package? So I'm trying to run

pants

from the latest release in

main

, so I set a

pants_version

and removed

[DEFAULT]

. Then

pants

runs, and errors on not being able to find

pants.explorer.server

. That makes sense, since it isn't bundled into Pants itself. So I add

"%(buildroot)s/src/python"

pythonpath

. But the error persists! Doing a little debugging, in

extension_loader

I do

import pants;print(pants.__file__)

and it prints

/home/josh/.cache/nce/65aa4f2a6c1f9bac672c0df94ae34c7170e5c071cda35e9b725945831905c122/bindings/venvs/2.18.0a0/lib/python3.9/site-packages/pants/__init__.py

. According to the internet, namespace packages should have

__file__

set to

None

(or just not have a

__file__

. WTF?!

bitter-ability-32190

08/31/2023, 4:25 PM

Hmmmm I'm missing something for sure...

Copy code

josh@cephandrius:~/work/pants$ mkdir -p other_pants/pants
josh@cephandrius:~/work/pants$ cp src/python/pants/__init__.py other_pants/pants
josh@cephandrius:~/work/pants$ touch other_pants/pants/bar.py
josh@cephandrius:~/work/pants$ python3.9
Python 3.9.17 (main, Jun  6 2023, 20:11:04) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append("src/python")
>>> sys.path.append("other_pants")
>>> import pants
>>> pants.__file__
'/home/josh/work/pants/src/python/pants/__init__.py'

bitter-ability-32190

08/31/2023, 4:34 PM

Although...

Copy code

>>> import sys; sys.path.append("src/python"); sys.path.append("other_pants")
>>> import pants.bar
>>> pants
<module 'pants' from '/home/josh/work/pants/src/python/pants/__init__.py'>
>>> import pants.util
>>> pants
<module 'pants' from '/home/josh/work/pants/src/python/pants/__init__.py'>

So I think

__file__

is a red herring?

bitter-ability-32190

08/31/2023, 4:38 PM

Ah I think I see... Yes

__file__

is a red herring. The true way to tell how things will be loaded is to look at

__path__

. In this case

pants

is loaded first to start, well, executing Pants. And in doing so,

pythonpath

from

pants.toml

isn't considered. That means the built-in Pants is marked down as the only path in

__path__

. To put this back into `python3.9`:

Copy code

josh@cephandrius:~/work/pants$ python3.9
Python 3.9.17 (main, Jun  6 2023, 20:11:04) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append("src/python")
>>> import pants.util
>>> pants.__path__
['/home/josh/work/pants/src/python/pants']
>>> sys.path.append("other_pants")
>>> import pants.bar
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pants.bar'

bitter-ability-32190

08/31/2023, 4:53 PM

So, what now? Well, it'd be nice to test if we're squatting on the

pants

name for plugins. Eventually I think we want another repo for blessed-but-not-baked-in plugins. This particular codepath isn't for plugins though. Secondly, I think our

backend_packages

should only be packages baked into Pants itself. Anything not baked in belongs in

internal_plugins

. @curved-television-6568 thoughts?

bitter-ability-32190

08/31/2023, 5:19 PM

OK I have the ability to add a test which tests this fundamental assumption. So that's good 🙂

curved-television-6568

08/31/2023, 5:35 PM

yea, so then

pants.explorer

should move out, as that is not meant to be baked in.

✅ 1

curved-television-6568

08/31/2023, 5:36 PM

but I don’t think it belongs in

internal_plugins

either.. it’s meant to become a public plugin

bitter-ability-32190

08/31/2023, 5:39 PM

Yeah I think that's where we need to figure out how/where we should put external plugins that aren't in our backend

✅ 1

curved-television-6568

08/31/2023, 5:40 PM

pants_plugin.explorer

bitter-ability-32190

08/31/2023, 5:42 PM

Maybe we discuss at the next meeting?

👍 1

bitter-ability-32190

08/31/2023, 5:42 PM

Or we sidecar on the third party "plugins" discussion

curved-television-6568

08/31/2023, 5:43 PM

kind of related I think.. so yea that makes sense

bitter-ability-32190

08/31/2023, 5:43 PM

I have some thoughts on what we can do for thirdparty plugins.

bitter-ability-32190

08/31/2023, 5:44 PM

E.g. • How should they be tested? • How should they easily support (and test) multiple Pants versions? • How shoudl they be catalogued? • Can we provide infra/tech for docs hosting of them?

curved-television-6568

08/31/2023, 5:46 PM

oh I love all those bullets

bitter-ability-32190

08/31/2023, 5:47 PM

While I'm spitballing: • How can we help plugin resolution (e.g. deps of plugins) • How can we help ensure plugins have 0 sdist dependencies ◦ We already broke this one, lol

curved-television-6568

08/31/2023, 5:50 PM

help plugin deps is done, or at least started: https://github.com/pantsbuild/pants/pull/19406

bitter-ability-32190

08/31/2023, 5:51 PM

I think we need to take it farther, honestly.

bitter-ability-32190

08/31/2023, 5:51 PM

Like, Pex lockfile far 🙂

curved-television-6568

08/31/2023, 5:51 PM

cool (so, started 😜 )

curved-television-6568

08/31/2023, 5:53 PM

buut.. if it’s an external plugin, you have your normal dist requirements.. aren’t they enough?

curved-television-6568

08/31/2023, 5:53 PM

so, it would be for the baked backends then..?

bitter-ability-32190

08/31/2023, 5:54 PM

No, because those can float in transitivity. meaning your

pants

is not the same as my

pants

in the same repo for the same commit

curved-television-6568

08/31/2023, 5:55 PM

but it sounds risky if a plugin puts a hard pin, that can get real wieldy in compat issues between multiple plugins. Feels like if we want two installs to be identical we need to share a common constraints.txt or similar.

bitter-ability-32190

08/31/2023, 5:55 PM

Thats why I'm thinking plugins are resolved from a single Pex lockfile 🙂

curved-television-6568

08/31/2023, 5:56 PM

curious how you’d see this working in practice (given multiple potential sources for plugins)

bitter-ability-32190

08/31/2023, 5:56 PM

what do you mean multiple sources?

curved-television-6568

08/31/2023, 5:56 PM

from different orgs, for instance

bitter-ability-32190

08/31/2023, 5:57 PM

One lockfile with multiple inputs?

curved-television-6568

08/31/2023, 5:57 PM

do you generate your own lockfile then, for the set of plugins you use?

bitter-ability-32190

08/31/2023, 5:57 PM

Yeah

curved-television-6568

08/31/2023, 5:58 PM

ok, so that would be the file you can share.. I called it “constraints.txt” but 🥔 🍅

bitter-ability-32190

08/31/2023, 5:58 PM

...I should dump this into an issue, if there already isn't one

👍 1

bitter-ability-32190

08/31/2023, 5:58 PM

Also, plugins are resolved through Pex, so it's the most natural fit, IMHO

💯 1

curved-television-6568

08/31/2023, 5:59 PM

what?

curved-television-6568

08/31/2023, 6:00 PM

oh, I see

bitter-ability-32190

08/31/2023, 6:00 PM

https://github.com/pantsbuild/pants/blob/d3ba1cb47ac00464ae4d6ee34b6b6a818d7b8b9a/src/python/pants/init/plugin_resolver.py#L77

curved-television-6568

08/31/2023, 6:00 PM

yea, was just looking at those lines 😅

Open in Slack

Previous Next