https://pantsbuild.org/ logo
Join Slack
Powered by
Apropos of Redox above, if we can get to the point...
# development
e
Apropos of Redox above, if we can get to the point of embedding interpreters then I think we can just freely pick a version.
🔥 1
w
yes.
would be incredible. but... not small
...self assigned yesterday?!? whoa. @hundreds-breakfast-49010 what did i miss?
h
@witty-crayon-22786 I've spent some time looking into how tractable this might be, in the context of improving pants command-line latency
so far all I've done is play around with pyoxidizer some, cf. danny's suggestion that it might be a viable way of embeddeding a python interpreter
to be clear what toolchain cares about right now is "make CLI latency as small as possible", not specifically "make the pants entry point be rust rather than python"
w
it's not 100% clear to me how much faster rust code wrapping python options parsing would be... probably a fair amount? ... but not clear.
but certainly an interesting experiment... you could do something like extracting exactly bootstrap options parsing (and nothing else), and that would be enough
h
we were talking the other day about the possibility that, if we're running pants with pantsd, we need to do very little work between initial program startup and sending a nailgun message to pantsd
w
yes. just bootstrap options parsing, effectively.
h
I was originally thinking that I'd write a small rust utility that did this small amount of work, and hten figure out how to hook that into the flow from the user running 
./pants
on the command line
w
but the thing is: it's possible that if you went "pure" rust client, you could just hack/reimplement enough of bootstrap options parsing to avoid actually embedding python
h
I'm not sure if it makes sense to use pyoxidizer right this second. I still don't think I understand the tool super-thoroughly, but I also haven't been looking at it all that long
right
w
because with regard to 
--enable-pantsd
itself, the hack would be small.
h
so maybe what we actually want is to write some rust code that does bootstrap options parsing (and also use this everywhere in python we use 
BootstrapOptions
, which would reduce code duplication)
e
Using the python interpreter for any of the client operations seems like a non-starter.
w
yea.
h
and then use that canonical rust implementation of the bootstrap options in the small rust utility that talks to pantsd
w
@enough-analyst-54434: well, i think it's a question of how quickly you want to get there... the advantage to "pyoxidizer for all of pants" is that it gets us to a much simpler distibution story, while simultaneously unlocking some more incremental wins in this area
but yes... pure rust client with hacks to get exactly the 
enable_pantsd
flag and nothing else could be achieved much more quickly.
h
even that solution still needs some work wrt the distribution story
w
yes.
...well, ish.
h
right now the distribution story is that the user curl's a shell script called 
pants
and that shell script does a bunch of bootstrapping stuff with python, and I don't understand all the dteails of how that works
w
the pants script could unzip the binary out of a whl in the venv... ditto pex
yea, that part is not bad.
e
Yeah - I think we talked about all this yesterday.
w
(sorry)
missed some thread.
h
is there a reason to even distribute a shell script? maybe the 
pants
binary should be a compiled rust binary?
e
It would need to learn to setup pants for python 3.6 or 3.7 or 3.8 ...
w
@enough-analyst-54434: so, the relevance of #7369 at the top of this thread is that pyoxidizer can statically link a python
h
yeah, and that's definitely exciting
but stu's point about how using pyoxidizer is probably not trivial applies
w
(i think...? hope i'm not lying there)
e
Ack, but we were switched subjects to not using pyoxideizer unless I dropped the thread
h
I mean it definitely looks like the whoel point of pyoxidizer is to embed a python interpreter
w
yep.
@hundreds-breakfast-49010: yes. but it is still possible to do that dynamically
h
@witty-crayon-22786 dynamically based on what? if we were going to use an embedded python interpreter, I assume we'd pick one and use it everywhere
probably python 3.7 fornow
we wouldn't have to care at all what python is installed on any user's machine
w
static and dynamic in the native-dependency-linking sense.
anyway
h
@enough-analyst-54434 I don't think we need to use pyoxidizer yet, although I also think it's a cool medium-term goal. I do think that introducing a new rust entrypoint to pants would involve some thinking around the distribution story in a way similar to how switching to pyoxidizer would
w
i don't think so... that part is really easy.
e
Agreed - although what Stu said seems correct to me. Distribution story stays same. Those that use pants shell script have it extract rust binary, those that use pex - not sure.
But pants python code could have a built in 1 time run and extract relevant bit part.
w
we've got scripts too. would do something similar.
e
Pants python code already does this for the native.so.
h
so,maybe we'd provide a compiled-from-rust binary called I dunno 
pants_cli
, and in the 
pants
shell script we'd swap out the line 
exec "${pants_dir}/bin/python" "${pants_dir}/bin/pants" "$@"
for just 
./pants_cli
and then 
pants_cli
would have to figure out how to do the same stuff that that shell script does, in the case where we're not using pantsd
in terms of dealing with whl's and whatever else that script does
e
Well, you'd still let the shell scrit fully install pants. Then, do the binary extract.
w
John and Eric both have the most experience with that part
but yea. not a blocker if the experiment plays out.
h
so @enough-analyst-54434 are you imagining that we'd put the logic for extracting a native compiled binary from a whl in the 
bootstrap_pants
function?
w
@hundreds-breakfast-49010: what the pantsbuild/pants repo does is different from end user repos.
in pantsbuild/pants you could cheat and just literally 
cargo run $mybinary
from the pants script.
h
right, I'm looking at 
pants
as installed in a random test repo
I know the 
pants
in the pants repo is different
w
k
h
I'm actually not sure how the shell script we ask people to curl to install pants gets built
we ask people to curl
Copy code
<https://pantsbuild.github.io/setup/pants>
but I don't think this script exists verbatim anywhere in the pants git repo
w
yes. it's in a separate setup repo: github.com/pantsbuild/setup/
as mentioned above though, that is a wide, clearly lit bridge that we can cross when we come to it.
h
I'm not personally sure how I'd go about implementing the rust pants initializer without crossing this bridge, so to speak
w
three other people are, heh
what i'm saying is: when we have a tool that works in the pantsbuild/pants repo, using whatever hacks we want there, distributing it will not be an issue.
h
right now I'm trying to think through what all the steps would be to actually make running 
./pants <whatever>
at the command line start up a rust program that does the initialization work we currently do in python, with the expectation that I might write this and submit a PR next week. it's very possible that I'm missing something important that would make this easier to do or harder to do than I think it would, because I personally lack familiarity with something other people already know about
so I'm mostly asking these questions to make sure that I understand everything I need to in order to do this work, or at least be sure that I don't need to understand it to do this work
w
yep, understood. but John and i have both confirmed in this thread that that will not be a challenge relative to the rest of the project. it should literally be this line: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1584835986097700?thread_ts=1584805095.053900&amp;cid=C0D7TNJHL
and the intent there is to assure you that you do not need to worry about it. the other parts are the parts to focus on.
h
ok cool
so, that means that I need to make sure that I can build a rust binary that will get inserted into a whl that we distribute
w
yep
h
maybe this should be an entirely new crate within the pants repo, so I don't need to worry about making it work with the 
native_engine.so
lib we already build?
w
it should, yes. we have 15+ crates already... no need to link it to the 
engine
crate.
(a few of which are binaries)
h
so okay a new top-level crate is easy to make, and it's easy enough to make that spit out a binary. I'm not sure what the process of packaging that binary into a whl would look like, or how'd we'd thinking about building for specific platforms
if the binary was small enough it might even be tractable to have the pants bootstrap script compile it on the system of the user, but then they'd have to have rust as a dependency to use pants, which we might not want
hm no that's probably not a good idea
w
i believe that you can ignore the entire packaging problem until you have a working implementation that ignores all of it
we have per-platform wheels
so per-platform binaries inside of those wheels is not challenging.
i referenced literally using 
cargo run $binary
in the pantsbuild/pants repo because it would allow you to completely ignore packaging until the whole thing is working end to end
👍 1
(i tend to abuse the word "literally"... i think that the command would be more like 
build-support/bin/native/cargo run $binary
, in order to use our bootstrapped cargo and rust)
👍 1
Open in Slack
PreviousNext
Powered by