https://pantsbuild.org/ logo
Powered by
e

enough-analyst-54434

03/21/2020, 3:38 PM
Apropos of Redox above, if we can get to the point of embedding interpreters then I think we can just freely pick a version.
🔥 1
w

witty-crayon-22786

03/21/2020, 11:45 PM
yes.
would be incredible. but... not small
...self assigned yesterday?!? whoa. @hundreds-breakfast-49010 what did i miss?
h

hundreds-breakfast-49010

03/22/2020, 12:03 AM
@witty-crayon-22786 I've spent some time looking into how tractable this might be, in the context of improving pants command-line latency
so far all I've done is play around with pyoxidizer some, cf. danny's suggestion that it might be a viable way of embeddeding a python interpreter
to be clear what toolchain cares about right now is "make CLI latency as small as possible", not specifically "make the pants entry point be rust rather than python"
w

witty-crayon-22786

03/22/2020, 12:06 AM
it's not 100% clear to me how much faster rust code wrapping python options parsing would be... probably a fair amount? ... but not clear.
but certainly an interesting experiment... you could do something like extracting exactly bootstrap options parsing (and nothing else), and that would be enough
h

hundreds-breakfast-49010

03/22/2020, 12:07 AM
we were talking the other day about the possibility that, if we're running pants with pantsd, we need to do very little work between initial program startup and sending a nailgun message to pantsd
w

witty-crayon-22786

03/22/2020, 12:08 AM
yes. just bootstrap options parsing, effectively.
h

hundreds-breakfast-49010

03/22/2020, 12:08 AM
I was originally thinking that I'd write a small rust utility that did this small amount of work, and hten figure out how to hook that into the flow from the user running 
./pants
on the command line
w

witty-crayon-22786

03/22/2020, 12:08 AM
but the thing is: it's possible that if you went "pure" rust client, you could just hack/reimplement enough of bootstrap options parsing to avoid actually embedding python
h

hundreds-breakfast-49010

03/22/2020, 12:08 AM
I'm not sure if it makes sense to use pyoxidizer right this second. I still don't think I understand the tool super-thoroughly, but I also haven't been looking at it all that long
right
w

witty-crayon-22786

03/22/2020, 12:09 AM
because with regard to 
--enable-pantsd
itself, the hack would be small.
h

hundreds-breakfast-49010

03/22/2020, 12:09 AM
so maybe what we actually want is to write some rust code that does bootstrap options parsing (and also use this everywhere in python we use 
BootstrapOptions
, which would reduce code duplication)
e

enough-analyst-54434

03/22/2020, 12:10 AM
Using the python interpreter for any of the client operations seems like a non-starter.
w

witty-crayon-22786

03/22/2020, 12:10 AM
yea.
h

hundreds-breakfast-49010

03/22/2020, 12:10 AM
and then use that canonical rust implementation of the bootstrap options in the small rust utility that talks to pantsd
w

witty-crayon-22786

03/22/2020, 12:11 AM
@enough-analyst-54434: well, i think it's a question of how quickly you want to get there... the advantage to "pyoxidizer for all of pants" is that it gets us to a much simpler distibution story, while simultaneously unlocking some more incremental wins in this area
but yes... pure rust client with hacks to get exactly the 
enable_pantsd
flag and nothing else could be achieved much more quickly.
h

hundreds-breakfast-49010

03/22/2020, 12:12 AM
even that solution still needs some work wrt the distribution story
w

witty-crayon-22786

03/22/2020, 12:12 AM
yes.
...well, ish.
h

hundreds-breakfast-49010

03/22/2020, 12:12 AM
right now the distribution story is that the user curl's a shell script called 
pants
and that shell script does a bunch of bootstrapping stuff with python, and I don't understand all the dteails of how that works
w

witty-crayon-22786

03/22/2020, 12:13 AM
the pants script could unzip the binary out of a whl in the venv... ditto pex
yea, that part is not bad.
e

enough-analyst-54434

03/22/2020, 12:13 AM
Yeah - I think we talked about all this yesterday.
w

witty-crayon-22786

03/22/2020, 12:14 AM
(sorry)
missed some thread.
h

hundreds-breakfast-49010

03/22/2020, 12:14 AM
is there a reason to even distribute a shell script? maybe the 
pants
binary should be a compiled rust binary?
e

enough-analyst-54434

03/22/2020, 12:14 AM
It would need to learn to setup pants for python 3.6 or 3.7 or 3.8 ...
w

witty-crayon-22786

03/22/2020, 12:15 AM
@enough-analyst-54434: so, the relevance of #7369 at the top of this thread is that pyoxidizer can statically link a python
h

hundreds-breakfast-49010

03/22/2020, 12:15 AM
yeah, and that's definitely exciting
but stu's point about how using pyoxidizer is probably not trivial applies
w

witty-crayon-22786

03/22/2020, 12:16 AM
(i think...? hope i'm not lying there)
e

enough-analyst-54434

03/22/2020, 12:16 AM
Ack, but we were switched subjects to not using pyoxideizer unless I dropped the thread
h

hundreds-breakfast-49010

03/22/2020, 12:16 AM
I mean it definitely looks like the whoel point of pyoxidizer is to embed a python interpreter
w

witty-crayon-22786

03/22/2020, 12:16 AM
yep.
@hundreds-breakfast-49010: yes. but it is still possible to do that dynamically
h

hundreds-breakfast-49010

03/22/2020, 12:17 AM
@witty-crayon-22786 dynamically based on what? if we were going to use an embedded python interpreter, I assume we'd pick one and use it everywhere
probably python 3.7 fornow
we wouldn't have to care at all what python is installed on any user's machine
w

witty-crayon-22786

03/22/2020, 12:18 AM
static and dynamic in the native-dependency-linking sense.
anyway
h

hundreds-breakfast-49010

03/22/2020, 12:19 AM
@enough-analyst-54434 I don't think we need to use pyoxidizer yet, although I also think it's a cool medium-term goal. I do think that introducing a new rust entrypoint to pants would involve some thinking around the distribution story in a way similar to how switching to pyoxidizer would
w

witty-crayon-22786

03/22/2020, 12:20 AM
i don't think so... that part is really easy.
e

enough-analyst-54434

03/22/2020, 12:20 AM
Agreed - although what Stu said seems correct to me. Distribution story stays same. Those that use pants shell script have it extract rust binary, those that use pex - not sure.
But pants python code could have a built in 1 time run and extract relevant bit part.
w

witty-crayon-22786

03/22/2020, 12:21 AM
we've got scripts too. would do something similar.
e

enough-analyst-54434

03/22/2020, 12:21 AM
Pants python code already does this for the native.so.
h

hundreds-breakfast-49010

03/22/2020, 12:21 AM
so,maybe we'd provide a compiled-from-rust binary called I dunno 
pants_cli
, and in the 
pants
shell script we'd swap out the line 
exec "${pants_dir}/bin/python" "${pants_dir}/bin/pants" "$@"
for just 
./pants_cli
and then 
pants_cli
would have to figure out how to do the same stuff that that shell script does, in the case where we're not using pantsd
in terms of dealing with whl's and whatever else that script does
e

enough-analyst-54434

03/22/2020, 12:22 AM
Well, you'd still let the shell scrit fully install pants. Then, do the binary extract.
w

witty-crayon-22786

03/22/2020, 12:22 AM
John and Eric both have the most experience with that part
but yea. not a blocker if the experiment plays out.
h

hundreds-breakfast-49010

03/22/2020, 12:24 AM
so @enough-analyst-54434 are you imagining that we'd put the logic for extracting a native compiled binary from a whl in the 
bootstrap_pants
function?
w

witty-crayon-22786

03/22/2020, 12:25 AM
@hundreds-breakfast-49010: what the pantsbuild/pants repo does is different from end user repos.
in pantsbuild/pants you could cheat and just literally 
cargo run $mybinary
from the pants script.
h

hundreds-breakfast-49010

03/22/2020, 12:25 AM
right, I'm looking at 
pants
as installed in a random test repo
I know the 
pants
in the pants repo is different
w

witty-crayon-22786

03/22/2020, 12:26 AM
k
h

hundreds-breakfast-49010

03/22/2020, 12:26 AM
I'm actually not sure how the shell script we ask people to curl to install pants gets built
we ask people to curl
Copy code
<https://pantsbuild.github.io/setup/pants>
but I don't think this script exists verbatim anywhere in the pants git repo
w

witty-crayon-22786

03/22/2020, 12:45 AM
yes. it's in a separate setup repo: github.com/pantsbuild/setup/
as mentioned above though, that is a wide, clearly lit bridge that we can cross when we come to it.
h

hundreds-breakfast-49010

03/22/2020, 12:47 AM
I'm not personally sure how I'd go about implementing the rust pants initializer without crossing this bridge, so to speak
w

witty-crayon-22786

03/22/2020, 12:48 AM
three other people are, heh
what i'm saying is: when we have a tool that works in the pantsbuild/pants repo, using whatever hacks we want there, distributing it will not be an issue.
h

hundreds-breakfast-49010

03/22/2020, 12:50 AM
right now I'm trying to think through what all the steps would be to actually make running 
./pants <whatever>
at the command line start up a rust program that does the initialization work we currently do in python, with the expectation that I might write this and submit a PR next week. it's very possible that I'm missing something important that would make this easier to do or harder to do than I think it would, because I personally lack familiarity with something other people already know about
so I'm mostly asking these questions to make sure that I understand everything I need to in order to do this work, or at least be sure that I don't need to understand it to do this work
w

witty-crayon-22786

03/22/2020, 12:53 AM
yep, understood. but John and i have both confirmed in this thread that that will not be a challenge relative to the rest of the project. it should literally be this line: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1584835986097700?thread_ts=1584805095.053900&amp;cid=C0D7TNJHL
and the intent there is to assure you that you do not need to worry about it. the other parts are the parts to focus on.
h

hundreds-breakfast-49010

03/22/2020, 12:54 AM
ok cool
so, that means that I need to make sure that I can build a rust binary that will get inserted into a whl that we distribute
w

witty-crayon-22786

03/22/2020, 12:55 AM
yep
h

hundreds-breakfast-49010

03/22/2020, 12:55 AM
maybe this should be an entirely new crate within the pants repo, so I don't need to worry about making it work with the 
native_engine.so
lib we already build?
w

witty-crayon-22786

03/22/2020, 12:56 AM
it should, yes. we have 15+ crates already... no need to link it to the 
engine
crate.
(a few of which are binaries)
h

hundreds-breakfast-49010

03/22/2020, 12:57 AM
so okay a new top-level crate is easy to make, and it's easy enough to make that spit out a binary. I'm not sure what the process of packaging that binary into a whl would look like, or how'd we'd thinking about building for specific platforms
if the binary was small enough it might even be tractable to have the pants bootstrap script compile it on the system of the user, but then they'd have to have rust as a dependency to use pants, which we might not want
hm no that's probably not a good idea
w

witty-crayon-22786

03/22/2020, 1:00 AM
i believe that you can ignore the entire packaging problem until you have a working implementation that ignores all of it
we have per-platform wheels
so per-platform binaries inside of those wheels is not challenging.
i referenced literally using 
cargo run $binary
in the pantsbuild/pants repo because it would allow you to completely ignore packaging until the whole thing is working end to end
👍 1
(i tend to abuse the word "literally"... i think that the command would be more like 
build-support/bin/native/cargo run $binary
, in order to use our bootstrapped cargo and rust)
👍 1
Powered by