https://pantsbuild.org/ logo
Join Slack
Powered by
Yeah, so apparently something in the pants repo br...
# development
l
Yeah, so apparently something in the pants repo broke my system so bad that nothing I can do seems to make that error go away.
f
you could nuke 
~/.cache/pants/pants_dev_deps
maybe?
l
I have done that
I've nuked 
~/.cache/pants
f
and then checkout main
and try bootstrapping again
l
that's what I'm doing
I've cloned the repo again
f
oh... are you in the Pants repo itself?
l
yes
f
probably best to use 
./pants
l
same result
It seems to be absolutely convinced that there's a python interpreter in /home/matthias/.cache/pants/pants_dev_deps/b6a9ea0e006135b304496084706b185ea6a919b6.venv/bin/python3.9
and nothing I can do seems to convince it otherwise.
f
it pantsd running?
l
no
ps -ef | grep -i pants
doesn't show anything
except the grep command, of course
f
maybe nuke 
~/.cache/nce/
l
done
Doesn't do anything
Hm, I've tried running that command manually and now at least I get a better error message.
f
which command?
l
Copy code
$ /home/matthias/.cache/pants/pants_dev_deps/b6a9ea0e006135b304496084706b185ea6a919b6.venv/bin/python3.9 -Im ensurepip --upgrade --default-pip
/home/matthias/.cache/pants/pants_dev_deps/b6a9ea0e006135b304496084706b185ea6a919b6.venv/bin/python3.9: No module named ensurepip
f
ah maybe the bootstrap process is grabbing some system interpreter that doesn't have ensurepip bundled with it
I guess running 
pants
and/or 
./pants
is recreating that 
pants_dev_deps
directory every time?
l
apparently so
I've installed Python 3.9 from a PPA for Ubuntu 22.04
(installed Ubuntu because that's what everybody in the company was using. That was a mistake)
f
ah... i'd avoid using ubuntu pythons for python development
l
I'd avoid using Ubuntu anything for anything at this point
But I've tried with Python 3.9 from Nixpkgs as well and also had issues with it.
f
So I'm not sure how the Pants-repo bootstrap locates its Python, but if we can figure out how to get it to use a standalone python or something like a pyenv or asdf python, we can probably make this work. Does someone who understands this bootstrap process have any ideas?
Are you using the deadsnakes PPAs?
l
yes
f
I would avoid using those, and use https://asdf-vm.com/ for getting development verisons of python
Linux package maintainers are often targeting a very different set of needs than people trying to develop for multiple versions of a language ecosystem. This applies even to things like Nix
That doesn't necessarily make what they do bad, and it doesn't make Ubuntu bad. But it does make it confusing 😩
l
From what I've experienced with Ubuntu for the past two months, I'm pretty confident to say it's bad
😐 1
f
My suggestion is to uninstall python3.9 and remove the deadsnakes ppa, and use asdf or pyenv to install dev versions Python
l
I'm going to try that
b
Alternatively it seems right now (somewhat out of luck) we're compatible with Python Build Standlone: https://gregoryszorc.com/docs/python-build-standalone/main/
I'm hoping we can solidify that, since thats what 
pants
uses
f
yeah i think PBS is the way we want to go in the future, but using that in the Pants repo is still a work-in-progress right? https://pantsbuild.slack.com/archives/C0D7TNJHL/p1691685217153919
b
It's not guaranteed to work. But likely should and will
f
Let's get it to work and document it when it does. I'll gladly update the contribution documentation to recommend using 
pants
over 
./pants
in-repo once that is done
b
We can actually have that error. There's an env var we can sniff
👍🏻 1
l
Well, I've downloaded the Python 3.9 standalone build and added the Python 3.9 binary to my PATH
That part at least seems to be working, but now I'm getting another error.
Copy code
error: failed to run custom build command for `grpc_util v0.0.1 (/home/matthias/pants/src/rust/engine/grpc_util)`

Caused by:
  process didn't exit successfully: `/home/matthias/pants/src/rust/engine/target/release/build/grpc_util-cddf00b0d183a9f4/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=protos/test.proto
  cargo:rerun-if-changed=protos

  --- stderr
  thread 'main' panicked at 'Could not find `protoc` installation and this build crate cannot proceed without
      this knowledge. If `protoc` is installed and this crate had trouble finding
      it, you can set the `PROTOC` environment variable with the specific path to your
      installed `protoc` binary.If you're on debian, try `apt-get install protobuf-compiler` or download it from <https://github.com/protocolbuffers/protobuf/releases>

  For more information: <https://docs.rs/prost-build/#sourcing-protoc>
  ', /home/matthias/.cargo/registry/src/index.crates.io-6f17d22bba15001f/prost-build-0.11.9/src/lib.rs:1457:10
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
error: failed to run custom build command for `protos v0.0.1 (/home/matthias/pants/src/rust/engine/protos)`

Caused by:
  process didn't exit successfully: `/home/matthias/pants/src/rust/engine/target/release/build/protos-c41201e5b06d9ea9/build-script-build` (exit status: 101)
  --- stdout
  cargo:rerun-if-changed=protos/bazelbuild_remote-apis/build/bazel/remote/execution/v2/remote_execution.proto
  cargo:rerun-if-changed=protos/bazelbuild_remote-apis/build/bazel/semver/semver.proto
  cargo:rerun-if-changed=protos/buildbarn/cas.proto
  cargo:rerun-if-changed=protos/googleapis/google/bytestream/bytestream.proto
  cargo:rerun-if-changed=protos/googleapis/google/rpc/code.proto
  cargo:rerun-if-changed=protos/googleapis/google/rpc/error_details.proto
  cargo:rerun-if-changed=protos/googleapis/google/rpc/status.proto
  cargo:rerun-if-changed=protos/googleapis/google/longrunning/operations.proto
  cargo:rerun-if-changed=protos/pants/cache.proto
  cargo:rerun-if-changed=protos/standard/google/protobuf/empty.proto
  cargo:rerun-if-changed=protos/bazelbuild_remote-apis
  cargo:rerun-if-changed=protos/buildbarn
  cargo:rerun-if-changed=protos/googleapis
  cargo:rerun-if-changed=protos/pants
  cargo:rerun-if-changed=protos/standard

  --- stderr
  thread 'main' panicked at 'Could not find `protoc` installation and this build crate cannot proceed without
      this knowledge. If `protoc` is installed and this crate had trouble finding
      it, you can set the `PROTOC` environment variable with the specific path to your
      installed `protoc` binary.If you're on debian, try `apt-get install protobuf-compiler` or download it from <https://github.com/protocolbuffers/protobuf/releases>

  For more information: <https://docs.rs/prost-build/#sourcing-protoc>
  ', /home/matthias/.cargo/registry/src/index.crates.io-6f17d22bba15001f/prost-build-0.11.9/src/lib.rs:1457:10
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Oh, apparently 
apt install protobuf-compiler
does something
f
this is a recent thing involving a change in some rust crate
l
I think basically all of this stuff should be in the docs. How to install the correct Python interpreter (Standalone Python seems to work), and rustup and apparently protoc.
Or better yet, get it running with Nix so the only thing people need to install is nix and everything else is taken care of automatically
😩
But now I'm getting this error again:
Copy code
There is no pantsd metadata at /home/matthias/pants/.pids/8b5fde57d45d/pantsd.
Traceback (most recent call last):
  File "/home/matthias/pants/src/python/pants/bin/pants_loader.py", line 18, in <module>
    from pants.bin.pants_runner import PantsRunner
  File "/home/matthias/pants/src/python/pants/bin/pants_runner.py", line 11, in <module>
    from pants.base.exception_sink import ExceptionSink
  File "/home/matthias/pants/src/python/pants/base/exception_sink.py", line 19, in <module>
    from pants.util.dirutil import safe_mkdir, safe_open
  File "/home/matthias/pants/src/python/pants/util/dirutil.py", line 21, in <module>
    from pants.util.strutil import ensure_text
  File "/home/matthias/pants/src/python/pants/util/strutil.py", line 17, in <module>
    from pants.engine.internals.native_engine import Digest
ImportError: libstdc++.so.6: cannot open shared object file: No such file or directory
f
l
That seems to be about a 32-Bit program
I think it might have something to do with my previous attempt at installing a Python 3.9 interpreter using Nix. Nix uses non-standard paths for shared objects.
But I was hoping I had removed anything to do with Nix…
f
I think basically all of this stuff should be in the docs. How to install the correct Python interpreter (Standalone Python seems to work), and rustup and apparently protoc.
I agree, this is a documentation gap.
Or better yet, get it running with Nix so the only thing people need to install is nix and everything else is taken care of automatically
We can't dictate what distros or packaging tech people use. Many (if not most) of our developers use Mac too, and are likely using brew to get deps. However, having good dev docs for challenges that we know in various setup come up would be great. I could see that going in https://github.com/pantsbuild/pants/blob/main/docs/markdown/Contributions/development/contributor-setup.md if you're willing to add a troubleshooting section there with your problems and steps and submit a PR. It would be much appreciated.
l
I guess I could write down what I did once I actually got it working 😂
f
I actually have an aborted attempt at creating a devcontainer for developing Pants but even that assumes people are 1) working with an IDE that allows devcontainers and 2) want to use devcontainers. I should probably add it there though.
l
And sure, you can't dictate what people use. But people usually go the path of least resistance, so if you just add a Nix flake and it provides everything people need in a hassle-free way, I guess that people would use it.
And Nix works on macOS too
f
I've never used nix standalone, (I've played with the whole NixOS though), so I don't know if I'm comfortable recommending it.
l
Well, it does seem to be what is causing trouble for me right now.
Copy code
/home/matthias/.cache/pants/pants_dev_deps/b6a9ea0e006135b304496084706b185ea6a919b6.venv/bin/python: /usr/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.36' not found (required by /nix/store/70i27j50bn0zj2zgbxww087xhbg60ps6-libxcrypt-
4.4.36/lib/libcrypt.so.2)
f
Sorry you're having trouble getting up and running though. I don't remember experiencing this much difficulty, but sometimes my memories just paper over problems I think are easy to solve and I may be forgetting installing missing dev headers and such
Yeah I think Nix is messing with your load paths
l
I nuked 
~/.cache/pants
again now, and now I get a different error.
f
l
I think the Nix problem is solved, this is something else…
Copy code
clang -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -I/tools/deps/libedit/include -fPIC -I/home/matthias/.cache/pants/named_caches/pex_root/venvs/9b295203f2e20a675f10f528c2d0f52164f7ff12/5d74400d2b55d43e0cfb00c9dab3c5e407b54e54/include -I/home/matthias/cpython-3.9.17+20230726-x86_64-unknown-linux-gnu-lto-full/install/include/python3.9 -c src/python-codec.c -o build/temp.linux-x86_64-cpython-39/src/python-codec.o
      error: command 'clang' failed: No such file or directory
      [end of output]
let's see if installing clang fixes it
hdrhistogram
seems to need this
OK, now finally I get some sensible output from 
pants list ::
Thanks for all the help
👍🏻 1
f
thanks for contributing to Pants! Sorry it was a struggle
b
Ugh 
hdrhistogram
is sdist-only? 😬 https://pypi.org/project/hdrhistogram/#files
l
I don't know what that means.
b
That's very unfortunate, as that means we can't reliably use it with PBS 😞
f
sdists are python "source distributions", but in this case it contains C code that needs to be compiled
we could maintain a wheelhouse in S3 and build a pre-compiled verison of it 😉
b
@flat-zoo-31952 let's jump this discussion to https://pantsbuild.slack.com/archives/C0D7TNJHL/p1691685217153919
👍🏻 1
l
well do you want to ship pre-compiled binaries for macOS on ARM, macOS on x86_64, Linux on x86_64 and who knows what else?
2 Views
Open in Slack
PreviousNext
Powered by