https://pantsbuild.org/ logo
Join Slack
Powered by
Hello, I’m reaching out because bootstrapping pant...
# general
s
Hello, I’m reaching out because bootstrapping pants today caused the whole installation on my macbook pro M1 to break (xcode, git, direnv, virtualenv, kukectl, pyenv’s python...). I managed to get back xcode working by re-setting the path to the command line tools, then git. However it seems all the others are still broken. Is there any explanation or resources about what happens during bootstraping and what is impacted? Looks like most resources are for most somewhat linked with python, maybe that not setting a python-bootstrap in the pants.toml had such really unexpected behaviour?
⁉️ 4
w
Whoa, wait, you did a 
./pants --version
or something and then your system tools stopped working?
What does "whole installation" refer to in this case? I'm hoping it's just the pants repo ?
s
I actually did a 
./pants lint ::
, we’re using version 2.13.0 currently and did not set a 
python-bootstrap
. It looks like it messed with the 
xcode-select path
on which relies 
git
. Then for the remaining package it looks like I had 
pyenv
, 
virtualenv
andd 
direnv
impacted. A 
brew reinstall direnv
fixed the problem but for 
pyenv
and 
virtualenv
I’m still checking what’s going on.
But I mean just the fact that we had some xcode paths messed up in the process is weird
w
I've never seen anything like this before. To the best of my knowledge, 
xcode-select
isn't touched by Pants (other than in CI documentation, on setting up the runner). Do you have any precommit scripts, or scripts of your own which manipulate system vars?
h
Oh wow, that is troubling! I have never heard of anything like this happening
You can look at the 
./pants
script, the standard one doesn’t change any paths or run 
xcode-select
(but of course someone may have modified your repo’s 
pants
script I suppose)
Many people use Pants on an M1 without incident, myself included, so this is very odd indeed
For context - is this a Pants installation that has worked for a while, or is this the first time you’re setting up Pants on this machine?
s
After removing the pants cache, reinstalling homebrew, direnv, pyenv, and others, I re downloaded the script with 
curl -L -o ./pants <https://pantsbuild.github.io/setup/pants> &&  \ chmod +x ./pants
. Then I bootstrapped pants with 
./pants --version
(2.13) which completed successfully, ran 
./pants lint ::
which completed. Right after completion, VSCode shut down, and goes the same problem again: pyenv links not working anymore, any terminal running into issues importing some CLIs that have been installed with homebrew (direnv, pyenv, virtualenv) as well as through direct link such as kubectl. OSX prompt to reinstall command line tools and so on, which is again due to a corruption of the xcode-select path of the command line tools.
h
Good lord! Sorry for the trouble.
Is this the first time running Pants on this machine?
Trying to figure out what may have changed
w
Could you also show your pants.toml?
s
Copy code
[GLOBAL]
pants_version = "2.13.0"
backend_packages = [
    "pants.backend.docker",
    "pants.backend.docker.lint.hadolint",
    "pants.backend.python",
    "pants.backend.python.lint.black",
    "pants.backend.python.lint.flake8",
    "pants.backend.python.typecheck.mypy",
]
concurrent = true
use_deprecated_directory_cli_args_semantics = false
use_deprecated_pex_binary_run_semantics = false

[source]
root_patterns = ["airflow/dags/", "packages/", "scripts/", "tasks/"]

[anonymous-telemetry]
enabled = false

[docker]
build_verbose = true
build_args = ["ENVIRONMENT"]

[docker.registries.ghcr]
address = "<http://ghcr.io/ledgerhq|ghcr.io/ledgerhq>"
default = true

[python]
interpreter_constraints = [">=3.10,<3.11"]
enable_resolves = true
tailor_pex_binary_targets = true

[python.resolves]
python-default = "3rdparty/python-default/python-default.lock"
airflow = "3rdparty/airflow/airflow.lock"

[python.resolves_to_interpreter_constraints]
python-default = [">=3.10,<3.11"]
airflow = [">=3.7.2,<3.11"]

[flake8]
config = ".flake8"
extra_requirements = [
  "darglint",
  "flake8-annotations",
  "flake8-bandit",
  "flake8-black",
  "flake8-bugbear",
  "flake8-docstrings",
  "flake8-import-order",
  "importlib-metadata==4.13.0",
]
lockfile = "3rdparty/tools/flake8.lock"

[black]
version = "black==22.3.0"
interpreter_constraints = [">=3.10,<3.11"]
lockfile = "3rdparty/tools/black.lock"

[pytest]
lockfile = "3rdparty/tools/pytest.lock"

[mypy]
interpreter_constraints = [">=3.10,<3.11"]
lockfile = "3rdparty/tools/mypy.lock"
version = "mypy==0.981"
I actually managed to run pants on the machine a few months back, on version 2.11 if I remember correctly. Back then I only had issue building .pex files including packages for which there wasn’t wheel for arm architecture. I didn’t try to lint anything though.
w
Have you been able to tell if any of your system ENV vars are being modified or wiped out on the 
./pants lint ::
?
s
erh no that I didn’t think of exporting all the ENV vars. That’s something I should’ve checked... I’ll try again tomorrow
w
Okay, as a sanity - here are some ideas: • Check out the 
.pants.d/pants.log
(or exceptions.log) to see if there is anything obvious logged • Export your env vars before and after to see if anything is changed (shouldn't be) • Disable backends and add them back one by one to see if any backend is causing the problem (especially docker/hadolint) • Check if any repo scripts are triggered somehow (via lint, or git, or precommit, etc) - basically, is this happening on ANY pants repo (ie. https://github.com/pantsbuild/example-python) or just THIS repo
s
@wide-midnight-78598 thanks you for these recommandations. After updating the 
pants.toml
by constraining python interpreters to >=3.8, it seems pants does not crash anymore like it did yesterday. Comparing the environment variables it looks like there’s no changes neither. However the xcode command lines problem still occurs, so I tried clearing the cache again and bootstrap pants. Now I’m running into:
Copy code
Scrubbing PYTHONPATH
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants==2.13.0 (from versions: 0.0.17, 0.0.18, 0.0.20, 0.0.21, 0.0.22, 0.0.23, 0.0.24, 0.0.25, 0.0.26, 0.0.27, 0.0.28, 0.0.29, 0.0.30, 0.0.31, 0.0.32, 0.0.33, 0.0.34, 0.0.35, 0.0.36, 0.0.37, 0.0.38, 0.0.39, 0.0.40, 0.0.41, 0.0.42, 0.0.43, 0.0.44, 0.0.45, 0.0.46, 0.0.47, 0.0.48, 0.0.49, 0.0.50, 0.0.51, 0.0.52, 0.0.53, 0.0.54, 0.0.55, 0.0.56, 0.0.57, 0.0.58, 0.0.59, 0.0.60, 0.0.61, 0.0.62, 0.0.63, 0.0.64, 0.0.65, 0.0.66, 0.0.67, 0.0.68, 0.0.69, 0.0.70, 0.0.71, 0.0.72, 0.0.73, 0.0.74, 0.0.75, 0.0.76, 0.0.77, 0.0.79, 0.0.80, 0.0.81, 0.0.82, 1.0.0, 1.0.1, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0)
ERROR: No matching distribution found for pantsbuild.pants==2.13.0
w
Hmm, has your system been trying to grab your xcode-installed python?
For M1, python 3.9 is involved somehow. One sec, let me look at the docs - I'm still on x86 macs
s
That would be weird, I’ve set 3.8, 3/9 and 3.10 pyenv version first on the path
w
s
And the thing is that in the pants.toml we have one resolve that has to be for python 3.7 so we’re cheating a bit by putting 3.8 now.
ok so that’s why
w
For anything else down this line- @happy-kitchen-89482 or @hundreds-father-404 are better people to talk to, as I don't know much about M1 or workarounds yet.
s
Well so yeah it seems it is the hadolint backend that literally breaks everything
h
oh wow
First, to clarify re interpreter selection: there’s the version that 
./pants
selects to run Pants itself against, which is unrelated to the interpreters that your code needs, so you weren’t cheating 🙂
So… we have a problem with hadolint
Thanks for the discovery, and really sorry for the trouble!
So @curved-television-6568 any ideas?
Oh, and back to the interpreters - yes, M1 requires Python 3.9 to run Pants itself on. I thought we had modified the 
./pants
script to handle that, but I must be misremembering
w
Well so yeah it seems it is the hadolint backend that literally breaks everything
Whelp, that was my guess unfortunately. Unsure if this is related (https://github.com/pantsbuild/pants/blob/a01ec979a3a5bca05b2b5a5d566d816102a86ed3/src/python/pants/backend/docker/lint/hadolint/rules.py#L80) I remember hadolint giving me a hassle outside of Pants a year or so ago, and I just stopped using it out of frustration (in spite of how helpful it is/was). Is it worth grabbing the hadolint binary outside of Pants and seeing if it's the problem in isolation? Looks like the ARM target points at x86, which may not be stable over macos releases? In https://github.com/pantsbuild/pants/blob/a01ec979a3a5bca05b2b5a5d566d816102a86ed3/src/python/pants/backend/docker/lint/hadolint/subsystem.py https://github.com/hadolint/hadolint/releases/download/v2.10.0/hadolint-Darwin-x86_64 
"v2.10.0|macos_arm64 |59f0523069a857ae918b8ac0774230013f7bcc00c1ea28119c2311353120867a|2514960",  # same as mac x86
c
Was just looking into that, and it seems the hadolint project only releases for x86 Mac, but brew has a formulae with packages for arm64, it seems.. https://github.com/Homebrew/homebrew-core/blob/master/Formula/hadolint.rb.
Maybe worth looking into fetching from them on Mac?
running a brew installed hadolint on M1 seems to work..
h
oh, interesting, is the hypothesis that this is chaos caused by rosetta, somehow?
c
idk- but if hadolint is causing issues, and we’re running the x86 version on arm64, I guess all bets may be off 🤷
could be worth trying to setup pants to download an arm64 version of hadolint and use that on an M1..
s
Yeah hadolint seems to work alright outside pants. Regarding the Python version indeed I should have looked into the github issue to spot that specificity, I got confused by the fact we need these several resolve with 3.7 or 3.10 and the fact 3.9 is the one used to bootstrap pants. For now I think we’ll simply remove hadolint, with our current set-up it probably the least important feature considering our dockerfile are no more than 3 lines long.
I jump at the opportunity to ask you about running pants on a ubuntu container running on a mac M1. host: 
Darwin 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:19:52 PDT 2022; root:xnu-8020.140.49~2/RELEASE_ARM64_T6000 arm64
remote: 
Linux bd2d5ff6d11c 5.10.124-linuxkit #1 SMP PREEMPT Thu Jun 30 08:18:26 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
Since there’s no distribution for linux (aarch64) i’ve managed to build the wheel from sources successfully within the container (
./pants package src/python/pants:pants-packaged
) and everything runs fine within the 
./pants repo
. Question is, how to use that wheel from our own repo?
h
Do you have an internal PyPI server you can host it on?
s
Erh nop that’s only for my own development environment. Is it an absolute requirement? I thought maybe modifying the 
bootstrap_pants
function of the pants script would be enough?
n
Just ran into this hadolint issue - we should probably raise an issue for it?
2 Views
Open in Slack
PreviousNext
Powered by