https://pantsbuild.org/ logo
#general
Title
# general
s

strong-answer-14518

10/12/2022, 1:37 PM
Hello, I’m reaching out because bootstrapping pants today caused the whole installation on my macbook pro M1 to break (xcode, git, direnv, virtualenv, kukectl, pyenv’s python...). I managed to get back xcode working by re-setting the path to the command line tools, then git. However it seems all the others are still broken. Is there any explanation or resources about what happens during bootstraping and what is impacted? Looks like most resources are for most somewhat linked with python, maybe that not setting a python-bootstrap in the pants.toml had such really unexpected behaviour?
⁉️ 4
w

wide-midnight-78598

10/12/2022, 1:40 PM
Whoa, wait, you did a
./pants --version
or something and then your system tools stopped working?
What does "whole installation" refer to in this case? I'm hoping it's just the pants repo ?
s

strong-answer-14518

10/12/2022, 1:57 PM
I actually did a
./pants lint ::
, we’re using version 2.13.0 currently and did not set a
python-bootstrap
. It looks like it messed with the
xcode-select path
on which relies
git
. Then for the remaining package it looks like I had
pyenv
,
virtualenv
andd
direnv
impacted. A
brew reinstall direnv
fixed the problem but for
pyenv
and
virtualenv
I’m still checking what’s going on.
But I mean just the fact that we had some xcode paths messed up in the process is weird
w

wide-midnight-78598

10/12/2022, 2:22 PM
I've never seen anything like this before. To the best of my knowledge,
xcode-select
isn't touched by Pants (other than in CI documentation, on setting up the runner). Do you have any precommit scripts, or scripts of your own which manipulate system vars?
h

happy-kitchen-89482

10/12/2022, 2:26 PM
Oh wow, that is troubling! I have never heard of anything like this happening
You can look at the
./pants
script, the standard one doesn’t change any paths or run
xcode-select
(but of course someone may have modified your repo’s
pants
script I suppose)
Many people use Pants on an M1 without incident, myself included, so this is very odd indeed
For context - is this a Pants installation that has worked for a while, or is this the first time you’re setting up Pants on this machine?
s

strong-answer-14518

10/12/2022, 4:26 PM
After removing the pants cache, reinstalling homebrew, direnv, pyenv, and others, I re downloaded the script with
curl -L -o ./pants <https://pantsbuild.github.io/setup/pants> &&  \ chmod +x ./pants
. Then I bootstrapped pants with
./pants --version
(2.13) which completed successfully, ran
./pants lint ::
which completed. Right after completion, VSCode shut down, and goes the same problem again: pyenv links not working anymore, any terminal running into issues importing some CLIs that have been installed with homebrew (direnv, pyenv, virtualenv) as well as through direct link such as kubectl. OSX prompt to reinstall command line tools and so on, which is again due to a corruption of the xcode-select path of the command line tools.
h

happy-kitchen-89482

10/12/2022, 5:41 PM
Good lord! Sorry for the trouble.
Is this the first time running Pants on this machine?
Trying to figure out what may have changed
w

wide-midnight-78598

10/12/2022, 6:26 PM
Could you also show your pants.toml?
s

strong-answer-14518

10/12/2022, 6:56 PM
Copy code
[GLOBAL]
pants_version = "2.13.0"
backend_packages = [
    "pants.backend.docker",
    "pants.backend.docker.lint.hadolint",
    "pants.backend.python",
    "pants.backend.python.lint.black",
    "pants.backend.python.lint.flake8",
    "pants.backend.python.typecheck.mypy",
]
concurrent = true
use_deprecated_directory_cli_args_semantics = false
use_deprecated_pex_binary_run_semantics = false

[source]
root_patterns = ["airflow/dags/", "packages/", "scripts/", "tasks/"]

[anonymous-telemetry]
enabled = false

[docker]
build_verbose = true
build_args = ["ENVIRONMENT"]

[docker.registries.ghcr]
address = "<http://ghcr.io/ledgerhq|ghcr.io/ledgerhq>"
default = true

[python]
interpreter_constraints = [">=3.10,<3.11"]
enable_resolves = true
tailor_pex_binary_targets = true

[python.resolves]
python-default = "3rdparty/python-default/python-default.lock"
airflow = "3rdparty/airflow/airflow.lock"

[python.resolves_to_interpreter_constraints]
python-default = [">=3.10,<3.11"]
airflow = [">=3.7.2,<3.11"]

[flake8]
config = ".flake8"
extra_requirements = [
  "darglint",
  "flake8-annotations",
  "flake8-bandit",
  "flake8-black",
  "flake8-bugbear",
  "flake8-docstrings",
  "flake8-import-order",
  "importlib-metadata==4.13.0",
]
lockfile = "3rdparty/tools/flake8.lock"

[black]
version = "black==22.3.0"
interpreter_constraints = [">=3.10,<3.11"]
lockfile = "3rdparty/tools/black.lock"

[pytest]
lockfile = "3rdparty/tools/pytest.lock"

[mypy]
interpreter_constraints = [">=3.10,<3.11"]
lockfile = "3rdparty/tools/mypy.lock"
version = "mypy==0.981"
I actually managed to run pants on the machine a few months back, on version 2.11 if I remember correctly. Back then I only had issue building .pex files including packages for which there wasn’t wheel for arm architecture. I didn’t try to lint anything though.
w

wide-midnight-78598

10/12/2022, 7:14 PM
Have you been able to tell if any of your system ENV vars are being modified or wiped out on the
./pants lint ::
?
s

strong-answer-14518

10/12/2022, 7:45 PM
erh no that I didn’t think of exporting all the ENV vars. That’s something I should’ve checked... I’ll try again tomorrow
w

wide-midnight-78598

10/12/2022, 7:48 PM
Okay, as a sanity - here are some ideas: • Check out the
.pants.d/pants.log
(or exceptions.log) to see if there is anything obvious logged • Export your env vars before and after to see if anything is changed (shouldn't be) • Disable backends and add them back one by one to see if any backend is causing the problem (especially docker/hadolint) • Check if any repo scripts are triggered somehow (via lint, or git, or precommit, etc) - basically, is this happening on ANY pants repo (ie. https://github.com/pantsbuild/example-python) or just THIS repo
s

strong-answer-14518

10/13/2022, 12:46 PM
@wide-midnight-78598 thanks you for these recommandations. After updating the
pants.toml
by constraining python interpreters to >=3.8, it seems pants does not crash anymore like it did yesterday. Comparing the environment variables it looks like there’s no changes neither. However the xcode command lines problem still occurs, so I tried clearing the cache again and bootstrap pants. Now I’m running into:
Copy code
Scrubbing PYTHONPATH
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants==2.13.0 (from versions: 0.0.17, 0.0.18, 0.0.20, 0.0.21, 0.0.22, 0.0.23, 0.0.24, 0.0.25, 0.0.26, 0.0.27, 0.0.28, 0.0.29, 0.0.30, 0.0.31, 0.0.32, 0.0.33, 0.0.34, 0.0.35, 0.0.36, 0.0.37, 0.0.38, 0.0.39, 0.0.40, 0.0.41, 0.0.42, 0.0.43, 0.0.44, 0.0.45, 0.0.46, 0.0.47, 0.0.48, 0.0.49, 0.0.50, 0.0.51, 0.0.52, 0.0.53, 0.0.54, 0.0.55, 0.0.56, 0.0.57, 0.0.58, 0.0.59, 0.0.60, 0.0.61, 0.0.62, 0.0.63, 0.0.64, 0.0.65, 0.0.66, 0.0.67, 0.0.68, 0.0.69, 0.0.70, 0.0.71, 0.0.72, 0.0.73, 0.0.74, 0.0.75, 0.0.76, 0.0.77, 0.0.79, 0.0.80, 0.0.81, 0.0.82, 1.0.0, 1.0.1, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0)
ERROR: No matching distribution found for pantsbuild.pants==2.13.0
w

wide-midnight-78598

10/13/2022, 12:49 PM
Hmm, has your system been trying to grab your xcode-installed python?
For M1, python 3.9 is involved somehow. One sec, let me look at the docs - I'm still on x86 macs
s

strong-answer-14518

10/13/2022, 12:51 PM
That would be weird, I’ve set 3.8, 3/9 and 3.10 pyenv version first on the path
w

wide-midnight-78598

10/13/2022, 12:54 PM
s

strong-answer-14518

10/13/2022, 12:54 PM
And the thing is that in the pants.toml we have one resolve that has to be for python 3.7 so we’re cheating a bit by putting 3.8 now.
ok so that’s why
w

wide-midnight-78598

10/13/2022, 12:57 PM
For anything else down this line- @happy-kitchen-89482 or @hundreds-father-404 are better people to talk to, as I don't know much about M1 or workarounds yet.
s

strong-answer-14518

10/13/2022, 2:38 PM
Well so yeah it seems it is the hadolint backend that literally breaks everything
h

happy-kitchen-89482

10/13/2022, 2:39 PM
oh wow
First, to clarify re interpreter selection: there’s the version that
./pants
selects to run Pants itself against, which is unrelated to the interpreters that your code needs, so you weren’t cheating 🙂
So… we have a problem with hadolint
Thanks for the discovery, and really sorry for the trouble!
So @curved-television-6568 any ideas?
Oh, and back to the interpreters - yes, M1 requires Python 3.9 to run Pants itself on. I thought we had modified the
./pants
script to handle that, but I must be misremembering
w

wide-midnight-78598

10/13/2022, 3:05 PM
Well so yeah it seems it is the hadolint backend that literally breaks everything
Whelp, that was my guess unfortunately. Unsure if this is related (https://github.com/pantsbuild/pants/blob/a01ec979a3a5bca05b2b5a5d566d816102a86ed3/src/python/pants/backend/docker/lint/hadolint/rules.py#L80) I remember hadolint giving me a hassle outside of Pants a year or so ago, and I just stopped using it out of frustration (in spite of how helpful it is/was). Is it worth grabbing the hadolint binary outside of Pants and seeing if it's the problem in isolation? Looks like the ARM target points at x86, which may not be stable over macos releases? In https://github.com/pantsbuild/pants/blob/a01ec979a3a5bca05b2b5a5d566d816102a86ed3/src/python/pants/backend/docker/lint/hadolint/subsystem.py https://github.com/hadolint/hadolint/releases/download/v2.10.0/hadolint-Darwin-x86_64
"v2.10.0|macos_arm64 |59f0523069a857ae918b8ac0774230013f7bcc00c1ea28119c2311353120867a|2514960",  # same as mac x86
c

curved-television-6568

10/13/2022, 3:07 PM
Was just looking into that, and it seems the hadolint project only releases for x86 Mac, but brew has a formulae with packages for arm64, it seems.. https://github.com/Homebrew/homebrew-core/blob/master/Formula/hadolint.rb.
Maybe worth looking into fetching from them on Mac?
running a brew installed hadolint on M1 seems to work..
h

happy-kitchen-89482

10/13/2022, 4:01 PM
oh, interesting, is the hypothesis that this is chaos caused by rosetta, somehow?
c

curved-television-6568

10/13/2022, 4:08 PM
idk- but if hadolint is causing issues, and we’re running the x86 version on arm64, I guess all bets may be off 🤷
could be worth trying to setup pants to download an arm64 version of hadolint and use that on an M1..
s

strong-answer-14518

10/15/2022, 11:20 AM
Yeah hadolint seems to work alright outside pants. Regarding the Python version indeed I should have looked into the github issue to spot that specificity, I got confused by the fact we need these several resolve with 3.7 or 3.10 and the fact 3.9 is the one used to bootstrap pants. For now I think we’ll simply remove hadolint, with our current set-up it probably the least important feature considering our dockerfile are no more than 3 lines long.
I jump at the opportunity to ask you about running pants on a ubuntu container running on a mac M1. host:
Darwin 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:19:52 PDT 2022; root:xnu-8020.140.49~2/RELEASE_ARM64_T6000 arm64
remote:
Linux bd2d5ff6d11c 5.10.124-linuxkit #1 SMP PREEMPT Thu Jun 30 08:18:26 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux
Since there’s no distribution for linux (aarch64) i’ve managed to build the wheel from sources successfully within the container (
./pants package src/python/pants:pants-packaged
) and everything runs fine within the
./pants repo
. Question is, how to use that wheel from our own repo?
h

happy-kitchen-89482

10/16/2022, 4:57 AM
Do you have an internal PyPI server you can host it on?
s

strong-answer-14518

10/16/2022, 1:55 PM
Erh nop that’s only for my own development environment. Is it an absolute requirement? I thought maybe modifying the
bootstrap_pants
function of the pants script would be enough?
n

nutritious-hair-72580

12/17/2023, 11:33 PM
Just ran into this hadolint issue - we should probably raise an issue for it?