https://pantsbuild.org/ logo
Join Slack
Powered by
When upgrading pants is it expected that you'll ne...
# general
s
When upgrading pants is it expected that you'll need to run 
SCIE_BOOT=update pants
? Nevermind that worked on my Mac but not Linux machine* when trying 2.19.0rc4 I get
Copy code
Failed to exec pants at "/home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4/bin/pants": Exec format error
If I try to run 
/home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4/bin/pants
(was hoping this might point me towards something but it doesn't) directly I get
Copy code
/home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4/bin/pants: line 3: import: command not found
/home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4/bin/pants: line 4: import: command not found
from: can't read /var/mail/pants.bin.pants_loader
/home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4/bin/pants: line 7: syntax error near unexpected token `('
/home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4/bin/pants: line 7: `    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])'
If I run 
rm -fr ~/.cache/nce/ ~/.cache/pants/ .pids/ .pants.d/
and 
pants
I get
Copy code
Bootstrapping Pants 2.19.0rc4
Installing pantsbuild.pants==2.19.0rc4 into a virtual environment at /home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4
Found existing installation: setuptools 58.1.0
Uninstalling setuptools-58.1.0:
  Successfully uninstalled setuptools-58.1.0
New virtual environment successfully created at /home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4.
Failed to exec pants at "/home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4/bin/pants": Exec format error
Without remove the nce it's just the short error 
Failed to exec pants at "/home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4/bin/pants": Exec format error
Coincidentally we just upgraded pants to 2.18.1 and my coworker is running into the same problem so any help would be much appreciated Edit - Added more details
b
Sometimes, yes
s
Oh its in the docs https://www.pantsbuild.org/docs/installation. I should revisit these
b
Occasionally we change how Pants is packaged or distributed, In those cases sometimes 
scie-pants
(the pants launcher) needs to be upgraded to handle it
s
Any thoughts on the 
Exec format error
issue?
b
Hmm nothing comes to mind
w
Wrong architecture?
If you search "Exec format error" in slack, it happens a bunch of times, and it's an arch error - so that might help pinpoint what's going on in your setup?
s
I'm not doing anything different for 2.19.0rc4 than* I did for any other version of Pants though
w
Is there anything running in docker?
s
Nope. Just a plain ol debian 10 machine
w
huh... that's so bizarre. And it works if you downgrade pants to 2.18 or something?
s
yeah exactly
w
Like, on the updated scie-pants (0.10.4) and pants 2.18 - it works, then you update to 2.19 rc4 and it bombs out... I wonder if a bad arch executable got in there recently
s
Copy code
$ pants
Bootstrapping Pants 2.18.1
Installing pantsbuild.pants==2.18.1 into a virtual environment at /home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.1
Found existing installation: setuptools 58.1.0
Uninstalling setuptools-58.1.0:
  Successfully uninstalled setuptools-58.1.0
New virtual environment successfully created at /home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.18.1.
18:51:17.34 [INFO] waiting for pantsd to start...
18:51:18.24 [INFO] pantsd started
18:51:18.42 [INFO] Initializing scheduler...
18:51:23.13 [INFO] Scheduler initialized.
No goals specified.
Use `pants help` to get help.
Use `pants help goals` to list goals.
What's odd is this is happening to my coworker when he tries to go from 2.17 to 2.18
I'm really interested in solving the 2.19 issue so I can help them tbh. I don't want to downgrade back to 2.17
w
Yeah, shouldnt happen at all - but if this happens to your co-worker going from 217 to 218 - makes me feel it's a scie-pants thing. I know there was a change with libcrypt on 0.10.4, but that should have fixed problems like this (and it was released months ago)
Actually, can we sanity this first?
Copy code
PANTS_BOOTSTRAP_VERSION=report pants
s
I get 
0.10.4
. I'll ask my coworker
I asked them to run 
SCIE_BOOT=update pants
which they said they did
w
Are they on mac or windows? On mac, I install using brew, fyi
s
They're also on linux
w
👍
The part that's confusing me the most is your co-worker having this problem on a different version of pants. Do you use any custom plugins or anything "non-standard"?
s
We don't use any custom plugins
w
And, from what you said, nothing really works?
Copy code
pants --version
pants list ::
it all exec format errors out?
s
yep 😕
w
Whaaaaaa... Let me test out on my mac, just as a sanity. Does this happen on any repo? e.g. 
gh repo clone pantsbuild/example-python
s
I'll ask. We've got some meetings coming up but we'll be meeting up in a couple hours. I can confirm if I clone that and update to "2.19.0rc4"
Copy code
$ rm -fr ~/.cache/nce/ ~/.cache/pants/ .pids/ .pants.d/
$ pants
Bootstrapping Pants 2.19.0rc4
Installing pantsbuild.pants==2.19.0rc4 into a virtual environment at /home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4
Found existing installation: setuptools 58.1.0
Uninstalling setuptools-58.1.0:
  Successfully uninstalled setuptools-58.1.0
New virtual environment successfully created at /home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4.
Failed to exec pants at "/home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4/bin/pants": Exec format error
w
This is so weird, I just tried that on my mac and on an ubuntu machine, and went from 2.18 to 2.19.0rc4, both work
b
What about rc3?
w
Copy code
NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="<https://www.ubuntu.com/>"
SUPPORT_URL="<https://help.ubuntu.com/>"
BUG_REPORT_URL="<https://bugs.launchpad.net/ubuntu/>"
PRIVACY_POLICY_URL="<https://www.ubuntu.com/legal/terms-and-policies/privacy-policy>"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
In addition to iterating through the various 2.19 variants (but if your colleague has this problem from 2.17 to 2.18, then I'm less sure), what about grabbing the scie binary directly from here: https://github.com/pantsbuild/scie-pants/releases/download/v0.10.4/scie-pants-linux-x86_64 Just to ensure there wasn't a problem in the 
update
process for scie-pants, after the libcrypt change
Oh, x86 or aarch64?
s
I ran the following 
rm -fr ~/.cache/nce/ ~/.cache/pants/ .pids/ .pants.d/
curl <https://github.com/pantsbuild/scie-pants/releases/download/v0.10.4/scie-pants-linux-x86_64>
bash get-pants.sh
Installed to 
/home/yjabri/bin/pants
. Sanity check that 
which pants
points there.
Copy code
pants
Bootstrapping Pants 2.19.0rc4
Installing pantsbuild.pants==2.19.0rc4 into a virtual environment at /home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4
Found existing installation: setuptools 58.1.0
Uninstalling setuptools-58.1.0:
  Successfully uninstalled setuptools-58.1.0
New virtual environment successfully created at /home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4.
Failed to exec pants at "/home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4/bin/pants": Exec format error
uname -m
gives me 
x86_64
w
Man.... wtf
I can't repro this on my system(s). Josh's idea of trying to progressively lower 2.19 versions to narrow down when this problem started is an idea. When I needed to do this in the past, I basically quick-sorted my way through the versions - so 2.19.0.dev0 to rc4, or a couple cherry picked options in there anyways. I don't know what's special about 2.19 in this regard though
s
If I clone pants and checkout release_2.19.0rc4 I can run 
PANTS_SOURCE=../pants pants
Sanity check
Copy code
$ PANTS_SOURCE=../pants pants --version
Pantsd has been turned off via Env.
2.19.0rc4
w
Is your colleague using the same OS as you? Debian 10, x86?
s
Yep
We have an internal remote development service that spins up machines. I ended up just creating a fresh instance with the same configuration they used and couldn't reproduce the issue.
I asked them to try out a new remote machine. I'll chalk this one to a solar flare flipping some python bits. Returning to 2.19.0rc4 I can run
Copy code
$ NO_SCIE_WARNING=1 /home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4/bin/pyth
on /home/yjabri/.cache/nce/3d6643e46b53e4cc0b2a0d5c768866226ddce3de1f57f80c4a02d8d39800fa8e/bindings/venvs/2.19.0rc4/bin/pants
23:23:24.25 [INFO] waiting for pantsd to start...
23:23:24.65 [INFO] pantsd started
23:23:24.84 [INFO] Initializing scheduler...
23:23:28.99 [INFO] Scheduler initialized.
No goals specified.
Use `pants help` to get help.
Use `pants help goals` to list goals.
yjabri@remote-yjabri-default:~/archipelago$
Afterward 
pants
works
When my coworker made a new machine they had the same issue
I think its a shebang issue?
if I ssh into their machine and try to setup pants on the python example repo it works
their username is longer than mine though
The problem they were facing is that in 2.18 the shebang changed from 
.../python
to 
.../python3.9 -sE
pushing it over the 128 limit
I'll file a bug I guess edit - I'll search slack first though to see if other folks have faced anything similar
w
That bug again?
We've had something to that effect - several years ago, and I think it cropped up again recently-ish
Goddamn, I even commented on that one - but I don't think I really clued in that it presents as the exec error
s
lol I definitely did a bad job searching here
Same error upgrading to 2.18
To be fair this is a pretty odd edge case 😛 . My favorite part of the error message I didn't pay enough attention to originally was 
from: can't read /var/mail/pants.bin.pants_loader
That should have been a clue that something very wrong was going on lol
I really appreciate the help though! I think I would have folded a lot earlier without it
w
😆
s
Given https://github.com/pantsbuild/pex/pull/2295/files#diff-b1297a55702e854b6233796710a712f1f3fd47905ecc1fabd9f03c63ac980b43R1473-R1487 should I still file an issue? Seems like its a known issue and there's a fix in place It looks like pex v2.1.154 should have the fix but I'm not sure which version of pex pants is using
10 Views
Open in Slack
PreviousNext
Powered by