Escalating this We appear to have a major problem building w Pants #development

Join Slack

Escalating this: We appear to have a major problem...

# development

happy-kitchen-89482

06/30/2022, 3:16 AM

Escalating this: We appear to have a major problem building wheels in CI

➕ 1

happy-kitchen-89482

06/30/2022, 3:16 AM

See https://github.com/pantsbuild/pants/pull/16012

happy-kitchen-89482

06/30/2022, 3:16 AM

That is a dummy commit with no changes, on top of recent main

happy-kitchen-89482

06/30/2022, 3:18 AM

This may only happen in PR builds, the main branch build without thew dummy commit works: https://github.com/pantsbuild/pants/actions/runs/2586928473

happy-kitchen-89482

06/30/2022, 3:19 AM

It happens on Linux x86 and MacOS x86, but not on MacOS ARM64

happy-kitchen-89482

06/30/2022, 3:21 AM

The error on linux x86 is a segfault:

Copy code

CalledProcessError(-11, ('./pants', '--no-dynamic-ui', '--concurrent', 'package', 'src/python/pants:pants-packaged', 'src/python/pants/testutil:testutil_wheel'))

happy-kitchen-89482

06/30/2022, 3:21 AM

The error on macOS x86 is whatever exit code -4 designates:

Copy code

CalledProcessError(-4, ('./pants', '--no-dynamic-ui', '--concurrent', 'package', 'src/python/pants:pants-packaged', 'src/python/pants/testutil:testutil_wheel'))

happy-kitchen-89482

06/30/2022, 3:23 AM

I'm fairly stumped

fast-nail-55400

06/30/2022, 3:32 AM

is it OOM situation?

hundreds-father-404

06/30/2022, 3:34 AM

i'm stumped trying to reason about what has changed in the last 24 hours. Maybe we should try rebuilding jobs that were succeeding 24 hours ago? And why did some of the PRs today not fail?

careful-address-89803

06/30/2022, 4:11 AM

One difference I see is that if it's a PR then we build the rust in debug mode

[[ "${GITHUB_EVENT_NAME}" == "pull_request" ]] && export MODE=debug

hundreds-father-404

06/30/2022, 4:16 AM

ohhh that's interesting!! that it's only PR builds failing. Great suggestoin!

hundreds-father-404

06/30/2022, 4:17 AM

and that would explain why CI worked for some PRs, they used

[ci skip-build-wheels]

careful-address-89803

06/30/2022, 4:21 AM

Also I think it's unlikely that it's an OOM. We're getting CalledProcessError -11 indicating SIGSEV. The OOM reaper usually sends signal 9 SIGKILL and I don't think it's possible to trap it

careful-address-89803

06/30/2022, 4:38 AM

also the docker image we use was last updated 2022-26T11:00, 4 days ago

❗ 1

careful-address-89803

06/30/2022, 5:16 AM

so good news (?), I get a repro in the docker container by just walking through the script. failure is on the step

./build-support/bin/release.sh build-local-pex

hundreds-father-404

06/30/2022, 5:18 AM

wow awesome! thank you so much for digging into this

hundreds-father-404

06/30/2022, 5:18 AM

only when using MODE=debug?

careful-address-89803

06/30/2022, 5:20 AM

still looking into it, I'm trying next with version n-1 of the docker container. I'm not really a rustacean but I do know about people changing dependencies and the build breaking

careful-address-89803

06/30/2022, 5:46 AM

ok, well, good news? I also get a repro on the

<http://quay.io/pypa/manylinux2014_x86_64:2022-06-20-d72b943|quay.io/pypa/manylinux2014_x86_64:2022-06-20-d72b943>

image. it has the same filesize as the current one, so that makes some sense. with the optimised build it too runs. trying with n-2, which has a different filesize

careful-address-89803

06/30/2022, 5:55 AM

wild, so,

<http://quay.io/pypa/manylinux2014_x86_64:2022-06-13-c365205|quay.io/pypa/manylinux2014_x86_64:2022-06-13-c365205>

also has the problem where unoptimised builds throw a sigsev. So, I think this might be something in our code itself? I've got to pack it in for the night. repro with docker

docker run --rm --name pants -it <http://quay.io/pypa/manylinux2014_x86_64:latest|quay.io/pypa/manylinux2014_x86_64:latest> bash

and then

Copy code

git clone <https://github.com/pantsbuild/pants.git>
git config --global safe.directory /pants
pushd /pants
curl --proto ''=https'' --tlsv1.2 -sSf <https://sh.rustup.rs> | sh -s -- -v -y --default-toolchain none
export PATH=${PATH}:${HOME}/.cargo/bin
export PATH=${PATH}:/opt/python/cp37-cp37m/bin:/opt/python/cp38-cp38/bin:/opt/python/cp39-cp39/bin
export MODE=debug

./build-support/bin/release.sh build-local-pex

My next step would be to git-bisect, but I probably won't have time before friday.

curved-television-6568

06/30/2022, 9:13 AM

Not sure if this is related or not, but I’ve just started seeing issues with running

./pants check

on more than a few files. Not done a full bisect yet, but

2.12.x

is OK, while

2.13.x

exhibits this issue. The end of a trace run without pantsd looks like:

Copy code

11:11:25.38 [DEBUG] Running Run MyPy on 13 files. under semaphore with concurrency id: 8, and concurrency: 1
11:11:25.38 [TRACE] The request is not nailgunnable! Short-circuiting to regular process execution
11:11:25.38 [DEBUG] Starting: Run MyPy on 13 files.
11:11:25.38 [DEBUG] Starting: setup_sandbox
Illegal instruction: 4

Command was:

Copy code

$ ./pants --no-pantsd --no-dynamic-ui -ltrace check src/python/pants/backend/docker/util_rules:

curved-television-6568

06/30/2022, 9:32 AM

This will be my first

git bisect

session, ever! TIL 😄 what a cool git feature

average-vr-56795

06/30/2022, 9:37 AM

Looks like the original one is maybe blowing out the stack in debug mode?

Copy code

#0  0x00007f0dd4c47c02 in __rust_probestack () from /Users/dwh/src/github.com/pantsbuild/pants/src/python/pants/engine/internals/native_engine.so
#1  0x00007f0dd3ed1442 in _$LT$alloc..vec..into_iter..IntoIter$LT$T$C$A$GT$$u20$as$u20$core..iter..traits..iterator..Iterator$GT$::next::hb3306f8227d7214a (self=0x0)
    at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/alloc/src/vec/into_iter.rs:143
#2  0x00007f0dd3ed50fe in core::iter::traits::iterator::Iterator::try_fold::h59fdcac3e0f60533 (self=0x7f0dcfe30e28, init=..., f=...) at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/iter/traits/iterator.rs:2185
#3  0x00007f0dd3f98ab6 in _$LT$core..iter..adapters..map..Map$LT$I$C$F$GT$$u20$as$u20$core..iter..traits..iterator..Iterator$GT$::try_fold::h4f928108bf44f512 (self=0x7f0dcfe30e28, init=..., g=...)
    at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/iter/adapters/map.rs:117
#4  0x00007f0dd3f9ebe2 in _$LT$I$u20$as$u20$alloc..vec..in_place_collect..SpecInPlaceCollect$LT$T$C$I$GT$$GT$::collect_in_place::hfdebe04bb5589b35 (self=0x7f0dcfe30e28, dst_buf=0x7f0db1516300, end=0x7f0db151be80)
    at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/alloc/src/vec/in_place_collect.rs:251
#5  0x00007f0dd3edf2c9 in alloc::vec::in_place_collect::_$LT$impl$u20$alloc..vec..spec_from_iter..SpecFromIter$LT$T$C$I$GT$$u20$for$u20$alloc..vec..Vec$LT$T$GT$$GT$::from_iter::hd20db111ec09bed6 (iterator=...)
    at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/alloc/src/vec/in_place_collect.rs:178
#6  0x00007f0dd3ee3141 in _$LT$alloc..vec..Vec$LT$T$GT$$u20$as$u20$core..iter..traits..collect..FromIterator$LT$T$GT$$GT$::from_iter::hfe20499127823b81 (iter=...)
    at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/alloc/src/vec/mod.rs:2554
#7  0x00007f0dd3f99b39 in core::iter::traits::iterator::Iterator::collect::h9ee1894b18fe5bd3 (self=...) at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/iter/traits/iterator.rs:1784
#8  0x00007f0dd3fa7f87 in _$LT$alloc..boxed..Box$LT$$u5b$I$u5d$$GT$$u20$as$u20$core..iter..traits..collect..FromIterator$LT$I$GT$$GT$::from_iter::h271b6de0f310c330 (iter=...)
    at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/alloc/src/boxed.rs:1889
#9  0x00007f0dd3f99a32 in core::iter::traits::iterator::Iterator::collect::h53ab4d6af960ad4a (self=...) at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/iter/traits/iterator.rs:1784
#10 0x00007f0dd3f0fc9f in futures_util::future::try_join_all::try_join_all::he289352d4da757d1 (i=...) at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-util-0.3.21/src/future/try_join_all.rs:91
#11 0x00007f0dd4006362 in store::Store::materialize_directory_helper::_$u7b$$u7b$closure$u7d$$u7d$::hd6bdc5abddf1b396 () at fs/store/src/lib.rs:1183
#12 0x00007f0dd3efbf3a in _$LT$core..future..from_generator..GenFuture$LT$T$GT$$u20$as$u20$core..future..future..Future$GT$::poll::h048fdbb04612eed1 (self=..., cx=0x7f0dd00025c8)
    at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/future/mod.rs:91
#13 0x00007f0dd3fbe8af in _$LT$core..pin..Pin$LT$P$GT$$u20$as$u20$core..future..future..Future$GT$::poll::hc671042fa21ab1eb (self=..., cx=0x7f0dd00025c8)
    at /rustc/fe5b13d681f25ee6474be29d748c65adcd91f69e/library/core/src/future/future.rs:124
#14 0x00007f0dd400761f in store::Store::materialize_directory_helper::_$u7b$$u7b$closure$u7d$$u7d$::_$u7b$$u7b$closure$u7d$$u7d$::h86eb3fdb07529c3f () at fs/store/src/lib.rs:1177

curved-television-6568

06/30/2022, 10:54 AM

I’ve reverted a commit, and that PR builds wheels OK in CI: https://pantsbuild.slack.com/archives/C0D7TNJHL/p1656584061779709?thread_ts=1656582894.343629&cid=C0D7TNJHL

happy-kitchen-89482

06/30/2022, 1:35 PM

Oh wow, all this great debugging happened while I was asleep

🪄 2

😴 2

happy-kitchen-89482

06/30/2022, 1:35 PM

Thanks for digging in everyone!

fast-nail-55400

06/30/2022, 1:37 PM

I cherry-picked this PR to

2.13.x

as well. Probably need to box that future now (for

load_bytes_with

fast-nail-55400

06/30/2022, 1:37 PM

But revert first and ask question later then.

fast-nail-55400

06/30/2022, 1:38 PM

Also https://github.com/pantsbuild/pants/pull/15996 will need to be reverted as well as it stacks on top.

happy-kitchen-89482

06/30/2022, 1:39 PM

@fast-nail-55400 can you get out a clean revert PR?

happy-kitchen-89482

06/30/2022, 1:39 PM

With both?

fast-nail-55400

06/30/2022, 1:40 PM

yes

fast-nail-55400

06/30/2022, 1:43 PM

https://github.com/pantsbuild/pants/pull/16016

happy-kitchen-89482

06/30/2022, 1:50 PM

Thanks!

fast-nail-55400

06/30/2022, 2:00 PM

revert from 2.13.x: https://github.com/pantsbuild/pants/pull/16017

fast-nail-55400

06/30/2022, 2:57 PM

revert landed on

main

✅ 1

fast-nail-55400

06/30/2022, 4:57 PM

revert landed earlier on

2.13.x

as well

happy-kitchen-89482

06/30/2022, 5:11 PM

Thanks!!

Open in Slack

Previous Next