Escalating this We appear to have a major problem building w

Question

Escalating this We appear to have a major problem building wheels in CI

happy-kitchen-89482 · Answer

See <https github com pantsbuild pants pull 16012>

happy-kitchen-89482 · Answer

That is a dummy commit with no changes on top of recent main

happy-kitchen-89482 · Answer

This may only happen in PR builds the main branch build without thew dummy commit works <https github com pantsbuild pants actions runs 2586928473>

happy-kitchen-89482 · Answer

It happens on Linux x86 and MacOS x86 but not on MacOS ARM64

happy-kitchen-89482 · Answer

The error on linux x86 is a segfault ```CalledProcessError 11 pants no dynamic ui concurrent package src python pants pants packaged src python pants testutil testutil wheel ```

happy-kitchen-89482 · Answer

The error on macOS x86 is whatever exit code 4 designates ```CalledProcessError 4 pants no dynamic ui concurrent package src python pants pants packaged src python pants testutil testutil wheel ```

happy-kitchen-89482 · Answer

I m fairly stumped

fast-nail-55400 · Answer

is it OOM situation

hundreds-father-404 · Answer

i m stumped trying to reason about what has changed in the last 24 hours Maybe we should try rebuilding jobs that were succeeding 24 hours ago And why did some of the PRs today not fail

careful-address-89803 · Answer

One difference I see is that if it s a PR then we build the rust in debug mode ` $ GITHUB EVENT NAME == pull request amp amp export MODE=debug`

hundreds-father-404 · Answer

ohhh that s interesting that it s only PR builds failing Great suggestoin

hundreds-father-404 · Answer

and that would explain why CI worked for some PRs they used ` ci skip build wheels `

careful-address-89803 · Answer

Also I think it s unlikely that it s an OOM We re getting CalledProcessError 11 indicating SIGSEV The OOM reaper usually sends signal 9 SIGKILL and I don t think it s possible to trap it

careful-address-89803 · Answer

also the docker image we use was last updated 2022 26T11 00 4 days ago

careful-address-89803 · Answer

so good news I get a repro in the docker container by just walking through the script failure is on the step ` build support bin release sh build local pex`

hundreds-father-404 · Answer

wow awesome thank you so much for digging into this

hundreds-father-404 · Answer

only when using MODE=debug

careful-address-89803 · Answer

still looking into it I m trying next with version n 1 of the docker container I m not really a rustacean but I do know about people changing dependencies and the build breaking

careful-address-89803 · Answer

ok well good news I also get a repro on the `<http quay io pypa manylinux2014 x86 64 2022 06 20 d72b943|quay io pypa manylinux2014 x86 64 2022 06 20 d72b943>` image it has the same filesize as the current one so that makes some sense with the optimised build it too runs trying with n 2 which has a different filesize

careful-address-89803 · Answer

wild so `` also has the problem where unoptimised builds throw a sigsev So I think this might be something in our code itself I ve got to pack it in for the night repro with docker `docker run rm name pants it bash` and then ```git clone git config global safe directory pants pushd pants curl proto =https tlsv1 2 sSf | sh s v y default toolchain none export PATH=$ PATH $ HOME cargo bin export PATH=$ PATH opt python cp37 cp37m bin opt python cp38 cp38 bin opt python cp39 cp39 bin export MODE=debug build support bin release sh build local pex``` My next step would be to git bisect but I probably won t have time before friday

curved-television-6568 · Answer

Not sure if this is related or not but I ve just started seeing issues with running ` pants check` on more than a few files Not done a full bisect yet but `2 12 x` is OK while `2 13 x` exhibits this issue The end of a trace run without pantsd looks like ```11 11 25 38 DEBUG Running Run MyPy on 13 files under semaphore with concurrency id 8 and concurrency 1 11 11 25 38 TRACE The request is not nailgunnable Short circuiting to regular process execution 11 11 25 38 DEBUG Starting Run MyPy on 13 files 11 11 25 38 DEBUG Starting setup sandbox Illegal instruction 4``` Command was ```$ pants no pantsd no dynamic ui ltrace check src python pants backend docker util rules ```

curved-television-6568 · Answer

This will be my first `git bisect` session ever TIL smile what a cool git feature

average-vr-56795 · Answer

Looks like the original one is maybe blowing out the stack in debug mode ``` 0 0x00007f0dd4c47c02 in rust probestack from Users dwh src github com pantsbuild pants src python pants engine internals native engine so 1 0x00007f0dd3ed1442 in $LT$alloc vec into iter IntoIter$LT$T$C$A$GT$$u20$as$u20$core iter traits iterator Iterator$GT$ next hb3306f8227d7214a self=0x0 at rustc fe5b13d681f25ee6474be29d748c65adcd91f69e library alloc src vec into iter rs 143 2 0x00007f0dd3ed50fe in core iter traits iterator Iterator try fold h59fdcac3e0f60533 self=0x7f0dcfe30e28 init= f= at rustc fe5b13d681f25ee6474be29d748c65adcd91f69e library core src iter traits iterator rs 2185 3 0x00007f0dd3f98ab6 in $LT$core iter adapters map Map$LT$I$C$F$GT$$u20$as$u20$core iter traits iterator Iterator$GT$ try fold h4f928108bf44f512 self=0x7f0dcfe30e28 init= g= at rustc fe5b13d681f25ee6474be29d748c65adcd91f69e library core src iter adapters map rs 117 4 0x00007f0dd3f9ebe2 in $LT$I$u20$as$u20$alloc vec in place collect SpecInPlaceCollect$LT$T$C$I$GT$$GT$ collect in place hfdebe04bb5589b35 self=0x7f0dcfe30e28 dst buf=0x7f0db1516300 end=0x7f0db151be80 at rustc fe5b13d681f25ee6474be29d748c65adcd91f69e library alloc src vec in place collect rs 251 5 0x00007f0dd3edf2c9 in alloc vec in place collect $LT$impl$u20$alloc vec spec from iter SpecFromIter$LT$T$C$I$GT$$u20$for$u20$alloc vec Vec$LT$T$GT$$GT$ from iter hd20db111ec09bed6 iterator= at rustc fe5b13d681f25ee6474be29d748c65adcd91f69e library alloc src vec in place collect rs 178 6 0x00007f0dd3ee3141 in $LT$alloc vec Vec$LT$T$GT$$u20$as$u20$core iter traits collect FromIterator$LT$T$GT$$GT$ from iter hfe20499127823b81 iter= at rustc fe5b13d681f25ee6474be29d748c65adcd91f69e library alloc src vec mod rs 2554 7 0x00007f0dd3f99b39 in core iter traits iterator Iterator collect h9ee1894b18fe5bd3 self= at rustc fe5b13d681f25ee6474be29d748c65adcd91f69e library core src iter traits iterator rs 1784 8 0x00007f0dd3fa7f87 in $LT$alloc boxed Box$LT$$u5b$I$u5d$$GT$$u20$as$u20$core iter traits collect FromIterator$LT$I$GT$$GT$ from iter h271b6de0f310c330 iter= at rustc fe5b13d681f25ee6474be29d748c65adcd91f69e library alloc src boxed rs 1889 9 0x00007f0dd3f99a32 in core iter traits iterator Iterator collect h53ab4d6af960ad4a self= at rustc fe5b13d681f25ee6474be29d748c65adcd91f69e library core src iter traits iterator rs 1784 10 0x00007f0dd3f0fc9f in futures util future try join all try join all he289352d4da757d1 i= at root cargo registry src github com 1ecc6299db9ec823 futures util 0 3 21 src future try join all rs 91 11 0x00007f0dd4006362 in store Store materialize directory helper $u7b$$u7b$closure$u7d$$u7d$ hd6bdc5abddf1b396 at fs store src lib rs 1183 12 0x00007f0dd3efbf3a in $LT$core future from generator GenFuture$LT$T$GT$$u20$as$u20$core future future Future$GT$ poll h048fdbb04612eed1 self= cx=0x7f0dd00025c8 at rustc fe5b13d681f25ee6474be29d748c65adcd91f69e library core src future mod rs 91 13 0x00007f0dd3fbe8af in $LT$core pin Pin$LT$P$GT$$u20$as$u20$core future future Future$GT$ poll hc671042fa21ab1eb self= cx=0x7f0dd00025c8 at rustc fe5b13d681f25ee6474be29d748c65adcd91f69e library core src future future rs 124 14 0x00007f0dd400761f in store Store materialize directory helper $u7b$$u7b$closure$u7d$$u7d$ $u7b$$u7b$closure$u7d$$u7d$ h86eb3fdb07529c3f at fs store src lib rs 1177```

curved-television-6568 · Answer

I ve reverted a commit and that PR builds wheels OK in CI <https pantsbuild slack com archives C0D7TNJHL p1656584061779709 thread ts=1656582894 343629 amp cid=C0D7TNJHL>

happy-kitchen-89482 · Answer

Oh wow all this great debugging happened while I was asleep

happy-kitchen-89482 · Answer

Thanks for digging in everyone

fast-nail-55400 · Answer

I cherry picked this PR to `2 13 x` as well Probably need to box that future now for `load bytes with`

fast-nail-55400 · Answer

But revert first and ask question later then

fast-nail-55400 · Answer

Also <https github com pantsbuild pants pull 15996> will need to be reverted as well as it stacks on top

happy-kitchen-89482 · Answer

< fast nail 55400> can you get out a clean revert PR

happy-kitchen-89482 · Answer

With both

fast-nail-55400 · Answer

yes

fast-nail-55400 · Answer

happy-kitchen-89482 · Answer

Thanks

fast-nail-55400 · Answer

revert from 2 13 x <https github com pantsbuild pants pull 16017>

fast-nail-55400 · Answer

revert landed on `main`

fast-nail-55400 · Answer

revert landed earlier on `2 13 x` as well

happy-kitchen-89482 · Answer

Thanks