pantsbuild #development

Channels

welcome

github-notifications

announce

average-vr-56795

10/05/2018, 4:20 PM

I’ve played with a lot of ideas in my head, and I’m pretty sure those are the two options which don’t need use to horribly special-case roots during execution

witty-crayon-22786

10/05/2018, 4:24 PM

@average-vr-56795: thanks, replied!

witty-crayon-22786

10/07/2018, 2:34 AM

i spent a few hours today tracking down https://github.com/pantsbuild/pants/issues/6565 , and determined that there is both good and bad news. the good news: i know precisely what is happening, and have a tiny deterministic repro. the bad news: everything we thought we knew about forking with locks is wrong.

witty-crayon-22786

10/07/2018, 2:35 AM

the bad news is also good news, in that a few mysteries are cleared up. but... i'm going to have to do some more thinking about the threading model tomorrow/monday.

aloof-angle-91616

10/07/2018, 2:35 AM

oh i was going to mention

aloof-angle-91616

10/07/2018, 2:36 AM

i had this deterministic repro of this hang on control-c (so a SIGTERM) on my linux laptop but it was in a branch which did tons of random other stuff

aloof-angle-91616

10/07/2018, 2:38 AM

and i added faulthandler nonsense and it was always blocked in two threads both in

store_gc_service.py

one in

_extend_lease()

and one in

_garbage_collect()

. i had a gdb stacktrace too i cannot find it somehow

aloof-angle-91616

10/07/2018, 2:39 AM

but basically the deadlock also extended to the two methods in

<http://store.rs|store.rs>

, one which extended the lease of some files, and the other which was garbage collecting them let me find the names at least

aloof-angle-91616

10/07/2018, 2:41 AM

well it was

garbage_collect_store

and

lease_files_in_graph

were the ones deadlocked but it was just every time it was deadlocked it was always those two threads in python on those methods and those two threads also in rust also in those methods and there was this one scary TODO comment that i will find

witty-crayon-22786

10/07/2018, 2:44 AM

those functions have python sleeps in them, so i think that those thread stacks might have been red herrings, with the hang occurring somewhere else. but i wouldn't rule it out given what i just confirmed.

aloof-angle-91616

10/07/2018, 2:47 AM

oh you have the exact thing i saw

aloof-angle-91616

10/07/2018, 2:47 AM

the stack trace is the same

witty-crayon-22786

10/07/2018, 2:47 AM

just posted.

aloof-angle-91616

10/07/2018, 2:48 AM

Copy code

Thread 0x00007000062f8000 (most recent call first):
File "/Users/stuhood/src/pants/src/python/pants/pantsd/service/store_gc_service.py", line 45 in _garbage_collect
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

Thread 0x0000700005ef5000 (most recent call first):
File "/Users/stuhood/src/pants/src/python/pants/pantsd/service/store_gc_service.py", line 41 in _extend_lease
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 754 in run
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 801 in __bootstrap_inner
File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 774 in __bootstrap

aloof-angle-91616

10/07/2018, 2:48 AM

was all i was referring to

witty-crayon-22786

10/07/2018, 2:48 AM

yep, me too. those lines are

sleep

lines, i think.

aloof-angle-91616

10/07/2018, 2:48 AM

yeah

witty-crayon-22786

10/07/2018, 2:48 AM

whelp, anyway. i'm going to go throw my computer out the window.

aloof-angle-91616

10/07/2018, 2:48 AM

i don't know that that explains the rust stacktrace i conveniently can't find

witty-crayon-22786

10/07/2018, 2:49 AM

see what i posted on the ticket.

aloof-angle-91616

10/07/2018, 2:49 AM

witty-crayon-22786

10/07/2018, 2:49 AM

have a good night!

aloof-angle-91616

10/07/2018, 2:53 AM

i posted something that i had open in another tab

aloof-angle-91616

10/07/2018, 2:53 AM

i always have great nights

aloof-angle-91616

10/08/2018, 6:28 AM

apparently bazel’s calling their excuse for not having good python java interop “””starlark””” now

aloof-angle-91616

10/08/2018, 6:29 AM

there’s still lead in flint’s water and people aren’t gonna die from it now they’ll slowly be lobotomized because they don’t have access to anything cleaner

aloof-angle-91616

10/08/2018, 6:29 AM

that’s the analogy

aloof-angle-91616

10/08/2018, 6:33 AM

(compared to python, or any EXISTING, ACTUAL programming language, which could be water. i could be the last airbender. i was gonna say i wouldn’t cut off my hair cause there’d be nothing to dye but nora has soundly disproven this line of reasoning)

average-vr-56795

10/08/2018, 5:52 PM

@happy-kitchen-89482 I’m interested in and excited by your oauth subsystem! In particular, I’m curious about how/where it will cache tokens and expiry information (and how that may interplay with the daemon)…

happy-kitchen-89482

10/08/2018, 6:02 PM

I wasn’t sure, which is why I left it to one side and did basic auth first. In that case, you basicauth once, receive a session cookie, and then send that on every request, instead of re-authing every request. So in that case the thing we’re caching is a cookie file, which contains tokens and expiry info and is well understood.