https://pantsbuild.org/ logo
Join Slack
Powered by
...sh*t. requests uses urllib3. dangit. i thought ...
# announce
w
...sh*t. requests uses urllib3. dangit. i thought that that was the stdlib version.
(i guess i'm glad that this isn't openssl failing to load or something, but sheesh.)
h
Ah yes this sounds right. Think requests is just a nice wrapper API around urrlib3
You agree this all broke because of an external / upstream issue, right? None of our code changes seem to have caused anything
w
Yea, pretty sure. Pypi status page is still fine, but we're hanging attempting to establish an https connection to pypi.org
h
I don’t understand why this is only impacting Linux also 🤔 maybe it does have something to do with OpenSSL or something specific to Centos6 Example scenario, PyPi upgraded their requirement to be more stringent and now our centos6 OpenSSL is outdated so works but is slower than it used to be
Hm nothing pops up in their GitHub changelog https://github.com/pypa/warehouse/commits/master
w
@hundreds-father-404: have you published a new 
pex
to consume for local/travis testing from 
pants
before?
h
I have not. Didn’t know how to do it. FYI the release guide https://github.com/pantsbuild/pex/blob/master/RELEASE.rst
w
thanks. might try monkey patching.
👍 1
h
What’s your hypothesis?
w
that pypi is crazy flaky right now, and because we don't have a connect timeout set, we're just hanging forever trying to establish a connection
h
That sounds reasonable to me. Even if that isn’t the case, a timeout seems like a good mechanism to implement. I’m afk tonight but please feel free to add me to reviews and ping with questions
e
Let me note Linux is the last place we use shims. Are we sure the python we think we're running is the python we're running?
w
i am not sure. but afaict this is 
urllib3
, a library from pypi
(rather than the stdlib copy)
e
Yes, urllib3 is the core of requests. Afaict, it's only used by requests.
w
so i'm not 100% sure what the connection to the python in use would be
... still openssl?
e
I have ~0 background on what you're fighting, but pyenv via shims is prone to not work in CI. When it doesn't you can get a python with old openssl.
Observation only.
w
ok, thank you. good to know.
h
I don’t think the issue is pyenv shims because these shards use Centos6, where we no longer use shims
e
Ok, but centos6 has too old openssl by default. The other shimmy thing is the ... forget the name weird tool used in centos6 to activate package sets. That may not be working as expected.
w
OH. so one other important datapoint: not all attempts to open https connections hang
which would point pretty strongly to flakiness
e
Well. If you can prove a stable python interpreter across https flakes, it seems likely pypi.
w
search for:
Copy code
Starting new HTTPS connection
in https://travis-ci.org/pantsbuild/pants/jobs/528027586
two attempts to connect... first succeeds
they're in separate processes, but
e
My search landed on a line with /shims/... FTR
w
hm... your search for 
Starting new HTTPS connection
?
e
So, I want to believe Eric, but I provisionally don't
w
both of the
Copy code
Starting new HTTPS connection
lines are proceeded by references to the shims, but one succeeds
e
All I'm saying is shims are suspect.
👌 1
w
ok, thanks!
h
Even tho I doubt its shims, certainly worth sanity checking
w
going to see if my various attempts were more/less flaky.
if it is actually deterministic, will look at that
h
Sg. Thanks for taking the lead here!
w
...yea, it's flaky. i had 1 successful connection in one case, 20 in another, and a completely successful run in a third
👍 1
(the linux bit is inexplicable and annoying though)
e
If the centos6 image contains two pythons with == x.y.z then it could be the case that the one picked is random. And one might have good openssl linked, one bad.
Wild guess
Basically, printing sys.executable next to each fetch would be good but perhaps hard.
w
the fetches are occurring "in" pants, afaict. because the debug logging is taking effect
so i can do that.
e
For my idea to make sense, would require two separate subprocess invocations of PEX so it can pick 2x
w
but i think that also means that the 
shims
line isn't (directly) affecting what pants is running under... that would be... the bootstrap pex...?
h
The other reason I think that’s unlikely (tho maybe still possible) is that centos6 doesn’t come with Py36. We install it ourselves via pyenv. The release script requires Py36 so shouldn’t be any ambiguity on the Py36 shard, although Py27 that could be happening
e
If that's not the case, my idea is bunk
w
k
annnnd off we go with connect/read timeouts set.
🎉 1
would kill for a local repro of this one.
h
(I empathize. How I felt with that abi3 PR. Killed me working on that one. Over 100 commits iirc till I could land it!)
e
Have you tried running in the centos6 docker image locally for repro?
Nominally this is easier to repro than osx issues
w
yea, i tried the image in this case... no luck with the repro
which i chalked up to network (somehow?) but which i suppose could totally be something to do with the bootstrap pexes...
@hundreds-father-404: re abi3: need a sisyphus emoji
😂 2
💯 2
2 Views
Open in Slack
PreviousNext
Powered by