@hundreds-father-404 is anyone else on the osx wheel building error right now cause i can pick that up

No. I only had time yesterday to do that initial discovery and think John might be afk. Would be very helpful if you had time to look into it. Happy to help throughout 🙂

then that is currently what i am being paid to do today

thank you sir.

currently no, as i'm not able to reproduce the issue in my own osx environment and

PEX_VERBOSE=9

has produced no clues yet when i run it through travis

is the PEX_VERBOSE not actually making it through to the relevant command?

it totally is

mm

but i cannot see why it won't resolve future since that should make the pex command fail and

set -e

is on, so that shouldn't happen, but this is being run inside a process output substitution with

$()

, and also one instance of it is piping its output so

set -o pipefail

needed to be turned on. goal right now is just to get the script to fail when the resolution fails

thanks for investigating.

haven't figured out why the script isn't failing yet, but i believe what might explain the resolution error is this excerpt: https://travis-ci.org/pantsbuild/pants/jobs/521515280

pex: creating PythonIdentity from id string: cp cp36m 36 3 6 8
pex: creating PythonIdentity from id string: cp cp36m 36 3 6 8
pex: creating PythonIdentity from id string: cp cp36m 36 3 6 8
pex: creating PythonIdentity from id string: cp cp36m 36 3 6 8
pex: creating PythonIdentity from id string: cp cp36m 36 3 6 8
pex: creating PythonIdentity from id string: cp cp27m 27 2 7 12
pex: creating PythonIdentity from id string: cp cp27m 27 2 7 12
pex: creating PythonIdentity from id string: cp cp27m 27 2 7 12
pex: creating PythonIdentity from id string: cp cp27m 27 2 7 10
pex: Could not identify /usr/bin/python2.6: received exit code 1 during execution of `['/usr/bin/python2.6', '-sE']` while trying to execute `['/usr/bin/python2.6', '-sE']`
pex: creating PythonIdentity from id string: cp cp27m 27 2 7 10
pex: Constraints on interpreters: ['>=3.4,<4', '==2.7.*'], Matching Interpreter: /Users/travis/.pants_pyenv/versions/3.6.8/bin/python3.6
pex: Constraints on interpreters: ['>=3.4,<4', '==2.7.*'], Matching Interpreter: /usr/bin/python
pex: Constraints on interpreters: ['>=3.4,<4', '==2.7.*'], Matching Interpreter: /Users/travis/.pants_pyenv/shims/python
pex: Constraints on interpreters: ['>=3.4,<4', '==2.7.*'], Matching Interpreter: /usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/bin/python2.7
pex: Selecting runtime interpreter based on pexrc: 999.7ms
pex: Re-executing: cmdline="['/usr/bin/python', '/var/folders/my/m6ynh3bn6tq06h7xr3js0z7r0000gn/T/build_pex.XXXXX.x8lsnftO/pex', 'future==0.17.1', 'beautifulsoup4>=4.6.0,<4.7', "configparser==3.7.1 ; python_version<'3'", "subprocess32==3.2.7 ; python_version<'3'", '--', '/Users/travis/build/pantsbuild/pants/src/python/pants/releases/packages.py', '--py3', 'build_and_print', '1.16.0.dev1+gitebd34c13']", sys.executable="/Users/travis/.pants_pyenv/versions/3.6.8/bin/python", PEX_PYTHON="None", PEX_PYTHON_PATH="None", COMPATIBILITY_CONSTRAINTS="['>=3.4,<4', '==2.7.*']"

here it says "re-executing" and then explicitly uses

/usr/bin/python

instead of e.g. the pyenv one at

/Users/travis/.pants_pyenv/versions/3.6.8/bin/python3.6

. so i'm gonna look into how pex figures out what to "re-execute"

So it looks like the issue is resolving the

future

requirement in

release.sh

from here https://github.com/pantsbuild/pants/blob/master/build-support/bin/release.sh#L104? That’s new to me - didn’t think about that and assumed the issue was the wheel used when running

ci.sh -b

.

yes

I think this PR might be what’s causing the issue https://github.com/pantsbuild/pex/pull/701/files. We now include

sys.executable

in the valid interpreters if

PEX_PYTHON

and

PEX_PYTHON_PATH

are left off. It seems maybe resolution is being inconsistent because now there are multiple interpreters to choose from?

that sounds like it would be exactly it, taking a look

Hm maybe that’s a false lead. We set

PEX_PYTHON_PATH

for OSX UCS4, and it’s still failing to resolve

future

. Worth checking though.

if i have a pex change that affects pants, how do i get my changed pex code into travis?

There isn’t any good way I know of 😞 best I could imagine is to clone Pex and point

release.sh

to your clone here: https://github.com/pantsbuild/pants/blob/master/build-support/bin/release.sh#L91. Although I doubt that would work because this is pointing to the Pex binary we release, not a source code file Usually we’ve identified what the issue is through reasoning, upstream it to Pex, re-release Pex, pull it down to Pants, and repeat if it didn’t actually work

if you check a binary into the repo with a tagged version (

my-pex-version==1.6.6+specialest-1

), you can load it via

--python-repos-repos=$somedir

oh, that's much easier

I'm flying today but can look at this if it's still unresolved. Should have a bit of time here in an hour or so and then tonight.

@enough-analyst-54434: i think that that would be appreciated, but @aloof-angle-91616 has more context

it would be appreciated, i am still working on that. current status is just https://travis-ci.org/pantsbuild/pants/jobs/521552427 (with

set -x

and

PEX_VERBOSE=9

) -- i've been pushing to https://github.com/pantsbuild/pants/tree/try-fix-silent-osx-wheel-build-error-for-1.16.x at pantsbuild/pants instead of making a PR yet for no particular reason. i had the one hunch that putting

PythonInterpreter.get()

first in the set of interpreters to match when

PEX_PYTHON_PATH

was unset (the change from pex 1.6.6 that @hundreds-father-404 pointed out) might have been causing pex to use

/usr/bin/python

instead of our pyenv python, and switched that around in https://github.com/cosmicexplorer/pex/tree/put-self-interpreter-at-end-of-path, but that didn't fix the issue (see travis link). using that pex branch did however finally make the release script fail (because despite having

set -e

, it refuses to exit when

future==0.17.1

resolution fails), but that's only because the other command to find the wheel failed, so that error is unrelated to the original issue. it would be super useful if we could understand why failing to resolve

future==0.17.1

doesn't cause the release script to exit (which is why this was allowed to start failing in the first place), and if not that (since i didn't have any luck with that) why the resolution is failing (i am under the impression this could be because we're resolving requirements for a different interpreter than we're expecting to, but can't figure out which).

Thanks Danny.

let me know if there's anything i can do in parallel, or just drop off your progress whenever you stop. got a late start today and i have an appointment from 3:30-5pm california time but otherwise this is my focus for today

Tracking here: https://github.com/pantsbuild/pants/issues/7593

So I hit something similar to this internally awhile back

Slurping up an interpreter with too old openssl linked?

well, I was thinking about hitting surprising future resolution bugs during bootstrap time

OK. Well the number one superbug of pex is horrible resolution error messages. They can map to any number of real issues.

I do have that complaint as well - thank god for verbose

k

I exported the paths to my constructed virtualenv and got past

It's easy enough to constrain we just were doing it wrong. Pyenv shims are for humans, not scripts.

-vvvvvvvvv

being more verbose than

PEX_VERBOSE=9

threw me off a bit

yeah, agree

Danny - it is not - sortof - it is subtle.

not that homebrew makes this easy either

PEX_VERBOSE=9 makes running any pex max verbose.

I am pushing us to abandon OSX completely, I am sick of it

When your pex cli tool is itself packaged as a pex, you can use both to get all the logging.

that makes sense

I just exported the opt-out, I think

@aloof-angle-91616 fwiw - the releases.sh not exiting early / failing was due to the perhaps surprising fact that

set -e

doesn't work like you might expect inside function bodies, where you must track exit codes yourself and exit if you want that.

that makes more sense

I don't know what you're referring to. They are only wrapped in a subshell when we need their stdout.

i was thinking like how there are the parens wrapping the function body here:

function run_pex() {
  # TODO: Cache this in case we run pex multiple times
  (
    PEX_VERSION="$(requirement pex | sed -e "s|pex==||")"

    pexdir="$(mktemp -d -t build_pex.XXXXX)"
    trap "rm -rf ${pexdir}" EXIT

    pex="${pexdir}/pex"

    curl -sSL "${PEX_DOWNLOAD_PREFIX}/v${PEX_VERSION}/pex" > "${pex}"
    chmod +x "${pex}"
    "${pex}" "$@"
  )
}

At any rate, I'm pretty sure the reason I don't tend to use set -e is exactly this. If you use alot of functions you need to be && and || and exiting and if you need to keep this discipline in functions I found it easier in the past to just keep the discipline in the whole script.

port to rust^H^H^H^Hpython

yeah, i can't see a better way than that given the (maybe reasonable)

set -e

behavior

Yeah. Well, you've got to understand any language you use, but bash is both hard to understand and not well understood and too convenient. A bad combo.

and also apple and some other vendors won't update their preinstalled versions because of the gpl v3

i was thinking like how there are the parens wrapping the function body here

Those are actually superfluous. You have to re-dclare

set -e

inside those parens to get the desired effect.

set -e

does not propagate to subshells.

true. but it also means you could

exit 1

inside of it to exit early without killing the whole script

(by experiment just now - I know bash just as well as about anyone, not that)

i definitely do not know what propagates to subshells by heart