https://pantsbuild.org/ logo
#pex
Title
h

hundreds-father-404

02/16/2021, 11:07 PM
Bummer, new venv mode broke running Black over >1 file for me. Going back to
652df2c
works when running
./pants lint src/python/pants/util/*.py
, but master branch fails with:
Untitled
cc @enough-analyst-54434 yay macOS...https://github.com/pantsbuild/pex/issues/1235. At least this reproduces with strictly Pex, no Pants
e

enough-analyst-54434

02/16/2021, 11:45 PM
Why do you think this is a Pex issue?
h

hundreds-father-404

02/16/2021, 11:46 PM
Vs a Black issue? It could be, although it's only happening when running via
--venv
mode, not
--unzip
and zip app
e

enough-analyst-54434

02/16/2021, 11:48 PM
OK - Its not at all clear Pex has anything to do with this. Since there are 0 concurrency issues while Pex is running, the backtrace is pure venv.
👍 1
I'll dig a bit and probably push back on the issue without more proof.
@witty-crayon-22786 and @hundreds-father-404 please try https://github.com/pantsbuild/pants/pull/11567 when you get a chance. Alternatively:
Copy code
pex -c black black==20.b1 --venv -o black.pex
PEX_TOOLS=1 ./black.pex venv black.venv
black.venv/bin/black pex/*.py
The error is in multiprocessing and mp has weird rules surrounding
__main__
- the bin scripts in a venv all have a
__name__
guard on
__main__
.
w

witty-crayon-22786

02/17/2021, 12:38 AM
mm, so Eric pinged me about this and i added some details to https://github.com/pantsbuild/pex/issues/1235, in case you hadn’t seen them.
@enough-analyst-54434: and that would be pex 2.1.30?
e

enough-analyst-54434

02/17/2021, 12:39 AM
I di - unuseful for someone without macOS.
👍 1
Well - yeah. You two claimed
pex ..
foo on that ticket
So whatever pex you both used.
w

witty-crayon-22786

02/17/2021, 12:41 AM
`20.8b1`*
one sec.
h

hundreds-father-404

02/17/2021, 12:42 AM
Yay, your Pants Pex fixes it! @witty-crayon-22786 nb that you should set
--interpreter-constraint='CPython==3.9.*'
for the Pex command and
PANTS_BLACK_INTERPRETER_CONSTRAINTS='["CPython>=3.9"]'
for Pants
w

witty-crayon-22786

02/17/2021, 12:42 AM
red herring
e

enough-analyst-54434

02/17/2021, 12:42 AM
binary is the Benjy bug
👍 1
Please
rm -rf ~/.pex/isolated
w

witty-crayon-22786

02/17/2021, 12:43 AM
that worked, thanks
e

enough-analyst-54434

02/17/2021, 12:44 AM
Worked as in black now works?
Yay, your Pants Pex fixes it!
Eric - the PR fixes it or the
PEX_TOOLS=1 ...
experiment?
w

witty-crayon-22786

02/17/2021, 12:44 AM
success:
Copy code
$ ~/src/venvs/pex-2.1.30/bin/pex -c black black==20.8b1 --interpreter-constraint='CPython==3.9.*' --venv -o black.pex
$ PEX_TOOLS=1 ./black.pex venv black.venv
$ black.venv/bin/black ~/src/pex/**/*.py
All done! ✨ 🍰 ✨
61 files left unchanged.
e

enough-analyst-54434

02/17/2021, 12:45 AM
OK - So PEX_TOOLS=1 POC works. That implies PR works.
Which also implies its a
__name__ == '__main__'
issue.
h

hundreds-father-404

02/17/2021, 12:46 AM
Both fix it. The PR works for me, and the Pex snippet works
Thanks for digging into this John!
1
e

enough-analyst-54434

02/17/2021, 12:48 AM
Could one of you try hacking such a thing in in your named_caches venv pex script?
h

hundreds-father-404

02/17/2021, 12:53 AM
Happy to, although I'm not following what that guard would look like? Sorry, I feel obtuse here
e

enough-analyst-54434

02/17/2021, 12:54 AM
Copy code
if __name__ != '__main__':
  return
👍 1
The opposite of the usual guard just to save a mega-indent as you are wont.
h

hundreds-father-404

02/17/2021, 1:00 AM
can't return because it's not a function, but I did this on what corresponds to line 150:
Copy code
11 if __name__ != "__main__":
 12     sys.exit(0)
Resulting in:
Copy code
error: cannot format /private/var/folders/hm/qjjq4w3n0fsb07kp5bxbn8rw0000gn/T/process-executionAkW4J8/src/python/pants/pantsd/pants_daemon.py: A process in the process pool was terminated abruptly while the future was running or pending.
This is on the master branch. Lmk if it'd be helpful to VC about this
e

enough-analyst-54434

02/17/2021, 1:12 AM
How about just try a vanilla
if __name__ == '__main__':
guard just under the shebang an indent the whole file under that.
"Safe importing of main module" is what we're working around here if you can take some time to fiddle: https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods If @#$!ing Apple would license their #$@ OS to be used under a vm running on Linux I could be more useful. They won't take my money though.
🤑 1
👀 1
h

hundreds-father-404

02/17/2021, 1:17 AM
Okay great, this worked on the master branch!
How about just try a vanilla if name == '__main__': guard just under the shebang an indent the whole file under that.
e

enough-analyst-54434

02/17/2021, 1:17 AM
@#$! OK. Thanks for your help. I'll spin up a Pex fix for this.
🙌 1
@hundreds-father-404 one more favor if you can:
Copy code
for py in python3.{8,9}; do echo -n "$py: " && $py -c 'import multiprocessing; print(multiprocessing.get_start_method())'; done
python3.8: fork
python3.9: fork
You should see a
spawn
for python3.9.
h

hundreds-father-404

02/17/2021, 1:49 AM
Confirmed that 3.7 is fork and 3.9 is spawn (I can't get 3.8 to install via pyenv because arm arch 🙃)
e

enough-analyst-54434

02/17/2021, 1:54 AM
OK - thank you very much.
❤️ 1