https://pantsbuild.org/ logo
Join Slack
Powered by
Bummer, new venv mode broke running Black over &gt...
# pex
h
Bummer, new venv mode broke running Black over >1 file for me. Going back to 
652df2c
works when running 
./pants lint src/python/pants/util/*.py
, but master branch fails with:
Untitled
Untitled
cc @enough-analyst-54434 yay macOS...https://github.com/pantsbuild/pex/issues/1235. At least this reproduces with strictly Pex, no Pants
e
Why do you think this is a Pex issue?
h
Vs a Black issue? It could be, although it's only happening when running via 
--venv
mode, not 
--unzip
and zip app
e
OK - Its not at all clear Pex has anything to do with this. Since there are 0 concurrency issues while Pex is running, the backtrace is pure venv.
👍 1
I'll dig a bit and probably push back on the issue without more proof.
@witty-crayon-22786 and @hundreds-father-404 please try https://github.com/pantsbuild/pants/pull/11567 when you get a chance. Alternatively:
Copy code
pex -c black black==20.b1 --venv -o black.pex
PEX_TOOLS=1 ./black.pex venv black.venv
black.venv/bin/black pex/*.py
The error is in multiprocessing and mp has weird rules surrounding 
__main__
- the bin scripts in a venv all have a 
__name__
guard on 
__main__
.
w
mm, so Eric pinged me about this and i added some details to https://github.com/pantsbuild/pex/issues/1235, in case you hadn’t seen them.
@enough-analyst-54434: and that would be pex 2.1.30?
e
I di - unuseful for someone without macOS.
👍 1
Well - yeah. You two claimed 
pex ..
foo on that ticket
So whatever pex you both used.
w
`20.8b1`*
one sec.
h
Yay, your Pants Pex fixes it! @witty-crayon-22786 nb that you should set 
--interpreter-constraint='CPython==3.9.*'
for the Pex command and 
PANTS_BLACK_INTERPRETER_CONSTRAINTS='["CPython>=3.9"]'
for Pants
w
red herring
e
binary is the Benjy bug
👍 1
Please 
rm -rf ~/.pex/isolated
w
that worked, thanks
e
Worked as in black now works?
Yay, your Pants Pex fixes it!
Eric - the PR fixes it or the 
PEX_TOOLS=1 ...
experiment?
w
success:
Copy code
$ ~/src/venvs/pex-2.1.30/bin/pex -c black black==20.8b1 --interpreter-constraint='CPython==3.9.*' --venv -o black.pex
$ PEX_TOOLS=1 ./black.pex venv black.venv
$ black.venv/bin/black ~/src/pex/**/*.py
All done! ✨ 🍰 ✨
61 files left unchanged.
e
OK - So PEX_TOOLS=1 POC works. That implies PR works.
Which also implies its a 
__name__ == '__main__'
issue.
h
Both fix it. The PR works for me, and the Pex snippet works
Thanks for digging into this John!
➕ 1
e
Could one of you try hacking such a thing in in your named_caches venv pex script?
h
Happy to, although I'm not following what that guard would look like? Sorry, I feel obtuse here
e
Copy code
if __name__ != '__main__':
  return
👍 1
The opposite of the usual guard just to save a mega-indent as you are wont.
h
can't return because it's not a function, but I did this on what corresponds to line 150:
Copy code
11 if __name__ != "__main__":
 12     sys.exit(0)
Resulting in:
Copy code
error: cannot format /private/var/folders/hm/qjjq4w3n0fsb07kp5bxbn8rw0000gn/T/process-executionAkW4J8/src/python/pants/pantsd/pants_daemon.py: A process in the process pool was terminated abruptly while the future was running or pending.
This is on the master branch. Lmk if it'd be helpful to VC about this
e
How about just try a vanilla 
if __name__ == '__main__':
guard just under the shebang an indent the whole file under that.
"Safe importing of main module" is what we're working around here if you can take some time to fiddle: https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods If @#$!ing Apple would license their #$@ OS to be used under a vm running on Linux I could be more useful. They won't take my money though.
🤑 1
👀 1
h
Okay great, this worked on the master branch!
How about just try a vanilla if name == '__main__': guard just under the shebang an indent the whole file under that.
e
@#$! OK. Thanks for your help. I'll spin up a Pex fix for this.
🙌 1
@hundreds-father-404 one more favor if you can:
Copy code
for py in python3.{8,9}; do echo -n "$py: " && $py -c 'import multiprocessing; print(multiprocessing.get_start_method())'; done
python3.8: fork
python3.9: fork
You should see a 
spawn
for python3.9.
h
Confirmed that 3.7 is fork and 3.9 is spawn (I can't get 3.8 to install via pyenv because arm arch 🙃)
e
OK - thank you very much.
❤️ 1
Open in Slack
PreviousNext
Powered by