https://pantsbuild.org/ logo
b

bitter-ability-32190

12/01/2022, 2:45 AM
From within a
PEX_INTERPRETER=1
how can I invoke the PEX's entry_point/script as if I had just invoked the pex (but still using this running Python environment)
e

enough-analyst-54434

12/01/2022, 2:51 AM
Forget Pex and answer the question like you would re invoking any entry point in a plain old Python interpreter. Do you just need to know how to find out what the entry point value is?
b

bitter-ability-32190

12/01/2022, 2:53 AM
That's an option, yes.
e

enough-analyst-54434

12/01/2022, 2:53 AM
PEX offers no code APIs so that's the only option.
What are you actually trying do if you don't mind
b

bitter-ability-32190

12/01/2022, 2:54 AM
Fix debugpy to not try and lookup the entry point by name
Ideally, we just do the debugpy setup and then let PEX do what it would've done if we didn't require setup
e

enough-analyst-54434

12/01/2022, 2:56 AM
I have 0 clues about debugpy and how it works. Are there docs I can read?
b

bitter-ability-32190

12/01/2022, 2:57 AM
There's a wiki. The TL;DR is that: ā€¢ it needs to be executed in the same environment as the code you intend to debug. Pants does this with an additional
PEX_PATH
entry on the test runner PEX ā€¢ It must be invoked as a module and then acts like a Python interpreter after it's args. E.g.
python -m debugpy <debugpy_args> <normal_py_interpreter_args>
I suspect the easiest way to cobble this together is to combine
PEX_MODULE=debugpy
with
-c '<load_the_entry_point>'
as the "python interpreter args"
e

enough-analyst-54434

12/01/2022, 2:58 AM
Does
python -m debugpy <debugpy> PEX
not work?
python PEX
does
b

bitter-ability-32190

12/01/2022, 3:01 AM
Thats what I'm trying right now, and getting strange results (on the debugpy side). I wanted to see if the entry-point way would also work so I could decide.
e

enough-analyst-54434

12/01/2022, 3:02 AM
Ok, well that's like I said. You'd need to get the ep from PEX-INFO json.
Let me know if you need more info on that, but Pants already interrogates PEX-INFO in some rules or tests or both as I recall.
šŸ‘€ 1
b

bitter-ability-32190

12/01/2022, 3:05 AM
I'm going to drill harder on why I can't just invoke the pex, then if I get too annoyed try that. Thanks
e

enough-analyst-54434

12/01/2022, 3:05 AM
Yeah, battle annoyance - that's certainly the most straightforward and robust path if you can figure it out.
b

bitter-ability-32190

12/01/2022, 3:07 AM
OK last bit before I pause for sleep.
PEX_MODULE=debugpy python pytest_runner.pex --listen 127.0.0.1:5678 --log-to-stderr  pytest_runner.pex
exhibits the odd behavior
PEX_MODULE=debugpy python pytest_runner.pex --listen 127.0.0.1:5678 --log-to-stderr -c 'import pytest;pytest.console_main()
(the entry point) doesn't šŸ¤”
e

enough-analyst-54434

12/01/2022, 3:37 AM
I'm pretty sure the issue is PEX re-exec. If you avoid that, no issues. Materials:
Copy code
$ pex debugpy -m debugpy -o debugpy.pex
$ pex pytest -c pytest -o pytest.loose.pex --layout loose
$ pex pytest -c pytest -o pytest.pex
Troubles due to the PEX doing a re-
exec
into
~/.pex/unzipped_pexes/...
:
Copy code
$ ./debugpy.pex --listen 127.0.0.1:5678 pytest.pex --version
0.06s - Error importing debugpy._vendored.force_pydevd (with sys.path entry: '/home/jsirois/.pex/installed_wheels/2a39e7da178e1f22f4bc04b57f085e785ed1bcf424aaf318835a1a7129eefe35/debugpy-1.6.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl')
Traceback (most recent call last):
  File "/home/jsirois/.pex/installed_wheels/2a39e7da178e1f22f4bc04b57f085e785ed1bcf424aaf318835a1a7129eefe35/debugpy-1.6.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_defaults.py", line 60, in on_pydb_init
    __import__(module_name)
ModuleNotFoundError: No module named 'debugpy'
0.00s - Could not connect to 127.0.0.1: 36263
Traceback (most recent call last):
  File "/home/jsirois/.pex/installed_wheels/2a39e7da178e1f22f4bc04b57f085e785ed1bcf424aaf318835a1a7129eefe35/debugpy-1.6.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py", line 493, in start_client
    s.connect((host, port))
ConnectionRefusedError: [Errno 111] Connection refused
Could not connect to 127.0.0.1: 36263
Traceback (most recent call last):
  File "/home/jsirois/.pex/installed_wheels/2a39e7da178e1f22f4bc04b57f085e785ed1bcf424aaf318835a1a7129eefe35/debugpy-1.6.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/debugpy/_vendored/pydevd/pydevd.py", line 3473, in main
    debugger.connect(host, port)
  File "/home/jsirois/.pex/installed_wheels/2a39e7da178e1f22f4bc04b57f085e785ed1bcf424aaf318835a1a7129eefe35/debugpy-1.6.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/debugpy/_vendored/pydevd/pydevd.py", line 1418, in connect
    s = start_client(host, port)
  File "/home/jsirois/.pex/installed_wheels/2a39e7da178e1f22f4bc04b57f085e785ed1bcf424aaf318835a1a7129eefe35/debugpy-1.6.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py", line 493, in start_client
    s.connect((host, port))
ConnectionRefusedError: [Errno 111] Connection refused
No troubles - no re-exec for loose:
Copy code
$ ./debugpy.pex --listen 127.0.0.1:5678 pytest.loose.pex/ --version
pytest 7.2.0
So you'd need to know the actual location of the PEX. That is done in one of 3 ways: 1. Make a loose PEX. 2. Use the PEX_TOOLS=1 to install the PEX in a venv at a directory you know. 3. When building the PEX use
--seed verbose
Pants does 3 as part of VenvPex creation to get a ~50ms boost over calling into the --venv PEX and suffering a re-
exec
.
Copy code
$ pex pytest -c pytest -o pytest.pex --seed verbose
{"pex_root": "/home/jsirois/.pex", "python": "/home/jsirois/bin/pex.venv/bin/python", "pex": "/home/jsirois/.pex/unzipped_pexes/3ee2eaac643589ce50a865add8ae6e2ea2f52640/__main__.py"}
$ ./debugpy.pex --listen 127.0.0.1:5678 /home/jsirois/.pex/unzipped_pexes/3ee2eaac643589ce50a865add8ae6e2ea2f52640/__main__.py --version
pytest 7.2.0
b

bitter-ability-32190

12/01/2022, 2:58 PM
Ahh 3 looks the most promising I think.
Copy code
$PEX_MODULE=debugpy python pytest_runner.pex --listen 127.0.0.1:5678 .cache/pex_root/venvs/e04f6a4578bcc4f46dc29bec06904079cd804d92/a5d126d40b834824d76a778912f43e6ad41c876b/pex --version
pytest 7.0.1
ā˜ļø thats from within the Pants sandbox for the call (albeit with the entry_point for the runner set back to whatever it was pre-debugpy). I also think it's safe to assume that path will exist (which the shim script can't assume) because as it stands we're executing from that very venv. I'll have to try it out and tweak it and comment as such
(ah to be safe the executable should be
./pytest_runner.pex_pex_shim.sh
)
Ah, this invocation makes me not have to do much Pants mucking:
Copy code
PEX_MODULE=debugpy ./pytest_runner.pex_pex_shim.sh --listen 127.0.0.1:5678 -c "venv=__import__('os').environ['VIRTUAL_ENV'];import runpy;runpy.run_path(venv + '/pex', run_name='__main__')" --version
Thanks, as always, for all your help @enough-analyst-54434 each time I understand PEX a little better šŸ™‚