https://pantsbuild.org/ logo
n

narrow-activity-17405

01/24/2023, 11:28 AM
How can I pass
PYTHONOPTIMIZE
to a pex file? Some of our code depends on
__debug__
- when I run
PYTHONOPTIMIZE=1 python
, then
__debug__
becomes false. But it seems that when I run
PYTHONOPTIMIZE=1 file.pex
,
__debug__
remains true. The same result with
python -O file.pex
🙂 Not sure how this is supposed to work... But I would like to turn on optimizations for production images based on pex files. Thanks for hints.
e

enough-analyst-54434

01/24/2023, 4:28 PM
If you either build your PEX file with
--venv
or else you install your PEX as a venv using pex-tools in your image, this works. You want to be doing the latter anyhow! For more on installing a PEX file in an image instead of just running it - which is not optimal in a number of dimensions - see: + https://pex.readthedocs.io/en/v2.1.121/recipes.html#pex-app-in-a-container + https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/
n

narrow-activity-17405

01/25/2023, 9:15 AM
Thanks for your reply. I used to just copy a pex file into the image. Recently, I followed the link you posted and switched to the "Simple Multi-Stage Build" approach. But still, I can't get optimizations "on". Example of dockerfile here: https://github.com/ZdenekM/arcor2/blob/master/src/docker/arcor2_execution/Dockerfile and the macro I use for pex targets is here: https://github.com/ZdenekM/arcor2/blob/master/pants-plugins/macros.py#L73
e

enough-analyst-54434

01/25/2023, 10:41 AM
Ah, ok. The problem here is the recipes are too formulaic. You need to step back and understand what's going on. Look at your Dockerfile. The optimization is too late! You already compiled your src and deps in two separate RUN statements above! That's when the pyc get created and
__debug__
sealed in.
Lines 3 & 7 need to be
RUN PEX_TOOLS=1 PYTHON OPTIMIZE=1 ...
n

narrow-activity-17405

01/25/2023, 10:44 AM
Ah, ok... I see 🙂 Thanks!
@enough-analyst-54434 I returned to this after some time as I noticed that assertions and statements under
if ___debug___
are still executed. The BUILD files were updated according to what you suggested. Any idea, please? 🙂 I also tried to work with a pex file without docker, but it is (obviously) the same.
Maybe the interpreter can't find
___pycache___
and runs the code from sources?
@enough-analyst-54434 Sorry to bother you... I guess it will be something trivial, but I'm not sure how it actually works under the hood, so it's a bit hard to debug for me. Could you take a look at that? Or should I provide a minimal example?
e

enough-analyst-54434

04/18/2023, 2:50 PM
@narrow-activity-17405 it's been a while, can you please re-state a brief summary of the steps I follow to reproduce your issue? For example, is it using the tip of master for https://github.com/ZdenekM/arcor2 and what exact commands should I run to witness the issue.
6 Views