OK, trying to pexify some legacy docker containers (with ensuing massive, massive reductions in container size and source code, which is all good...) ...but... having trouble with some namespace packages in the pex just... not existing. The pexes are built with

execution_mode="venv"

, but basically what's happening is that in my virtual environment I can import the namespace package and do a

dir

and see lots of stuff; the pex shows the identical dependencies in there, and I can still import the namespace package, but the result is an empty module. I can post more specifics, but is this a known thing with pexes that can be worked around?

This is relevant from

pex --help

:

...
--venv-site-packages-copies, --no-venv-site-packages-copies
                        If --venv is specified, populate the venv site packages using hard links or copies of resolved PEX dependencies instead of symlinks. This can be used to work around problems with tools or libraries that are confused by symlinked source files.
                        (default: False)
...

But I thought Pants was passing

--venv-site-packages-copies

by default due to problems like this. Looking....

2.10.0.dev3 (using several of the new docker features so I'm at mildly-bleeding-edge at the moment)

No, this hasn't changed since introduced, just wanted to see if you were on a version that knows about the feature as a baseline check.

I don't have any pex in pants.toml.

Ok. Is your error including enough info to know where the venv is on the file system?

I can poke around, stand by

Thanks. If so, please cd to the venv site-packages directory and sanity check the lay of the land. Does it look to you like said ns packages are there?

well, I see some in the pex_root/venvs dir but I'm not sure how to figure out which one is relevant. I don't get any error message; the contents of the namespace package just aren't there. I can certainly see some relevant packages in the dist when built, but it would be easier with a venv in `pex_root`; let me kill all of them and briefly recreate.

So, you're in

repl

? Or an

export

venv?

Yes, or the one in the docker container

OK - how do you build the venv in the docker container? Do you build one or let PEX do it automatically on 1st run?

I have a

pex_binary

target which I can test independently (and which has this error) and the docker container just relies on it as its entrypoint.

OK, so you don't do that. (you should after we figure this out!)

I wasn't doing that; I was doing the "auto-pants thing":

COPY workers.remote.remote_worker /bin/worker

...and letting Pants do its magic to build the pex and install it. But the pex still does that outside the container in any case.

Um, sorry for the scattershot. Is the ns package in 1st party code, 3rdparty code or bridging both worlds?

BTW going into the cache venv and sourcing

bin/activate

gives me the same missing libs, but ... whoa. OK, first, it's 3rdparty (IBM's qiskit). Second, yeah, there's some weirdness. In the venv, I just see some dist-info stuff:

venvs/90ead2017bd04f8310cd1e14573bd10fcec8e7ac/41e856e498bc4b3ee69c2c959cfa129f9a925ee5 via 🐍 v3.9.9 (omni)
❯ find . -name "*qiskit*"
./lib/python3.9/site-packages/pex-ns-pkgs/1/qiskit_aer-0.9.0.dist-info
./lib/python3.9/site-packages/pex-ns-pkgs/1/qiskit_terra-0.18.3.dist-info
./lib/python3.9/site-packages/pex-ns-pkgs/1/qiskit_aer.libs
./lib/python3.9/site-packages/pex-ns-pkgs/1/qiskit
./lib/python3.9/site-packages/pex-ns-pkgs/2/qiskit_ibmq_provider-0.16.0.dist-info
./lib/python3.9/site-packages/pex-ns-pkgs/2/qiskit

...but in the built dist package, I see a lot of the source files for some of those packages, under

.deps

, ie

0  Stored        0   0% 1980-01-01 00:00 00000000  .deps/qiskit_terra-0.18.3-cp39-cp39-manylinux2010_x86_64.whl/qiskit/circuit/library/boolean_logic/
     645  Defl:N      373  42% 1980-01-01 00:00 4d758fbb  .deps/qiskit_terra-0.18.3-cp39-cp39-manylinux2010_x86_64.whl/qiskit/circuit/library/boolean_logic/__init__.py
    2644  Defl:N     1070  60% 1980-01-01 00:00 63275574  .deps/qiskit_terra-0.18.3-cp39-cp39-manylinux2010_x86_64.whl/qiskit/circuit/library/boolean_logic/inner_product.py
    3858  Defl:N     1467  62% 1980-01-01 00:00 9257cc40  .deps/qiskit_terra-0.18.3-cp39-cp39-manylinux2010_x86_64.whl/qiskit/circuit/library/boolean_logic/quantum_and.py

...which I don't see unpacked in the venv.

I think you need

ls -la

to start to get un-baffled.

fair, fair. The three packages are indeed listed in the pex info:

... "qiskit_aer-0.9.0-cp39-cp39-m
       │ anylinux_2_12_x86_64.manylinux2010_x86_64.whl": "d346d7a5f1f40d0e537d5cdeed409591d957e0a3", "qiskit_ibmq_
       │ provider-0.16.0-py3-none-any.whl": "d7e2598eedab1acd9ec10faa9b9d9989f70ca2a0", "qiskit_terra-0.18.3-cp39-
       │ cp39-manylinux2010_x86_64.whl": "ddf8328426dcec30bd78ffcb4640f42e7f82ec00",...

Can you try this?:

PEX_TOOLS=1 ./my.pex venv right/here
source right/here/bin/activate

trying it now

Add --collisions-ok

I see how the venv build works. Still got an error but that's because I was including

--compile

.

So, yeah, follow https://pex.readthedocs.io/en/v2.1.67/recipes.html#pex-app-in-a-container +

--collisions-ok

--compile

created great rage.

How so?

well, the venving seems to have deleted my pex, so I'll, er, try again. But at least this gives me something to iterate on.

You could try

PEX_TOOLS=1 /the/python/I/meant my.pex venv ...

if your PEX does not have interpreter constraints (see the pex info you generated to see if it does).

Ah, I thought the

--rm all

was to clean the dest, not delete the pex, but that's great. So after the warning for the collision I get a pause and then

...
    raise cls.NonZeroExit(cmd, process.returncode, stdout, stderr)
pex.executor.Executor.NonZeroExit: received exit code 1 during execution of `['/home/vputz/.pyenv/versions/3.9.9/bin/python3.9', '-s', '-E', '-m', 'compileall', '/tmp/remote_worker']` while trying to execute `['/home/vputz/.pyenv/versions/3.9.9/bin/python3.9', '-s', '-E', '-m', 'compileall', '/tmp/remote_worker']`

And the .... has no useful information?

just the full traceback; had omitted but (the first bit with the dwave packages is the collision, then the crash with the compile):

/home/vputz/.pex/unzipped_pexes/b23715185de82b6720330283368463c29b19d760/.bootstrap/pex/tools/commands/venv.py:230: PEXWarning: Encountered collision building venv at /tmp/remote_worker from /home/vputz/projects/forge/dist/workers.remote.remote_worker/remote_worker.pex:
1. /tmp/remote_worker/lib/python3.9/site-packages/dwave/__init__.py was provided by:
	sha1:03ed2c399e8fecd324f4e9719168a3ea3fa02900 -> /home/vputz/.pex/installed_wheels/8eefaf7bf166f47206f8e3258b0fb79b819698f8/dwave_preprocessing-0.3.1.post0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl/dwave/__init__.py
	sha1:52c2da64768c46c18626ac887b146a01ae869677 -> /home/vputz/.pex/installed_wheels/07e459934624620d022a978475227daa5f972918/dwave_system-1.9.0-py3-none-any.whl/dwave/__init__.py
	sha1:764c752b19626766a09406a6d221938432bdf4fb -> /home/vputz/.pex/installed_wheels/cf15b1693c5c3d3dfe1d88b071b5d430f8ea2092/dwave_cloud_client-0.9.2-py3-none-any.whl/dwave/__init__.py
  pex_warnings.warn(message)
Traceback (most recent call last):
  File "/home/vputz/.pex/unzipped_pexes/b23715185de82b6720330283368463c29b19d760/.bootstrap/pex/pex.py", line 473, in execute
    exit_value = tools.main(pex=PEX(sys.argv[0]))
  File "/home/vputz/.pex/unzipped_pexes/b23715185de82b6720330283368463c29b19d760/.bootstrap/pex/tools/main.py", line 90, in main
    result = catch(pex_command.run, pex)
  File "/home/vputz/.pex/unzipped_pexes/b23715185de82b6720330283368463c29b19d760/.bootstrap/pex/commands/command.py", line 130, in catch
    return func(*args, **kwargs)
  File "/home/vputz/.pex/unzipped_pexes/b23715185de82b6720330283368463c29b19d760/.bootstrap/pex/tools/commands/venv.py", line 569, in run
    pex.interpreter.execute(["-m", "compileall", venv_dir])
  File "/home/vputz/.pex/unzipped_pexes/b23715185de82b6720330283368463c29b19d760/.bootstrap/pex/interpreter.py", line 1190, in execute
    stdout, stderr = Executor.execute(cmd, stdin_payload=stdin_payload, env=env, **kwargs)
  File "/home/vputz/.pex/unzipped_pexes/b23715185de82b6720330283368463c29b19d760/.bootstrap/pex/executor.py", line 99, in execute
    raise cls.NonZeroExit(cmd, process.returncode, stdout, stderr)
pex.executor.Executor.NonZeroExit: received exit code 1 during execution of `['/home/vputz/.pyenv/versions/3.9.9/bin/python3.9', '-s', '-E', '-m', 'compileall', '/tmp/remote_worker']` while trying to execute `['/home/vputz/.pyenv/versions/3.9.9/bin/python3.9', '-s', '-E', '-m', 'compileall', '/tmp/remote_worker']`

What happens if you run

/home/vputz/.pyenv/versions/3.9.9/bin/python3.9 -s -E -m compileall /tmp/remote_worker

manually?

Hmm, runs to completion although lists a syntax error for one library

And exit code?

Exit code seems to be 1. Let me fix that syntax error for fun.

Pex often reveals the how the sausage is made, and it can be scary.

Yeah, that's the issue, sigh.

Yes.

and that file is never used so never jit compiled thus no error. Good to know; we can work with this.

OK - so I think then your use case is fully solved. Is that correct @echoing-farmer-15630? I think Pants will still need to plumb an option for

--venv-site-packages-copies

since not everyone will have this be a valid workaround for them.

OK, think I learned a lot here I can work with. Thanks a TON; this is going to improve things quite a bit.

Exellent.

Yep, I think this will work; I'll chew on it a bit and come crying when I find something that doesn't, but I think this will do the trick.

Here's the bug tracking plumbing ``--venv-site-packages-copies`` to `pex_binary`: https://github.com/pantsbuild/pants/issues/14559

Thanks! Incidentally, not knowing if you have any feedback to the pex group (or are on it) but the instructions at https://pex.readthedocs.io/en/v2.1.67/recipes.html#pex-app-in-a-container leave the pex in the layers so the docker image actually is doubly big even with "--rm". You can fix that with a multistage build, ie

FROM baselayer as pex
COPY my_pex /pex
# this expands and builds the venv
RUN PEX_TOOLS=1 /pex/my_pex.pex venv --rm all --compile --collisions-ok /venv
# we now start a new layer and copy the venv from the old
FROM baselayer as venv
COPY --from=pex /venv /venv
...

... this really makes a difference in some images; one of mine went from 3.4Gb to 2.43Gb (it includes some hideously big libraries like Torch); another went from 1.33Gb to 1.09Gb, so it's not an insignificant savings! You can use a tool like dive (https://github.com/wagoodman/dive) to inspect the images and see that the original method leaves the pex "space" in there even though it's deleted.

Aha, yeah. Then there is no point in the

--rm all

IIUC. I am the de-facto Pex group but its also open source! So do you want to take a crack at this?: https://github.com/pantsbuild/pex/blob/v2.1.67/docs/recipes.rst

Sure, sorry--socked in today (also a new parent so time is ridiculously tight) but I can have a go in a day or so!

I think I obsoleted your crack at it @echoing-farmer-15630. This came up: https://github.com/pantsbuild/pex/pull/1634 Enjoy time with your young one instead!

Honestly, that's awesome (the split between deps and srcs). It really will help with stability splitting that up over multiple layers. Nice work!