There s some knowledge that I don t have about pex I guess b Pants #general

There's some knowledge that I don't have about pex...

modern-wolf-36228

09/07/2021, 1:58 PM

There's some knowledge that I don't have about pex, I guess, but I don't know what it is. So, I have different pex files, one of them is called

a.pex

and the other is

b.pex

. Those don't have any related project dependency between themselves (to be sure I've extracted and checked the files inside both of them). The thing is that Pydantic checks if you have the same validation functions being used, to force you to not duplicate code. So I have this error:

Copy code

pydantic.errors.ConfigError: duplicate validator function "a.config.Settings.mongo_connection"; if this is intended, set `allow_reuse=True`

I have a

mongo_connection

validator on both

a.pex

and

b.pex

. Which I agree is duplicated, and I should add as a dependency for both of them, but I'm migrating from a polyrepo setup, so I'll probably have many of those. Anyway, my question is: how

a.pex

is able to see

hundreds-father-404

09/07/2021, 2:02 PM

Hi! It looks from https://github.com/nazrulworld/fhir.resources/issues/41 like this can happen based on your import statements, that it's purely a Pydantic quirk and not related to Pants. Might that be possible?

modern-wolf-36228

09/07/2021, 2:19 PM

It makes sense. Is there something I can use from

pex

to verify I'm in a pex environment? Similar to this for ipython?

enough-analyst-54434

09/07/2021, 2:21 PM

Before answering that, I'd like to step back. a.pex cannot see b.pex unless you run a.pex with

PEX_PATH=b.pex

exported as an environment variable or you built a.pex with

--pex-path=b.pex

✅ 2

enough-analyst-54434

09/07/2021, 2:21 PM

Can you clarify how pydantic is involved here? Is it embedded in your PEX and running from there or do you somehow run pydantic over the contents of your PEX files?

modern-wolf-36228

09/07/2021, 2:28 PM

I'm going to stop using

a.pex

and

b.pex

for the sake of simplicity on my side. In the image below: • a = text_extraction • b = classify I'm running

./pants package ::

and the BUILD files of

src/python/text_extraction

and

src/python/classify

are respectively:

Copy code

python_library(dependencies=["./tasks", "./utils"])

python_distribution(
    name="wheel",
    dependencies=[":text_extraction"],
    provides=setup_py(
        name="text_extraction",
        version="0.0.0",
    ),
    setup_py_commands=["bdist_wheel"]
)

pex_binary(
    name="binary",
    dependencies=[":text_extraction"],
    entry_point="<none>",
)

and:

Copy code

python_library()

python_distribution(
    name="wheel",
    dependencies=[":classify"],
    provides=setup_py(
        name="classify",
        version="0.0.0",
    ),
    setup_py_commands=["bdist_wheel"]
)

pex_binary(
    name="binary",
    entry_point="__main__.py",
)

I didn't remove the

python_distribution()

, but I guess they are not important for this case. Finally, I'm running

text_extraction

with:

Copy code

PEX_SCRIPT=celery ./dist/src.python.text_extraction/binary.pex -A <http://text_extraction.worker.app|text_extraction.worker.app> worker -l info -Q text_extraction

modern-wolf-36228

09/07/2021, 2:29 PM

I can also open an issue if the size is bothering.

enough-analyst-54434

09/07/2021, 2:33 PM

Ok, so it sounds like the only relevant thing is the contents of a.pex - we can work backwards from there. You're claiming, or pydantic is counter-claiming, that

zipinfo a.pex

reveals no duplicate `a/config`paths / files / objects? I'm not sure if it's sensitive material, but can you provide the output of

zipinfo a.pex

👍 1

modern-wolf-36228

09/07/2021, 2:37 PM

The name of the files it's fine, I can't show the content tho, fwiw. I can try to reproduce it with a sample later on, if what I can provide is not enough.

out.txt

modern-wolf-36228

09/07/2021, 2:38 PM

No reference to classify (b.pex).

enough-analyst-54434

09/07/2021, 2:39 PM

And what is the real exact pedantic error message?

modern-wolf-36228

09/07/2021, 2:39 PM

Copy code

pydantic.errors.ConfigError: duplicate validator function "text_extraction.config.Settings.mongo_connection"; if this is intended, set `allow_reuse=True`

modern-wolf-36228

09/07/2021, 2:40 PM

(I have the same

mongo_connection

definition on

classify.config.Settings

enough-analyst-54434

09/07/2021, 2:42 PM

And how do you get this pydantic error, what command do you run?

modern-wolf-36228

09/07/2021, 2:43 PM

Copy code

PEX_SCRIPT=celery ./dist/src.python.text_extraction/binary.pex -A <http://text_extraction.worker.app|text_extraction.worker.app> worker -l info -Q text_extraction

enough-analyst-54434

09/07/2021, 2:45 PM

Can you try re-building the pex after adding

execution_mode="venv"

to the relevant

pex_binary

target and then re-running that command line?: https://www.pantsbuild.org/docs/reference-pex_binary#codeexecution_modecode

modern-wolf-36228

09/07/2021, 2:45 PM

Sure!

modern-wolf-36228

09/07/2021, 2:50 PM

Same error.

modern-wolf-36228

09/07/2021, 2:50 PM

(thanks for the patience and help btw 👍)

enough-analyst-54434

09/07/2021, 3:18 PM

So, using venv mode the implication is this is no longer a Pants or Pex issue as Eric suggested. To make this easier to debug, you can make an explicit venv instead of the implicit one (the implicit one is created on the fly in ~/.pex/venvs/... if it hasn't already been created). To create an explicit venv:

Copy code

PEX_TOOLS=1 ./dist/src.python.text_extraction/binary.pex venv create/the/venv/right/here

You can then interact with that venv like any other.

🙌 1

modern-wolf-36228

09/07/2021, 3:50 PM

Thanks! And for curiosity, what is the recommended way to know I'm running inside a pex package? (related to the

__IPYTHON__

question from above)

modern-wolf-36228

09/07/2021, 3:51 PM

I saw this: https://github.com/pantsbuild/pex/issues/721 But maybe is too old 🤔

enough-analyst-54434

09/07/2021, 4:44 PM

Well, if you're running in venv mode then you aren't running inside PEX at all. Otheriwse, there is no easy way. Say there was a way though, what would you do with that information?

5 Views

Open in Slack

Previous Next