https://pantsbuild.org/ logo
#general
Title
# general
m

modern-wolf-36228

09/07/2021, 1:58 PM
There's some knowledge that I don't have about pex, I guess, but I don't know what it is. So, I have different pex files, one of them is called
a.pex
and the other is
b.pex
. Those don't have any related project dependency between themselves (to be sure I've extracted and checked the files inside both of them). The thing is that Pydantic checks if you have the same validation functions being used, to force you to not duplicate code. So I have this error:
Copy code
pydantic.errors.ConfigError: duplicate validator function "a.config.Settings.mongo_connection"; if this is intended, set `allow_reuse=True`
I have a
mongo_connection
validator on both
a.pex
and
b.pex
. Which I agree is duplicated, and I should add as a dependency for both of them, but I'm migrating from a polyrepo setup, so I'll probably have many of those. Anyway, my question is: how
a.pex
is able to see
b
?
h

hundreds-father-404

09/07/2021, 2:02 PM
Hi! It looks from https://github.com/nazrulworld/fhir.resources/issues/41 like this can happen based on your import statements, that it's purely a Pydantic quirk and not related to Pants. Might that be possible?
m

modern-wolf-36228

09/07/2021, 2:19 PM
It makes sense. Is there something I can use from
pex
to verify I'm in a pex environment? Similar to this for ipython?
e

enough-analyst-54434

09/07/2021, 2:21 PM
Before answering that, I'd like to step back. a.pex cannot see b.pex unless you run a.pex with
PEX_PATH=b.pex
exported as an environment variable or you built a.pex with
--pex-path=b.pex
.
āœ… 2
Can you clarify how pydantic is involved here? Is it embedded in your PEX and running from there or do you somehow run pydantic over the contents of your PEX files?
m

modern-wolf-36228

09/07/2021, 2:28 PM
I'm going to stop using
a.pex
and
b.pex
for the sake of simplicity on my side. In the image below: ā€¢ a = text_extraction ā€¢ b = classify I'm running
./pants package ::
and the BUILD files of
src/python/text_extraction
and
src/python/classify
are respectively:
Copy code
python_library(dependencies=["./tasks", "./utils"])

python_distribution(
    name="wheel",
    dependencies=[":text_extraction"],
    provides=setup_py(
        name="text_extraction",
        version="0.0.0",
    ),
    setup_py_commands=["bdist_wheel"]
)

pex_binary(
    name="binary",
    dependencies=[":text_extraction"],
    entry_point="<none>",
)
and:
Copy code
python_library()

python_distribution(
    name="wheel",
    dependencies=[":classify"],
    provides=setup_py(
        name="classify",
        version="0.0.0",
    ),
    setup_py_commands=["bdist_wheel"]
)

pex_binary(
    name="binary",
    entry_point="__main__.py",
)
I didn't remove the
python_distribution()
, but I guess they are not important for this case. Finally, I'm running
text_extraction
with:
Copy code
PEX_SCRIPT=celery ./dist/src.python.text_extraction/binary.pex -A <http://text_extraction.worker.app|text_extraction.worker.app> worker -l info -Q text_extraction
I can also open an issue if the size is bothering.
e

enough-analyst-54434

09/07/2021, 2:33 PM
Ok, so it sounds like the only relevant thing is the contents of a.pex - we can work backwards from there. You're claiming, or pydantic is counter-claiming, that
zipinfo a.pex
reveals no duplicate `a/config`paths / files / objects? I'm not sure if it's sensitive material, but can you provide the output of
zipinfo a.pex
?
šŸ‘ 1
m

modern-wolf-36228

09/07/2021, 2:37 PM
The name of the files it's fine, I can't show the content tho, fwiw. I can try to reproduce it with a sample later on, if what I can provide is not enough.
No reference to classify (b.pex).
e

enough-analyst-54434

09/07/2021, 2:39 PM
And what is the real exact pedantic error message?
m

modern-wolf-36228

09/07/2021, 2:39 PM
Copy code
pydantic.errors.ConfigError: duplicate validator function "text_extraction.config.Settings.mongo_connection"; if this is intended, set `allow_reuse=True`
(I have the same
mongo_connection
definition on
classify.config.Settings
).
e

enough-analyst-54434

09/07/2021, 2:42 PM
And how do you get this pydantic error, what command do you run?
m

modern-wolf-36228

09/07/2021, 2:43 PM
Copy code
PEX_SCRIPT=celery ./dist/src.python.text_extraction/binary.pex -A <http://text_extraction.worker.app|text_extraction.worker.app> worker -l info -Q text_extraction
e

enough-analyst-54434

09/07/2021, 2:45 PM
Can you try re-building the pex after adding
execution_mode="venv"
to the relevant
pex_binary
target and then re-running that command line?: https://www.pantsbuild.org/docs/reference-pex_binary#codeexecution_modecode
m

modern-wolf-36228

09/07/2021, 2:45 PM
Sure!
Same error.
(thanks for the patience and help btw šŸ‘)
e

enough-analyst-54434

09/07/2021, 3:18 PM
So, using venv mode the implication is this is no longer a Pants or Pex issue as Eric suggested. To make this easier to debug, you can make an explicit venv instead of the implicit one (the implicit one is created on the fly in ~/.pex/venvs/... if it hasn't already been created). To create an explicit venv:
Copy code
PEX_TOOLS=1 ./dist/src.python.text_extraction/binary.pex venv create/the/venv/right/here
You can then interact with that venv like any other.
šŸ™Œ 1
m

modern-wolf-36228

09/07/2021, 3:50 PM
Thanks! And for curiosity, what is the recommended way to know I'm running inside a pex package? (related to the
__IPYTHON__
question from above)
I saw this: https://github.com/pantsbuild/pex/issues/721 But maybe is too old šŸ¤”
e

enough-analyst-54434

09/07/2021, 4:44 PM
Well, if you're running in venv mode then you aren't running inside PEX at all. Otheriwse, there is no easy way. Say there was a way though, what would you do with that information?
5 Views