https://pantsbuild.org/ logo
Join SlackCommunities
Powered by
There's some knowledge that I don't have about pex...
# general
m
There's some knowledge that I don't have about pex, I guess, but I don't know what it is. So, I have different pex files, one of them is called 
a.pex
and the other is 
b.pex
. Those don't have any related project dependency between themselves (to be sure I've extracted and checked the files inside both of them). The thing is that Pydantic checks if you have the same validation functions being used, to force you to not duplicate code. So I have this error:
Copy code
pydantic.errors.ConfigError: duplicate validator function "a.config.Settings.mongo_connection"; if this is intended, set `allow_reuse=True`
I have a 
mongo_connection
validator on both 
a.pex
and 
b.pex
. Which I agree is duplicated, and I should add as a dependency for both of them, but I'm migrating from a polyrepo setup, so I'll probably have many of those. Anyway, my question is: how 
a.pex
is able to see 
b
?
h
Hi! It looks from https://github.com/nazrulworld/fhir.resources/issues/41 like this can happen based on your import statements, that it's purely a Pydantic quirk and not related to Pants. Might that be possible?
m
It makes sense. Is there something I can use from 
pex
to verify I'm in a pex environment? Similar to this for ipython?
e
Before answering that, I'd like to step back. a.pex cannot see b.pex unless you run a.pex with 
PEX_PATH=b.pex
exported as an environment variable or you built a.pex with 
--pex-path=b.pex
.
āœ… 2
Can you clarify how pydantic is involved here? Is it embedded in your PEX and running from there or do you somehow run pydantic over the contents of your PEX files?
m
I'm going to stop using 
a.pex
and 
b.pex
for the sake of simplicity on my side. In the image below: ā€¢ a = text_extraction ā€¢ b = classify I'm running 
./pants package ::
and the BUILD files of 
src/python/text_extraction
and 
src/python/classify
are respectively:
Copy code
python_library(dependencies=["./tasks", "./utils"])

python_distribution(
    name="wheel",
    dependencies=[":text_extraction"],
    provides=setup_py(
        name="text_extraction",
        version="0.0.0",
    ),
    setup_py_commands=["bdist_wheel"]
)

pex_binary(
    name="binary",
    dependencies=[":text_extraction"],
    entry_point="<none>",
)
and:
Copy code
python_library()

python_distribution(
    name="wheel",
    dependencies=[":classify"],
    provides=setup_py(
        name="classify",
        version="0.0.0",
    ),
    setup_py_commands=["bdist_wheel"]
)

pex_binary(
    name="binary",
    entry_point="__main__.py",
)
I didn't remove the 
python_distribution()
, but I guess they are not important for this case. Finally, I'm running 
text_extraction
with:
Copy code
PEX_SCRIPT=celery ./dist/src.python.text_extraction/binary.pex -A <http://text_extraction.worker.app|text_extraction.worker.app> worker -l info -Q text_extraction
I can also open an issue if the size is bothering.
e
Ok, so it sounds like the only relevant thing is the contents of a.pex - we can work backwards from there. You're claiming, or pydantic is counter-claiming, that 
zipinfo a.pex
reveals no duplicate `a/config`paths / files / objects? I'm not sure if it's sensitive material, but can you provide the output of 
zipinfo a.pex
?
šŸ‘ 1
m
The name of the files it's fine, I can't show the content tho, fwiw. I can try to reproduce it with a sample later on, if what I can provide is not enough.
out.txt
No reference to classify (b.pex).
e
And what is the real exact pedantic error message?
m
Copy code
pydantic.errors.ConfigError: duplicate validator function "text_extraction.config.Settings.mongo_connection"; if this is intended, set `allow_reuse=True`
(I have the same 
mongo_connection
definition on 
classify.config.Settings
).
e
And how do you get this pydantic error, what command do you run?
m
Copy code
PEX_SCRIPT=celery ./dist/src.python.text_extraction/binary.pex -A <http://text_extraction.worker.app|text_extraction.worker.app> worker -l info -Q text_extraction
e
Can you try re-building the pex after adding 
execution_mode="venv"
to the relevant 
pex_binary
target and then re-running that command line?: https://www.pantsbuild.org/docs/reference-pex_binary#codeexecution_modecode
m
Sure!
Same error.
(thanks for the patience and help btw šŸ‘)
e
So, using venv mode the implication is this is no longer a Pants or Pex issue as Eric suggested. To make this easier to debug, you can make an explicit venv instead of the implicit one (the implicit one is created on the fly in ~/.pex/venvs/... if it hasn't already been created). To create an explicit venv:
Copy code
PEX_TOOLS=1 ./dist/src.python.text_extraction/binary.pex venv create/the/venv/right/here
You can then interact with that venv like any other.
šŸ™Œ 1
m
Thanks! And for curiosity, what is the recommended way to know I'm running inside a pex package? (related to the 
__IPYTHON__
question from above)
I saw this: https://github.com/pantsbuild/pex/issues/721 But maybe is too old šŸ¤”
e
Well, if you're running in venv mode then you aren't running inside PEX at all. Otheriwse, there is no easy way. Say there was a way though, what would you do with that information?
5 Views
Open in Slack
PreviousNext
Powered by