https://pantsbuild.org/ logo
Join Slack
Powered by
Is there a way to specifically depend on an extras...
# general
c
Is there a way to specifically depend on an extras requirement? ie. I have a test that's failing because it needs the pydantic email validator (
pydantic[email]
), which I have in the 
requirements.txt
, but dep inference doesn't seem smart enough to pull it in
āœ… 1
šŸ‘€ 1
also I appear to be very close to being ready to go on the 
2.2
-> 
2.11
upgrade, but it did take a near complete rewrite of all of our BUILD files, as well as completely overhauling how we handle 3rd party dependencies, mostly becuase we were doing it completely wrong from the beginning
ā— 1
b
1. Use PEX Json lockfiles (they're great!) (Best in Pants 2.11) 2. Try the last snippet here https://www.pantsbuild.org/docs/python-third-party-dependencies#requirements-with-undeclared-dependencies (use 
overrides
and add the dependencies the extra would've
šŸ‘ 1
h
huh, why a rewrite? šŸ˜®
c
huh, why a rewrite?
python_sources target for each file, dep inference turned off, poorly managed 3rd-party deps
also we are using the pex resolver
h
Ah okay. Did you end up using 
overrides
?
c
working on that now šŸ™‚
ā¤ļø 1
bizarrely, this doesn't work. Neither does explicitly adding the 
email-validator
dependency to the 
python_sources
possibly a red herring
h
what does not work?
c
the overrides for the 
pydantic[email]
requirement
h
what does your BUILD file look like? I'm not totally following the thread
c
<module>/tests/BUILD
Copy code
python_tests(
    interpreter_constraints=["CPython>=3.8"],
    dependencies=[
        ':test_utils',
        '3rd-party/<redacted>:requirements#pydantic',
        '3rd-party/<redacted>:requirements#email-validator',
    ]
)

python_test_utils(
    name="test_utils",
    dependencies=[
        '<redacted>/lib',
        '3rd-party/<redacted>:requirements#pydantic',
        '3rd-party/<redacted>:requirements#email-validator',
    ]
)
3rd-party/<resolve>/BUILD
Copy code
python_requirements(
    name='requirements',
    module_mapping={
        ...
    },
    overrides={
        'PyJWT': {'dependencies': [':requirements#cryptography']},
        'pydantic': {'dependencies': [':requirements#email-validator']},
    },
    resolve='<resolve>'
)
error:
Copy code
11:19:21.00 [ERROR] Completed: Run Pytest - <module>/tests/test_emailer.py failed (exit code 4).
ImportError while loading conftest '/tmp/process-execution9vX2Qn/<module>/tests/conftest.py'.
<module>/tests/conftest.py:12: in <module>
    from <module>.lib.user import User
<module>/lib/user.py:13: in <module>
    class User(BaseModel):
pydantic/main.py:204: in pydantic.main.ModelMetaclass.__new__
    ???
pydantic/fields.py:488: in pydantic.fields.ModelField.infer
    ???
pydantic/fields.py:419: in pydantic.fields.ModelField.__init__
    ???
pydantic/fields.py:539: in pydantic.fields.ModelField.prepare
    ???
pydantic/fields.py:801: in pydantic.fields.ModelField.populate_validators
    ???
pydantic/networks.py:422: in __get_validators__
    ???
pydantic/networks.py:411: in pydantic.networks.import_email_validator
    ???
E   ImportError: email-validator is not installed, run `pip install pydantic[email]`
b
./pants dependencies path/to/conftest.py
?
āž• 1
c
It should, I looked at pydantics 
setup.py
to get it šŸ™‚
Copy code
3rd-party/<resolve>:requirements#email-validator
3rd-party/<resolve>:requirements#pydantic
3rd-party/<resolve>:requirements#requests
<module>/lib/__init__.py
<module>/lib/auth_services/__init__.py
<module>/lib/email/agents/__init__.py
<module>/lib/email/emailers/__init__.py
<module>/lib/email/renderers/__init__.py
<module>/lib/user.py
b
Everything looks right šŸ¤Æ
c
that's why I'm confused šŸ™‚
b
(FWIW you shouldn't need those explicit deps in your 
BUILD
. If you're feeling up to it, you can remove them and verify using 
./pants dependencies
)
c
I added them because the 
override
in 
python_requirements
wasn't working
I also added them to the "business logic" 
python_sources
targets for sanity checks
possibly need to regenerate the lockfile? Doing that now
takes awhile, we have 70 explicit deps and who knows how many transitive deps
nope šŸ˜ž
b
That's unexpected. Try 
./pants --no-process-cleanup test ...
and poke around the sandbox to ensure the dep is listed in the PEX
c
Copy code
/tmp/process-executionfL1ITo
āŸ© cat requirements.pex/PEX-INFO | jq
{
  "bootstrap_hash": "af29c852ae20e6f7e7a8f830fa9efe17f5ab5503",
  "build_properties": {
    "pex_version": "2.1.84"
  },
  "code_hash": "9facd39bdd34791be8291d078a9bb7cc4b790088",
  "distributions": {
    "Jinja2-3.1.2-py3-none-any.whl": "ee8bc6748d864a510e8298d9de49c5e41b75faed18b5649089a1a09295a6c8ef",
    "MarkupSafe-2.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "c257f7495d4439d1d890eebd2b82e169a272dd2b67193
9b904d56886417910a5",
    "certifi-2021.10.8-py2.py3-none-any.whl": "4c386567b236d2f41f64f294487210a0832da94a5e744aef4af035e84614c2c8",
    "charset_normalizer-2.0.12-py3-none-any.whl": "02a5b56ad13945ae0116e3345ce35bc4966356a2f2d0bd99fa89cc227d5ac1e6",
    "dnspython-2.2.1-py3-none-any.whl": "33d2bd8600fafd29eb81071ae12f2bea0fd3e9674f26fa3313146a85ce7081d1",
    "email_validator-1.2.1-py2.py3-none-any.whl": "33b030891faf46cd3eabd426ac872e96f73a9721764bfbf470f4a607dbf95285",
    "idna-3.3-py3-none-any.whl": "f6618af934a714454313a29b891c03fe8cde2be7bf19e20e3abc11d26ce689ac",
    "pydantic-1.9.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "426840d1c16f525423dabddb077b10d569a7f367b8e240c
it's there šŸ˜•
ā— 1
b
I'm stumped. Is something messing with the 
sys.path
or something else nefarious?!
c
not in this module (we have some of that elsewhere to support Jupyter notebooks)
it wouldn't have any effect here, I don't think it's even tracked in pants
b
I'd splash around the sandbox and just try and get 
import email_validator
working in 
conftest.py
or the test itself
šŸ‘ 1
h
fyi 
PEX_EXTRA_SYS_PATH
== Pex's version of 
PYTHONPATH
/ 
sys.path
. You can mess around w/ that in 
__run.sh
šŸ‘ 1
c
I... don't get this, but doing a 
rm -rf ~/.cache/pants
fixed it.......
ā— 1
šŸ˜± 1
b
Yikes, but also yay?
h
big yikes
c
if I had to guess, it's related to the lockfiles not invalidating something
h
fwit, in the future, we always appreciate when folks due an 
mv
rather than 
rm -rf
so that we can inspect. please let us know if you encounter something like this again glad you're unblocked at least
šŸ‘ 1
6 Views
Open in Slack
PreviousNext
Powered by