https://pantsbuild.org/ logo
#general
Title
# general
c

calm-ambulance-65371

05/05/2022, 3:45 PM
Is there a way to specifically depend on an extras requirement? ie. I have a test that's failing because it needs the pydantic email validator (
pydantic[email]
), which I have in the
requirements.txt
, but dep inference doesn't seem smart enough to pull it in
āœ… 1
šŸ‘€ 1
also I appear to be very close to being ready to go on the
2.2
->
2.11
upgrade, but it did take a near complete rewrite of all of our BUILD files, as well as completely overhauling how we handle 3rd party dependencies, mostly becuase we were doing it completely wrong from the beginning
ā— 1
b

bitter-ability-32190

05/05/2022, 3:49 PM
1. Use PEX Json lockfiles (they're great!) (Best in Pants 2.11) 2. Try the last snippet here https://www.pantsbuild.org/docs/python-third-party-dependencies#requirements-with-undeclared-dependencies (use
overrides
and add the dependencies the extra would've
šŸ‘ 1
h

hundreds-father-404

05/05/2022, 3:50 PM
huh, why a rewrite? šŸ˜®
c

calm-ambulance-65371

05/05/2022, 3:55 PM
huh, why a rewrite?
python_sources target for each file, dep inference turned off, poorly managed 3rd-party deps
also we are using the pex resolver
h

hundreds-father-404

05/05/2022, 3:56 PM
Ah okay. Did you end up using
overrides
?
c

calm-ambulance-65371

05/05/2022, 3:56 PM
working on that now šŸ™‚
ā¤ļø 1
bizarrely, this doesn't work. Neither does explicitly adding the
email-validator
dependency to the
python_sources
possibly a red herring
h

hundreds-father-404

05/05/2022, 4:19 PM
what does not work?
c

calm-ambulance-65371

05/05/2022, 4:19 PM
the overrides for the
pydantic[email]
requirement
h

hundreds-father-404

05/05/2022, 4:20 PM
what does your BUILD file look like? I'm not totally following the thread
c

calm-ambulance-65371

05/05/2022, 4:22 PM
<module>/tests/BUILD
Copy code
python_tests(
    interpreter_constraints=["CPython>=3.8"],
    dependencies=[
        ':test_utils',
        '3rd-party/<redacted>:requirements#pydantic',
        '3rd-party/<redacted>:requirements#email-validator',
    ]
)

python_test_utils(
    name="test_utils",
    dependencies=[
        '<redacted>/lib',
        '3rd-party/<redacted>:requirements#pydantic',
        '3rd-party/<redacted>:requirements#email-validator',
    ]
)
3rd-party/<resolve>/BUILD
Copy code
python_requirements(
    name='requirements',
    module_mapping={
        ...
    },
    overrides={
        'PyJWT': {'dependencies': [':requirements#cryptography']},
        'pydantic': {'dependencies': [':requirements#email-validator']},
    },
    resolve='<resolve>'
)
error:
Copy code
11:19:21.00 [ERROR] Completed: Run Pytest - <module>/tests/test_emailer.py failed (exit code 4).
ImportError while loading conftest '/tmp/process-execution9vX2Qn/<module>/tests/conftest.py'.
<module>/tests/conftest.py:12: in <module>
    from <module>.lib.user import User
<module>/lib/user.py:13: in <module>
    class User(BaseModel):
pydantic/main.py:204: in pydantic.main.ModelMetaclass.__new__
    ???
pydantic/fields.py:488: in pydantic.fields.ModelField.infer
    ???
pydantic/fields.py:419: in pydantic.fields.ModelField.__init__
    ???
pydantic/fields.py:539: in pydantic.fields.ModelField.prepare
    ???
pydantic/fields.py:801: in pydantic.fields.ModelField.populate_validators
    ???
pydantic/networks.py:422: in __get_validators__
    ???
pydantic/networks.py:411: in pydantic.networks.import_email_validator
    ???
E   ImportError: email-validator is not installed, run `pip install pydantic[email]`
b

bitter-ability-32190

05/05/2022, 4:26 PM
./pants dependencies path/to/conftest.py
?
āž• 1
c

calm-ambulance-65371

05/05/2022, 4:26 PM
It should, I looked at pydantics
setup.py
to get it šŸ™‚
Copy code
3rd-party/<resolve>:requirements#email-validator
3rd-party/<resolve>:requirements#pydantic
3rd-party/<resolve>:requirements#requests
<module>/lib/__init__.py
<module>/lib/auth_services/__init__.py
<module>/lib/email/agents/__init__.py
<module>/lib/email/emailers/__init__.py
<module>/lib/email/renderers/__init__.py
<module>/lib/user.py
b

bitter-ability-32190

05/05/2022, 4:29 PM
Everything looks right šŸ¤Æ
c

calm-ambulance-65371

05/05/2022, 4:29 PM
that's why I'm confused šŸ™‚
b

bitter-ability-32190

05/05/2022, 4:29 PM
(FWIW you shouldn't need those explicit deps in your
BUILD
. If you're feeling up to it, you can remove them and verify using
./pants dependencies
)
c

calm-ambulance-65371

05/05/2022, 4:29 PM
I added them because the
override
in
python_requirements
wasn't working
I also added them to the "business logic"
python_sources
targets for sanity checks
possibly need to regenerate the lockfile? Doing that now
takes awhile, we have 70 explicit deps and who knows how many transitive deps
nope šŸ˜ž
b

bitter-ability-32190

05/05/2022, 4:41 PM
That's unexpected. Try
./pants --no-process-cleanup test ...
and poke around the sandbox to ensure the dep is listed in the PEX
c

calm-ambulance-65371

05/05/2022, 4:44 PM
Copy code
/tmp/process-executionfL1ITo
āŸ© cat requirements.pex/PEX-INFO | jq
{
  "bootstrap_hash": "af29c852ae20e6f7e7a8f830fa9efe17f5ab5503",
  "build_properties": {
    "pex_version": "2.1.84"
  },
  "code_hash": "9facd39bdd34791be8291d078a9bb7cc4b790088",
  "distributions": {
    "Jinja2-3.1.2-py3-none-any.whl": "ee8bc6748d864a510e8298d9de49c5e41b75faed18b5649089a1a09295a6c8ef",
    "MarkupSafe-2.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "c257f7495d4439d1d890eebd2b82e169a272dd2b67193
9b904d56886417910a5",
    "certifi-2021.10.8-py2.py3-none-any.whl": "4c386567b236d2f41f64f294487210a0832da94a5e744aef4af035e84614c2c8",
    "charset_normalizer-2.0.12-py3-none-any.whl": "02a5b56ad13945ae0116e3345ce35bc4966356a2f2d0bd99fa89cc227d5ac1e6",
    "dnspython-2.2.1-py3-none-any.whl": "33d2bd8600fafd29eb81071ae12f2bea0fd3e9674f26fa3313146a85ce7081d1",
    "email_validator-1.2.1-py2.py3-none-any.whl": "33b030891faf46cd3eabd426ac872e96f73a9721764bfbf470f4a607dbf95285",
    "idna-3.3-py3-none-any.whl": "f6618af934a714454313a29b891c03fe8cde2be7bf19e20e3abc11d26ce689ac",
    "pydantic-1.9.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "426840d1c16f525423dabddb077b10d569a7f367b8e240c
it's there šŸ˜•
ā— 1
b

bitter-ability-32190

05/05/2022, 4:45 PM
I'm stumped. Is something messing with the
sys.path
or something else nefarious?!
c

calm-ambulance-65371

05/05/2022, 4:46 PM
not in this module (we have some of that elsewhere to support Jupyter notebooks)
it wouldn't have any effect here, I don't think it's even tracked in pants
b

bitter-ability-32190

05/05/2022, 4:47 PM
I'd splash around the sandbox and just try and get
import email_validator
working in
conftest.py
or the test itself
šŸ‘ 1
h

hundreds-father-404

05/05/2022, 4:47 PM
fyi
PEX_EXTRA_SYS_PATH
== Pex's version of
PYTHONPATH
/
sys.path
. You can mess around w/ that in
__run.sh
šŸ‘ 1
c

calm-ambulance-65371

05/05/2022, 4:54 PM
I... don't get this, but doing a
rm -rf ~/.cache/pants
fixed it.......
ā— 1
šŸ˜± 1
b

bitter-ability-32190

05/05/2022, 4:55 PM
Yikes, but also yay?
h

hundreds-father-404

05/05/2022, 4:55 PM
big yikes
c

calm-ambulance-65371

05/05/2022, 4:55 PM
if I had to guess, it's related to the lockfiles not invalidating something
h

hundreds-father-404

05/05/2022, 4:56 PM
fwit, in the future, we always appreciate when folks due an
mv
rather than
rm -rf
so that we can inspect. please let us know if you encounter something like this again glad you're unblocked at least
šŸ‘ 1
6 Views