https://pantsbuild.org/ logo
Join Slack
Powered by
heya :wave: pants seems to struggle with inferring...
# general
b
heya šŸ‘‹ pants seems to struggle with inferring ownership of (python) imports of third-party modules, and I cannot pinpoint what's causing trouble. For instance, pants is complaining about this import: 
from dagster import Definitions
. The dependency package is also dagster, so not a case of non-standard package name. A couple of additional findings: ā€¢ there are several 
python_requirements
targets with the same package included, I'm explicitly setting one of them as a dependency ā€¢ explicitly adding an entry to the module_mapping of the corresponding 
python_requirement
helps: 
module_mapping={"dagster": ["dagster.Definitions"]}
(
"dagster.*"
wouldn't work) Anyone here that could point to the right direction? TIA
e
"several 
python_requirements
targets with the same package included" What is your reason for this? are you trying to work with multiple resolves?
b
no, we have distinct components in our monorepo, trying to group dependencies per component. One component can be a python package, another a script with its own venv, an image etc
e
I think that trying to group dependencies per component might be what's causing you trouble. As far as pants is concerned, having different 
python_requirements
targets is just a way of declaring them. (
python_requirements
is just a macro to create a lot of individual 
python_requirement
targets) Assuming you are only using a single resolve/lockfile, each 
python_requirements
target can be read as "Hey pants, please put this list of 3rd party dependencies into the lockfile".
I think I see two opportunities here: ā€¢ If you want completely separated and distinct dependency management for each component, you'll want to look into setting up multiple lockfiles, so they can be truly independent. (In general, my opinion is that the only reason to do this is if your various components need to be able to use conflicting versions of packages)
ā€¢ (my case at $DAY_JOB) - Use one "universal resolve": ā—¦ Set up a global 
python_requirements
at `3rdparty/python/requirements.txt ā—¦ put all dependencies used by anything here (you can still use separate 
python_requirements
, but it will be more confusing since the requirements are not actually separate ā—¦ Any python source file can import and use these requirements, and pants dependency inference will ensure that the systems use the minimal subset of dependencies required. (eg. if you build a pex file, it will only contain the 3rd party packages that are (transiticely) imported by one of the source files)
I like to think of this as if I declare a pool of 3rd party dependencies that my python code is allowed to use, but then pants takes care of the actual mapping and sandboxing
(The multiple resolves situation works the same way, but you have to be explicit about which resolve a requirement should go in, and which resolve any particular source file should run with, including using 
parametrize
to describe multiple resolves for a dependency to be installed into, and for a source to run in, and to lint, typecheck, and test against)
b
we did use multiple resolves in the past for the reason you suggest; got back to a single-resolve setup when conflicts were eventually removed; Our experience with multiple resolves aligns with your suggestion (use only if necessary ) šŸ˜„ So separate python requirements are in part overdue technical debt; however there's a good chance that conflicts will occur again, and this setup is easier for keeping tabs on each component. Another issue that came up was with optional dependencies: we distribute a PEX binary without them, and expect the end user to make them available if needed. We do define them for tests though, so they end up in our global resolve but not in the PEX. This could probably be done by explicitly black-listing optional dependencies on the pex target, but there's a higher risk something will slip. anyway, I'll definitely give a global 
python_requirements
another go at some point soon! appreciate all the input šŸ™
e
no problem. good luck!
6 Views
Open in Slack
PreviousNext
Powered by