https://pantsbuild.org/ logo
#general
Title
# general
e

echoing-farmer-15630

07/16/2021, 6:54 PM
Say, questions about dependencies. I have a project with two subprojects (libraries), say A and B, with subtrees of BUILD files. Each have their own requirements.txt files, with a fair amount of overlap (call the requirements a,b,c, whatever). And that means that a lot of the A library files generate the dreaded "The target A/foo/bar imports 'a', but ... it is ambigious which to use: ['A:a', 'B:a']" messages. I can resolve this by declaring each "A:a" in dependencies, but that's a lot of dependencies in a number of BUILD files under A, when I feel like there should be a way to say that "All the BUILD files under A should draw their dependencies from the
python_requirements
under A and not elsewhere in the project"
w

witty-crayon-22786

07/16/2021, 6:59 PM
in general, we’d recommend merging your requirements lists/files rather than having two of them
1
pants will extract the subset of the requirements that are relevant to each project automatically
e

echoing-farmer-15630

07/16/2021, 7:03 PM
In general I would too, but the requirements.txt files are also used for some non-pants magic with dockerfiles and virtualenvs so it's not as trivial as I'd like to do that in this case unfortunately, at least not at the present time. If there's a way to get pants to output requirements.txt-compatible dependencies for whole subtrees of a project, I may be able to create those sorts of files on the fly, but I'm not aware of that at least.
w

witty-crayon-22786

07/16/2021, 7:04 PM
h

hundreds-father-404

07/16/2021, 7:05 PM
Ack on interacting w/ non-pants, we definitely are trying to prioritize allowing Pants to interoperate with other workflows
w

witty-crayon-22786

07/16/2021, 7:05 PM
so in this case, something like
./pants dependencies --transitive --type=3rdparty subproject::
e

echoing-farmer-15630

07/16/2021, 7:05 PM
Oooooo, okay, very nice indeed.
🙌 1
w

witty-crayon-22786

07/16/2021, 7:06 PM
oh, hm. correction: you will need the
--transitive
flag i believe. editted the above.
e

echoing-farmer-15630

07/16/2021, 7:12 PM
Okay, that could take us a pretty long way, all right, though there will be some flailing along the way and quite possibly some custom plugins involved. Given the usual other fires, In the short term I'm probably going to do a lot of manual dependencies 🙂 I'm pretty much the, er, "pants champion" here and there are always some hurdles of course. But this sort of dependency resolution is absolutely marvelous; thanks so much. @hundreds-father-404, the situation here has to do with both some brittle "Create a virtual env" scripts but moreso with our multilayer dockerfile setup which doesn't play nicely with the docker plugin I saw for pants (but I'll keep looking and/or writing one) where we have to install the requirements for a lib at one layer and then copy or mount the source in another layer for hot-reloading and reduction of thrash. Knowing that I can create the requirements as necessary, though, is a huge help.
✔️ 1
h

happy-kitchen-89482

07/16/2021, 7:22 PM
One thing you can do is have the project-specific
requirements.txt
files "import" a common
requirements.txt
e.g.,
-r requirements-common.txt
is a valid line inside
subproject_a/requirements.txt
1
So you could put all the shared reqs in that common one, and that also ensures that the same versions are used across the subprojects
but generally we would like to enhance dep inference to be able to prefer "local" providers of symbols
🙏 1
to support use-cases like yours
w

witty-crayon-22786

07/16/2021, 7:24 PM
@echoing-farmer-15630: if you have time, i think @happy-kitchen-89482 or @curved-television-6568 might be interested in hearing about your docker setup too. the various usecases of docker are each a little bit different and overlapping, but the fact that you have a form of layering working is interesting.
c

curved-television-6568

07/16/2021, 7:48 PM
yeah, I’d be interested in hearing what workflows involving docker you have, @echoing-farmer-15630, so we can take that into the upcoming docker support in pants.
p

proud-dentist-22844

07/16/2021, 8:33 PM
but generally we would like to enhance dep inference to be able to prefer “local” providers of symbols
I would love this! If dependency inference could see that the subdirectory in the same directory/module of the python file that imports it (or very close to it) is preferred over another similarly named directory somewhere else in the repo, that would be a huge boon.
h

hundreds-father-404

07/16/2021, 8:34 PM
https://github.com/pantsbuild/pants/pull/12326 is a first step towards that type of support. (It only helps with
pex_binary
and
python_awslambda
, as those are the only places it's 100% safe to implement this shortcut)
w

witty-crayon-22786

07/16/2021, 8:35 PM
i’m less sure about that one… it’s really, really magical to have "nearby" modules chosen without ambiguity. the pex_binary case is not actually ambiguous because you’re matching a fully qualified single source file… the only reason we need to disambiguate is because we convert into a module first, which is what makes it ambiguous
1
h

happy-kitchen-89482

07/17/2021, 9:59 PM
I think we can capture some sensible intuitive behavior here. It seems to be useful.
2 Views