worried-painter-3138203/20/2023, 11:01 AM
bitter-ability-3219003/20/2023, 12:57 PM
worried-painter-3138203/20/2023, 1:05 PM
The more nuanced answer is that if we allowed sources it'd be difficult to make sure the sandbox has what youd expect for package dependencies, since we'd expose the package and all the input sources.How so? packages end up in
afaict, there's no file-level ambiguity when collecting docker context, right? If a dockerfile contains a
that matches an addresses filepath, I dont see how that could be misinterpreted?
To be fair you can do what I want by just `file`:ing everything.
bitter-ability-3219003/20/2023, 1:42 PM
worried-painter-3138203/20/2023, 1:50 PM
Pants internals doesn't package into dist for docker contexts thoughSo it does, bummer! Although, packaging has the convention of generating a dotted string as a containing directory, chances of collisions with sources ought to be slim, and detectable (warn or error), so I still don't see this as a showstopper, particularly?
Oh right and then there's the issue of file v resourceI understand that python wants to differentiate these, I dont know that docker_image has to care? Just copy the path if it matches something 🤷🏻♂️
docker build . -f <my-file>
fails with filenotfound, and the solution being add boilerplate.
pants package <my-file>
bitter-ability-3219003/20/2023, 2:09 PM
worried-painter-3138203/20/2023, 2:45 PM
but it hasn't been done yet. It's completely doable, yes, but it hasn't been done yet. And not in a way that is consistent with the rest of the pants ecosystem.I'm just prodding to wrack up ammo for an issue to open for this particular use case so I can get to it and make it so, because I like the idea. The codebase is malleable enough, I think, and letting
depend on any target with a
doesn't really change anything fundamental in the API? There's no implicit behaviour added either really, because of docker features.
So I agree we should be able to do this. I've been complaining about this flavor of assumptions for half a year now 😅.I did find your design doc for file_deps. I think the paradigm shift is not necessary to make usage of this particular backend (docker) easier to work with.
bitter-ability-3219003/20/2023, 3:35 PM
worried-painter-3138203/20/2023, 3:42 PM
is what you're thinking off?
Naive approach? Just hydrate the transitive dependencies under the directory?
bitter-ability-3219003/20/2023, 3:45 PM
instructions might start getting more than expected 😕
worried-painter-3138203/20/2023, 3:56 PM
If you have a target with a source field that depends on another target, user expectation would be to copy both. There is a dependency, after all. So transitive targets are a must.Are we talking build context or docker container? The latter I disagree with, and the former I think is what I think just works?
We might've solved the OP case by just looking at direct deps, but now we're still not doing what the user expects in the "normal Pants case" 😕There is no precedent what pants does in the normal case for docker_images, it doesn't support sources at all, currently. Conflating pex/python_distribution (search through an entry point) and docker_image copy instructions seem like an anti-pattern to me.
And just populating the sandbox with transitive deps isn't drop-in. Sandbox creation could take leagues more time due to the duplication of packageables and their deps, and like you point out existingYou're describing misuse though, I dont think that's a metric to optimize for? Potentially something to warn for though. And to theinstructions might start getting more than expected 😕
grabbing too much, sure that is not covered.
What I want:
When this gets hairy:
COPY some/packaged/thing # <-- this is not great, then, but dont you think this is also kinda unintuitive considering the above line?
bitter-ability-3219003/20/2023, 4:03 PM
line) or an explicit dep.
Instead of considering what Pants should do for inferred deps, we have to consider it in the very generic case of "the dependency exists", explicit or implicit.
And in that case, if a user has specified a target with a dependency, it's completely reasonable to expect the dependencies to exist in the context.
worried-painter-3138203/20/2023, 4:12 PM
If you have a target with a source field that depends on another target, user expectation would be to copy both. There is a dependency, after all. So transitive targets are a must.Can I not find transitive targets given COPY instructions and explicit deps, hydrate the "transitive_tgt.closure" sources in the sandbox, place build ctx at the root and let docker do its thing? What am I missing?
because otherwise we don't have enough info to know how to consume transitive dependencies.I think I do?
bitter-ability-3219003/20/2023, 4:21 PM
instructions, since inferred deps and explicit deps are the same to Pants: just direct dependencies.
For a docker image target, it depends on a
which itself has a
In the build context we should populate the file and it's dependencies and their dependencies.
Therefore we just populated the transitive targets of all our direct dependencies, and if we aren't careful that has unexpected consequences
worried-painter-3138203/20/2023, 4:29 PM
For the moment, forgetSure, I'm just making sure I'm not lost on the way. The goal should be to construct a correct build ctx given the docker instructions, after all. Okey, I think I need elaboration oninstructions, since inferred deps and explicit deps are the same to Pants: just direct dependencies.
and if we aren't careful that has unexpected consequencesfor this to fall into place for me. I agree with what you said until this point. I don't see how that is any different from assembling a pex, except I let it use
Thank you for humouring me btw 🙏
bitter-ability-3219003/20/2023, 4:37 PM
I don't see how that is any different from assembling a pex, except I let it useSandbox creation is exactly 1:1 with "building a PEX" or any other process for that matter. The difference is a PEX gobbles up everything in the sandbox (-ish, but lets simplify). In Docker we can't really assume anything because the user caninstead of
Consider a dockerfile with an explicit dependency on a
whose instructions are just along the lines of
worried-painter-3138203/20/2023, 5:54 PM
Consider a dockerfile with an explicit dependency on aI think most users would find success bywhose instructions are just along the lines of
trying to find the targets in
and iterate dependencies and hydrate the full thing, so that their build ctx doesn't error out on them.
The issue you're describing is real, agree, but it isn't difficult for the user who knows enough pants-foo to add explicit targets in BUILD-files to also add an ignore pattern for everything else that might be inferred by the
, if it is a performance issue? My suspicion is that the current
inference usage is more prevalent, but I have no data on this other than what my org does.
bitter-ability-3219003/20/2023, 5:58 PM
worried-painter-3138203/20/2023, 6:01 PM
in your example, what are you illustrating?
bitter-ability-3219003/20/2023, 6:02 PM
worried-painter-3138203/20/2023, 6:03 PM
or w/e it's called at
, and just materialize/hydrate all the things?
bitter-ability-3219003/20/2023, 6:04 PM
worried-painter-3138203/20/2023, 6:14 PM
Plugins and codegen can populate files anywhere in the sandbox, so inference alone doesn't solve the "what belong in the sandbox" question.True, but they could be explicit dependencies though, right?
bitter-ability-3219003/20/2023, 6:14 PM
worried-painter-3138203/20/2023, 6:53 PM
But for two, that's assuming the user wants everything under a directory and not some subsetThat is mapped to a target, yes I think so? That's already better than what
does, imo, because that exposes everything...
docker build .
to guide this op.
bitter-ability-3219003/20/2023, 7:32 PM
worried-painter-3138203/20/2023, 7:37 PM
I really don't think thats an assumption we can make for the user though. Especially since existing dockerfiles with pants already make assumptions about, well, not this 😅An option and/or feature flag can flip that switch, I dont think that's an issue in general?
I really don't think thats an assumption we can make for the user though.The current behaviour is also a decision, though. Authoring a docker_image target is the highest point of friction I encountered with devs in my org, and there's slack threads in #general asking what is going on
bitter-ability-3219003/20/2023, 7:45 PM