https://pantsbuild.org/ logo
#general
Title
# general
f

freezing-vegetable-92896

11/10/2021, 12:15 AM
Is there an equivalent of
::
that I can use in the
sources
field of a
files
target? I'm trying to package up an entire directory for use on spark. I know the better long term solution is to have specific targets for a given job, but I'd like to have something that just over-estimates to get something good enough without explicitly listing all the targets. We currently have a hack using a
files(sources=[""**/*.py""])
but that does not nice things to target discovery, so I'm hoping there is a better way to do this?
We currently have a hack using a
files(sources=[""**/*.py""])
but that does not nice things to target discovery, so I'm hoping there is a better way to do this?
h

hundreds-father-404

11/10/2021, 12:20 AM
Hm what do you mean not nice things?
sources=["**/*.py"]
is how you would do it Perhaps you want to use
resources
instead though? How are you trying to load these files?
f

freezing-vegetable-92896

11/10/2021, 12:24 AM
Hmm, maybe I was mis-using things, or pants 2.8 improved things. I was previously getting problems where it seemed like it was causing everything to depend on everything, but that might have gone away now
I’m just trying to get a zip file with the files in the same locations as they are in our repo which is how spark expects to find them
I suspect there is a better way to do this using pex, but we’re still feeling our way forward
h

hundreds-father-404

11/10/2021, 12:26 AM
I was previously getting problems where it seemed like it was causing everything to depend on everything
Generally, dependency inference doesn't happen for
files
and
resources
targets. You'd have to manually add to
dependencies
field of what you want, or for an
archive
target to its
files
field
I’m just trying to get a zip file with the files in the same locations as they are in our repo which is how spark expects to find them
Got it, so then it sounds like you're using
archive
and the
files
target? Cool
w

witty-crayon-22786

11/10/2021, 12:30 AM
you’re trying to capture all python files in, yea?
it’s hard to do that correctly with globs…
files
will not strip source roots, so if you wanted them to actually be loadable as python code by spark, you’d probably want to use
resources
1
but yea, in the medium term, you’d probably want an explicit
dependencies
list somewhere for only the “reflective access” portion of your code. if you have an explicit
import
statement, you shouldn’t need something in
dependencies
… but if it is loaded via a string, you might need either an explicit dependency in the
dependencies
list (which is the largest single
dependencies
list in Pants own codebase, used for our plugins), or alternatively, if you know that a type is loaded via a string, you can enable [python-infer].string_imports
f

freezing-vegetable-92896

11/10/2021, 12:36 AM
We’ve been able to get something to work with a
Copy code
files(
    name="files",
    sources=["**/*.py"]
)
target. The code we need here happens to not use source roots in a problematic way, so that constructs something we can use in spark
👍 1
I definitely plan to use dependency inference in the future, right now we are just looking for a quick hack that gives us something we can use for arbitrary jobs that arn’t set up with pants yet
1
h

hundreds-father-404

11/10/2021, 12:38 AM
Cool, makes sense! On that note, it's really important to us that Pants is easy to incrementally /partially adopt. So please do keep asking questions like this and let us know if there are things that would help!
👍 1
1
f

freezing-vegetable-92896

11/10/2021, 12:41 AM
The backstory here is I started using pants to hack out a rough and ready
dependencie.zip
for some new spark jobs we wanted to run because I was doing leading work towards fully adopting pants, then my co-worker made an extension with the
**/*.py
glob. In the commit am working on today to actually check in all the
BUILD
files and start us down the course of actually adopting pants proper I thought the recursive glob was a problem, but we don’t want to loose the functionality it enables until we get further in and have the better solutions in place
✔️ 1
Pants has generally been a pretty smooth path for me (especially once I discovered this slack and started asking questions, but it is taking some time to wrap my head around how everything works, and I am sometimes ending up in weird path dependent states where I think something is a problem because it was in a goofy hacky/incorrect state I was exploring earlier
b

busy-vase-39202

11/10/2021, 12:49 AM
Great to hear that it's been a generally good experience! We really appreciate hearing questions and especially POV from you and others on what takes time to wrap one's head around. That gives us important insights into how to improve docs, tutorials, etc. to make Pants more approachable for you as well as everyone else. So please do keep sharing your challenges to whatever extent you're comfortable, not only for your own benefit but also as a contribution to the project. We love that. :-)
👍 1
4 Views