https://pantsbuild.org/ logo
#general
Title
# general
a

alert-boots-15052

04/23/2020, 9:14 AM
Hey all, seeing a bit of chatter about "pants v2". I'm trying to get a new project set up, is this still way too early for looking at that? Not seeing any info online about this
h

hundreds-father-404

04/23/2020, 2:45 PM
Hello! Are you using Python or another language? If python, we recommend starting with V2 by using a rewrite of the docs at https://pants.readme.io/docs/welcome-to-pants (some parts are still a work in progress) V2 is generally a better and more performance experience, outside of a few areas where we’re applying polish.
a

alert-boots-15052

04/24/2020, 2:08 AM
hmm yeah that looks good. My wrinkle is I also have some stuff running through grunt/webpack. I haven't quite found a good equivalent to Bazel's
genrule
(is it
prep_command
?), I guess I can write up a plugin for this pretty easily though
h

hundreds-father-404

04/24/2020, 2:08 AM
Are you using Python, JavaScript, or both?
a

alert-boots-15052

04/24/2020, 2:09 AM
Both. Basically a very large Django application with a frontend component (that gets dumped into static files)
conceptually I would just like to say "please run this command based on these files, some build results will end up here"
h

hundreds-father-404

04/24/2020, 2:11 AM
Cool. We have a similar setup at Toolchain. Django app + JavaScript. We’re starting to look into adding JS support to V2 in the #v2node channel, but for now, you would want to use the V2 implementation for Python (those docs) and the V1 implementation for JavaScript. Are you already using Pants? If so, which version?
a

alert-boots-15052

04/24/2020, 2:12 AM
I'm not using it yet (this is still exploratory), I have v1.27.0rc0 installed
👍 1
But yeah, we already have a "normal" build system and Pants being Python is very tempting, but I would ideally just like to wrap the existing system in Pants (then split it up later) but not seeing an easy way to do this
h

happy-kitchen-89482

04/24/2020, 2:17 AM
Can you clarify what you mean by "wrap the existing system in Pants"?
a

alert-boots-15052

04/24/2020, 2:18 AM
Well, for example now I run
grunt build-js-files
. I would love to just specify a Pants ... target? That just calls that command (but I specify the files that are needed to run that)
Like we have 5 or 6 build steps already, I think we could use Pants and caching to get advantages from the "same input/same output" principle.
h

hundreds-father-404

04/24/2020, 2:20 AM
Likely, it would be easiest to create a simple plugin/new goal to do this. You could do something like
./pants grunt-build '**/*.js'
, for example You specifically need to run
grunt build-js-files
, right? Any other commands?
a

alert-boots-15052

04/24/2020, 2:20 AM
If I wrote out a new goal, would I then be able to write a
py_test
target so that
./pants test
reruns that build if I have input changes?
Well there's Django stuff like
manage.py compilemessages
(for translation files), I could write up wrapper files for this probably?
h

hundreds-father-404

04/24/2020, 2:22 AM
While we don’t yet have official V2 support for Node, I think we could easily create a plugin to run
grunt build-js-files
or something similar on some targets. An explicit focus of the V2 version of Pants is to make it highly extensible and easy to add new functionality. We’ll be adding docs on how to write plugins next month, and I’d be happy to pair on it too if you think the plugin would do the job.
Regarding Python, there is already support for what you’re describing. Pants will understand when the inputs to the test command haven’t changed and will leverage the cache for those tests. Another option is
./pants --changed-since=HEAD test
, which will find all the changed files sine that commit and only run those
a

alert-boots-15052

04/24/2020, 2:25 AM
When trying stuff out I was hitting things telling me that the dependencies to a
python_tests
could only be
python_library
-like targets
h

hundreds-father-404

04/24/2020, 2:27 AM
Hm, I don’t think we have validation for that. What is the error message? Generally you would express that type of dependency via a
files()
or
resources()
target. For example, you could do this:
Copy code
files(
  name="static_assets",
  sources=["*.css", "*.html", "*.js"]
)

python_library(
  name="app"
  dependencies=[
    ":static_assets",
  ]
)
a

alert-boots-15052

04/24/2020, 2:28 AM
So basically what happens here.... is: •
grunt build-js-files
takes
src/***/**.js
and generates
dist/main.js
and
dist/assets/*.png
manage.py compilemessages
generates some
<http://messages.mo|messages.mo>
file for translations in the app •
py.test app/tests
runs tests (including selenium tests, which requires the above built files to be present)
can I make a`files` declaration for stuff that is generated from some other command (like a plugin that would run the commands above)?
sorry for spamming on this, it's very nice to be able to get help on this kind of stuff though
h

hundreds-father-404

04/24/2020, 2:32 AM
declaration for stuff that is generated from some other command
Good timing with this question! We’re actually actively working on support for codegen the next two weeks, meaning for example that you can convert Thrift files into Python code and depend on that in your
python_library
targets, for example. We could possibly leverage that, but I think we might be able to circumvent it for something simpler. Generally, a V2 goal can indeed create files in your build root. You could create a
grunt-build
goal that will write files to any location in the build root, like
dist/
. Question, do you check in
dist/
to version control?
a

alert-boots-15052

04/24/2020, 2:32 AM
no, we don't commit it
👍 1
h

hundreds-father-404

04/24/2020, 2:32 AM
sorry for spamming on this, it’s very nice to be able to get help on this kind of stuff though
No need to apologize! Happy to help 🙂 our goal is to solve problems exactly like you’re encountering!
no, we don’t commit it
Okay, then codegen would likely be more idiomatic, but in the meantime, I think we could still make it work. The semi-unidiomatic thing is that you will want to check in your BUILD file to VCS, so you will want your BUILD file defined one level above
dist
, like at the root level. Then, you could use a
sources
glob, like this:
Copy code
files(
  name="static_assets",
  sources=["dist/main.js", "dist/assets/*.png"],
)
Your targets can then depend on
//:static_assets
, where the
//
clarifies this target is at the build root.
a

alert-boots-15052

04/24/2020, 2:45 AM
Makes sense, I'll try to use this with a plugin or something to see if I can get things working. Definitely think that a general codegen entrypoint would make adoption of this tool much easier for existing projects 😃
💯 1
h

hundreds-father-404

04/24/2020, 2:47 AM
Are you thinking of trying to do a V1 plugin or V2? While we haven’t yet written the docs for V2 (it’s the next thing on my roadmap after I finish codegen), it would likely be much easier to implement and brings benefits like better caching. I’d be happy to pair with you on it 🙂 Beyond being happy to try to help, it would be useful feedback for me in writing the plugin docs on what makes sense and what is confusing.
a

alert-boots-15052

04/24/2020, 2:55 AM
I would be OK trying out the V2 plugin, is there some base Python class I can look at for that? And maybe an example file that's already in the project? I'm pretty good at back-solving from an existing implementation
What I'll do here is make a new repo with a simple version of my use case and I'll share the link in this Slack, and try to figure out the details
👍 1
h

hundreds-father-404

04/24/2020, 2:59 AM
And maybe an example file that’s already in the project? I’m pretty good at back-solving from an existing implementation
I’ll look for some good examples right now. Generally I’m the same with being good at back-solving, although warning that there are a fair amount of new concepts with writing V2 “rules” that might be confusing without a little bit of explanation. All the concepts are sound and inspired by functional programming, and once you know the basics it’s pretty simple to throw together new functionality, but it might be a little confusing without some orientation (whether that be the upcoming docs or someone talking it through). Of course, we’re happy to answer any questions along the way
src/python/pants/core/project_info/cloc.py
is a good example of a stand-alone goal that does something similar to what you’ll want for the Grunt goal. Ignore lines 26-46. First step to your new goal: create a subclass of
GoalSubsystem
and
Goal
, very similar to this file.
GoalSubsystem
is the public api, like any options and the name of the goal. Then, the
@goal_rule
is where the interesting logic happens. High level flow of what’s going on: 1) We’re asking for a couple things to be given to us by the V2 engine. These will be given via dependency injection; we don’t care how Pants gets us these things. 2) One of the things we’re asking for in the function signature is
SourcesSnapshots
. This is a way to ask for source files, where we don’t care at all about the metadata in targets. All we care about is the raw source files. If a user runs
./pants cloc :example
, we get all the source files belonging to the target
example
. If they say
./pants cloc example.js
, we’ll get that JS file. 3) We set up a
Process
, which is very similar to Python’s
subprocess.Popen
but is a way the Pants engine understands it and does things like safe caching. Note that we run
/usr/bin/perl
. For an initial prototype, it will be easiest if you can use an absolute path like
/usr/bin/grunt
. We can help you to generify this once it works. Also, note the
input_files
to populate the chroot with the files we need, and
output_files
to “capture” the result. 4)
await Get[ProcessResult](Process, req)
actually runs the process -- The other relevant file is
src/python/pants/core/goals/binary.py
, lines 69-71. It shows how to materialize files from a tmpdir into the actual build root so that you write to
dist/
a

alert-boots-15052

04/24/2020, 5:26 AM
plugin registration is the same as before yeah? I should be able to get stuff working through the plugin docs on the v1 documentation?
h

hundreds-father-404

04/24/2020, 5:35 AM
It’s similar in that you have a register.py file. The main entry point is rules() though. See backend/python/register.py as an example You also set backed_packages2 instead of backend_packages
3 Views