jolly-midnight-72759
11/16/2020, 7:22 PM./pants --changed-since="${REF}"
contains that pipeline. If multiple project pipelines have changed, then multiple triggers will be fired. The atomic unit of v2 is the file. So that implies that when looking at the output of change-since
, the tool needs to create a list of all the projects involved then fire the triggers. That begs two questions:
* How do I define a project?
* How do I map a file to a project?nyc/src/python/brooklyn
nyc/src/python/queens
nyc/tests/python/brooklyn_tests
nyc/tests/python/queens_tests
then the project would be nyc
and not brooklyn
and queens
.hundreds-father-404
11/16/2020, 7:27 PMjolly-midnight-72759
11/16/2020, 7:27 PMhundreds-father-404
11/16/2020, 7:28 PMNote that even though the projects live in different top-level folders, you are still able to import from other projects. If you would like to limit this, you can use ./pants dependees or ./pants dependencies in CI to track where imports are being used. See Project introspection.(Someone was also wondering about a Pants feature to automate some of that) Tangibly, youβd want to use
--changed-dependees=transitive
here I thinkjolly-midnight-72759
11/16/2020, 7:30 PM./pants roots
π€©hundreds-father-404
11/16/2020, 7:31 PMwitty-crayon-22786
11/16/2020, 7:33 PMjolly-midnight-72759
11/16/2020, 7:33 PMnyc/BUILD
file have a "cicd_pipeline" target?witty-crayon-22786
11/16/2020, 7:34 PMjolly-midnight-72759
11/16/2020, 7:34 PMhundreds-father-404
11/16/2020, 7:34 PMif you defined a target type associated with the pipeline, you could filter to just that typeWhat would this look like in terms of
--changed-since
? Users would have to add an explicit dependencies
field to the pipeline target so that it knows which files belong to it?
(The target having a sources
field that globs over the whole project isnβt a great idea, but possible. It means that things like file args will resolve to both targets, the Python one and the pipeline one)But not as complex as the jupyter one.Yeah this would only be using the Target API, which is much much easier. No Rules API this time
witty-crayon-22786
11/16/2020, 7:35 PMjolly-midnight-72759
11/16/2020, 7:35 PM./pants pipeline target
.hundreds-father-404
11/16/2020, 7:36 PMwe could runΒ .pants pipeline target.That part would require the Rules API hehe. But weβre happy to help with that
witty-crayon-22786
11/16/2020, 7:36 PMjolly-midnight-72759
11/16/2020, 7:37 PMβ File: build-support/buildkite/generate-project-tests-pipeline.sh
ββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
1 β #!/bin/bash
2 β
3 β set -eu
4 β
5 β echo "steps:"
6 β
7 β for atest in $(./pants filter --filter-target-type="python_tests" ${1}::)
8 β do
9 β echo " - command: \"./pants test ${atest}\""
10 β echo " label: \"${1} test $(basename ${atest})\""
11 β done
witty-crayon-22786
11/16/2020, 7:38 PMjolly-midnight-72759
11/16/2020, 7:38 PMgenerate-poject-tests-pipeline.sh nyc
would create a pipeline that tests ALL the stuff in the nyc
project. I'd like this to only test the changed-since
stuff.hundreds-father-404
11/16/2020, 7:39 PMsing targets like i was suggesting would require that the target had useful dependencies on code in the project.Thatβs a pretty major restriction and will be easy for your coworkers to forgot to do, which is dangerous with CI There are two possibilities I see with the custom target approach: 1) Set the
sources
field to glob over everything. Itβs not a huge deal if a non-Python target and a Python target refer to the same thing afaict. (Two Python targets means dep inference doesnβt work). Things like test
will just ignore the irrelevant file. The downside is that the output of --changed-since
and list
will be longer, which may confuse your coworkers
2) Benjy speculated the other day if we should add dependencies
globs..I highly doubt we want that, including for perf reasons, but itβs possible perhaps.witty-crayon-22786
11/16/2020, 7:40 PMhundreds-father-404
11/16/2020, 7:40 PMwitty-crayon-22786
11/16/2020, 7:41 PMhundreds-father-404
11/16/2020, 7:42 PM--changed-since
mixed with the script, except that parts of it would be automated, like using Pants to filter out irrelevant thingsjolly-midnight-72759
11/16/2020, 7:59 PMchanged-since
is mutually exclusive of target arguments. Couldn't this be solved by ./pants --changed-since=${REF} --target-regex="nyc*"
or something like that? I could just use grep
on the output of the changed-since
, but this seems like something filter
could own, too.witty-crayon-22786
11/16/2020, 8:00 PMjolly-midnight-72759
11/16/2020, 8:01 PM./pants pipeline
path prematurely.pants
methods. I suspect there is an easy pattern in here that I just haven't kneaded out yet.