https://pantsbuild.org/ logo
#general
Title
# general
b

brief-engineer-67497

11/26/2018, 5:10 PM
Hello, I'm having issues chaining tasks. I'v read https://www.pantsbuild.org/dev_tasks.html and I uses
Copy code
def product_types(cls):
    return ['expanded_yaml']
and
Copy code
def prepare(cls, options, round_manager):
    super(MyGenerator, cls).prepare(options, round_manager)
    round_manager.require_data('expanded_yaml')
but then how do I pass the value I'm interrested in ? (i.e the output directory in my case) I tried
self.context.products.register_data('expanded_yaml', target_workdir)
for the producer and `self.context.products.get_data('expanded_yaml')`for the consumer. It seemed to work but after cleaning all caches and invoking directly gen on the consumer target, the result of
self.context.products.get_data('expanded_yaml')
is None …
e

enough-analyst-54434

11/26/2018, 5:15 PM
As a sanity check, does
./pants --explain gen
show that your task - installed in the gen goal - depends on the task that calls
self.context.products.register_data('expanded_yaml', target_workdir)
?
b

brief-engineer-67497

11/26/2018, 5:19 PM
not sure how to read the output :
Copy code
./pants --explain gen
Goal Execution Order:

bootstrap -> imports -> unpack-jars -> deferred-sources -> gen

Goal [TaskRegistrar->Task] Order:

bootstrap [substitute-aliased-targets->SubstituteAliasedTargets_bootstrap_substitute_aliased_targets, jar-dependency-management->JarDependencyManagementSetup_bootstrap_jar_dependency_management, bootstrap-jvm-tools->BootstrapJvmTools_bootstrap_bootstrap_jvm_tools, provide-tools-jar->ProvideToolsJar_bootstrap_provide_tools_jar]
imports [ivy-imports->IvyImports_imports_ivy_imports]
unpack-jars [unpack-jars->UnpackJars_unpack_jars]
deferred-sources [deferred-sources->DeferredSourcesMapper_deferred_sources]
gen [antlr-java->AntlrJavaGen_gen_antlr_java, antlr-py->AntlrPyGen_gen_antlr_py, jaxb->JaxbGen_gen_jaxb, protoc->ProtobufGen_gen_protoc, ragel->RagelGen_gen_ragel, thrift-java->ApacheThriftJavaGen_gen_thrift_java, thrift-py->ApacheThriftPyGen_gen_thrift_py, wire->WireGen_gen_wire, yaml-template->YamlTemplateApply_gen_yaml_template, twirl->TwirlGen_gen_twirl, controllers->ControllersGen_gen_controllers, routes->RoutesGen_gen_routes]
e

enough-analyst-54434

11/26/2018, 5:20 PM
What are the the names of the task types you wrote and what order do you expect them to run in?
b

brief-engineer-67497

11/26/2018, 5:20 PM
the producer is
YamlTemplateApply_gen_yaml_template
and the consumer
ControllersGen_gen_controllers
e

enough-analyst-54434

11/26/2018, 5:21 PM
ok - so they are both installed in the gen goal and in the right order, yaml before contoller
b

brief-engineer-67497

11/26/2018, 5:22 PM
yes, I was a bit surprise that I have to declare the
backend_packages
in the right order in the pants.ini to make it work.
e

enough-analyst-54434

11/26/2018, 5:22 PM
It seemed to work but after cleaning all caches and invoking directly gen on the consumer target,
Does the consumer target express a dependency on a target pointing to the file the yaml task operates on?
(express a BUILD file dependency?)
b

brief-engineer-67497

11/26/2018, 5:24 PM
yes
e

enough-analyst-54434

11/26/2018, 5:24 PM
Yes, I was a bit surprise that I have to declare the
backend_packages
in the right order in the pants.ini to make it work.
Flagging this for a re-visit after figuring out main issue.
b

brief-engineer-67497

11/26/2018, 5:24 PM
(ok ^_^)
e

enough-analyst-54434

11/26/2018, 5:25 PM
OK - do the tasks use self.invalidated... guards?
b

brief-engineer-67497

11/26/2018, 5:25 PM
hmm both are
SimpleCodegenTask
and I implement mainly
execute_codegen
e

enough-analyst-54434

11/26/2018, 5:25 PM
ok
b

brief-engineer-67497

11/26/2018, 5:27 PM
if you want to have a look, I'm at this point https://gist.github.com/lgirault/7a748a911fc6cff528ad9be5ebf6588a
e

enough-analyst-54434

11/26/2018, 5:28 PM
b

brief-engineer-67497

11/26/2018, 5:28 PM
yes ?
e

enough-analyst-54434

11/26/2018, 5:29 PM
So, only when execute_codegen is called (which is only when target has changed) is the product registered
It needs to be registered on every run, even if grabbing the result from the cache.
Let me see if SimpleCodegen has a hook for this...
It does not.
b

brief-engineer-67497

11/26/2018, 5:30 PM
XD
damn
So I should do a pr on pants ?
e

enough-analyst-54434

11/26/2018, 5:31 PM
I think all prior uses of SimpleCodegen in the pants codebase worked against some implicit products (java and scala and python sources)
You should, yes.
All the dark corners are yours @brief-engineer-67497
b

brief-engineer-67497

11/26/2018, 5:32 PM
^_^
so I'm still figuring out python. what is your advice to branch my build on a local version of pants ?
e

enough-analyst-54434

11/26/2018, 5:33 PM
We have that documented - just a second
See 'Running from sources' at the top of this doc: https://www.pantsbuild.org/howto_develop.html
b

brief-engineer-67497

11/26/2018, 5:34 PM
nice !
thank you very much
ok, I'll work on that
e

enough-analyst-54434

11/26/2018, 5:37 PM
The API is probably something like
register_products(self, target, target_workdir)
. So same shape as
execute_codegen
with a default noop implementation and the main difference that it's called for every vt after execute_codegen runs for the invalid ones: https://github.com/pantsbuild/pants/blob/14bfee29bb2f44c3b10e4144c67a118805522c45/src/python/pants/task/simple_codegen_task.py#L237
But just a sec before diving in - let me look at you gist a bit
b

brief-engineer-67497

11/26/2018, 5:38 PM
ok,
e

enough-analyst-54434

11/26/2018, 5:39 PM
Other impls don't pass data through products, they generate synthetic targets to own the generated files and inject those targets in the graph
What does your yaml step generate?
b

brief-engineer-67497

11/26/2018, 5:40 PM
(btw I know that the registered target should be a dict indexed by the producing target, this is a first draft)
e

enough-analyst-54434

11/26/2018, 5:40 PM
The target that captures the generate stuff is
Resources
b

brief-engineer-67497

11/26/2018, 5:40 PM
my yaml step apply some templating
e

enough-analyst-54434

11/26/2018, 5:40 PM
What is the end result? CSS, HTML, something else?
b

brief-engineer-67497

11/26/2018, 5:40 PM
a bit like macros in c
yaml to yaml
it's to handle "generics"
e

enough-analyst-54434

11/26/2018, 5:41 PM
aha - ok
So, traditionally for codegen, things would work like this:
b

brief-engineer-67497

11/26/2018, 5:43 PM
and inject those targets in the graph
that seems like what I'm trying to do
e

enough-analyst-54434

11/26/2018, 5:43 PM
You'd introduce a
yaml_template
target that would own the templated yaml in use BUILD files. Your YamlTemplateApply would operate only on those targets and it would emit
yaml
targets as a result. The
ControllersGen
task would then only operate on
yaml
targets.
b

brief-engineer-67497

11/26/2018, 5:44 PM
maybe not what I've done but conceptually
e

enough-analyst-54434

11/26/2018, 5:44 PM
Here, better target names are likely
route_templates
and
routes
if the yamls are special routing yamls and not general yaml files.
b

brief-engineer-67497

11/26/2018, 5:45 PM
well I'm using the templating for a specific purpose but the engine is really generic (no pun intended)
it could be used for other purposes
I'll try to make it open source
e

enough-analyst-54434

11/26/2018, 5:46 PM
Let's work backwards. Is
ControllersGen
generic to any yaml file or should it only work on specific yaml files?
b

brief-engineer-67497

11/26/2018, 5:46 PM
this one need specific yaml files. ones that uses openapi
actually even more specific, openapi allow extensions and I use these fields to store information required by the code gen
Other impls don't pass data through products, they generate synthetic targets to own the generated files and inject those targets in the graph
yet the documentation speaks of
Products: How one Task consumes the output of another
. But I can see that some people want to remove them 😛
Copy code
# TODO(John Sirois): Kill products and simply have users register ProductMapping subtypes
    # as data products.  Will require a class factory, like `ProductMapping.named(typename)`.
e

enough-analyst-54434

11/26/2018, 6:07 PM
Yes, these are all a bit unrelated. It is true that codegen generally takes in a special target type (
protobuf
) and synthesizes to a 'normal' target type (
java-library
) instead of using products. It's also true that products currently has a split between products that are just string keys vs products that are full blown custom types (the comment you reference).
b

brief-engineer-67497

11/26/2018, 6:10 PM
I was just being mischievous ^_^ but anyway I'm kind in a frozen state. I should go through the building graph node injection to achieve my target ?
e

enough-analyst-54434

11/26/2018, 6:12 PM
Sorry - paging between a few things.
So, the critical bit here is
ControllersGen
which looks broken in the gist. Presumably if you hand it random yaml it dies?
b

brief-engineer-67497

11/26/2018, 6:12 PM
no problem, as usual, I'm very grateful for your time
yes
e

enough-analyst-54434

11/26/2018, 6:13 PM
So it needs a special input target or field on an existing target from which to get special yaml
Right now you have a generic yaml templating task feeding it random yaml
b

brief-engineer-67497

11/26/2018, 6:15 PM
e

enough-analyst-54434

11/26/2018, 6:15 PM
Yeah
Ideas:
b

brief-engineer-67497

11/26/2018, 6:17 PM
and on a general architecture point of view I totally agree with your point but even If I make a generic yaml_template target and a specific mics_spec target wich extends the first one I have no way to ensure that it's a valid openapi spec
in a way, the compilation of the generated controller source is part of this validation
but I'm all ears
e

enough-analyst-54434

11/26/2018, 6:20 PM
1. Since template yaml is not valid yaml (presumably), the template files should have a different extension, say
.tyml
. 2. The generic
YamlTemplateApply
task could operate on any target with
.tyml
sources and produce matching
.yaml
output sources. 3. The
ControllersGen
task could operate on
MicsSpec
targets only, retrieving the mapped
.yaml
sources.
If that makes sense at a high level I can point to a few more details.
b

brief-engineer-67497

11/26/2018, 6:23 PM
it makes a lot of sense. just to nitpick, yaml is a very "open" format, so my template yaml is actually valid yaml (on a structural point of view)
e

enough-analyst-54434

11/26/2018, 6:24 PM
OK. Would it be onerous or wrong to demand a special extension?
b

brief-engineer-67497

11/26/2018, 6:25 PM
not at all
just to give you an example, In my template file I have
responses: $tref:DataResponse[<mics-type://myResource>]
and in my output file it would be
Copy code
responses:
      '400':
        $ref: '#/components/responses/error'
      '401':
        $ref: '#/components/responses/error'
      '404':
        $ref: '#/components/responses/error'
      '403':
        $ref: '#/components/responses/error'
      '500':
        $ref: '#/components/responses/error'
      '200':
        description: Success !
        content:
          application/json:
            schema:
              type: object
              required:
              - status
              - data
              properties:
                status:
                  type: string
                  enum:
                  - ok
                data:
                  $ref: <mics-type://MyResource>
      '503':
        $ref: '#/components/responses/error'
e

enough-analyst-54434

11/26/2018, 6:28 PM
OK. Then the only other details are: 1. In
YamlTemplateApply
synthetic_target_type
would return whatever the input target type was - basically it can't know or care where the
.tyml
is coming from or going to. 2. That's it actually!
This would allow
YamlTemplateApply
to remain generic and operate on
MicsSpec
, for example, without knowing it.
The product mapping goes away
And
ControllersGen
just operates on
MicsSpec
targets that own yaml files (the synthetic generated ones or otherwise hand-written ones that never ran through
YamlTemplateApply
).
Does that make sense?
(if so - no pants modifications needed)
b

brief-engineer-67497

11/26/2018, 6:32 PM
it makes sense. The only point is about the special extension. Using yaml I have syntaxic coloration out of the box without any configuration … I'll just check what It require to get it back (because on a semantic point of view, you're right a separate suffix would be appreciable)
e

enough-analyst-54434

11/26/2018, 6:33 PM
OK - cool.
b

brief-engineer-67497

11/26/2018, 6:33 PM
ok the "config" is like a no brainer
1. In
YamlTemplateApply
synthetic_target_type
would return whatever the input target type was - basically it can't know or care where the
.tyml
is coming from or going to.
hmm actually not sure what you mean by "return whatever the input target type was "
so it can be a YamlTemplate target or a MicsSpec target and it would "return" the same target ?
b

brief-engineer-67497

11/26/2018, 6:37 PM
ha ok, so you mean it quite litterally
XD
e

enough-analyst-54434

11/26/2018, 6:38 PM
And
is_gentarget
would simply look to see if the target has sources matching
.tyml
.
So it is type-agnostic as well.
b

brief-engineer-67497

11/26/2018, 6:41 PM
ok so that's for the
YamlTemplateApply
part but then what about the communication with
ControllersGen
? it would operates only on
MiscSpec
targets but how do I operate on the one synthetized by YamlTemplateApply rather than the original one listed in the dependencies ?
e

enough-analyst-54434

11/26/2018, 6:41 PM
This will mean, for each
a -> mics
in the graph you'll get a
a -> (mics, mics')
after the
YamlTemplateApply
task runs. So
ControllersGen
will need to be smart enough to scan both
mics
and skip it since it owns no yaml files, and
mics'
and use it since it owns generated yaml files.
In other words,
ControllersGen
now has two target selection criteria: 1. type(target) == MicsSpec 2. any(s.endswith('.yaml') for s in target.sources)
b

brief-engineer-67497

11/26/2018, 6:44 PM
ok but back to the
for each
a -> mics
in the graph you'll get a
a -> (mics, mics')
how do I add node/target to the graph ? I'm reading ProtobufGen but not sure I'm finding it
e

enough-analyst-54434

11/26/2018, 6:44 PM
SimpleCodegen does it for you
b

brief-engineer-67497

11/26/2018, 6:45 PM
what a nice guy !
^_^
cool
e

enough-analyst-54434

11/26/2018, 6:45 PM
Also, you'll find
mics'
has Target.derived_from ==
mics
Which I don't think is useful for your case, but is for some cases.
b

brief-engineer-67497

11/26/2018, 6:46 PM
ok
ok well, as we say in french "y'a plus qu'à"
(i.e all it remains is to do it)
thank you very much once again.
e

enough-analyst-54434

11/26/2018, 6:51 PM
You're welcome
b

brief-engineer-67497

11/26/2018, 8:20 PM
sooo … one last question
I have my target def
TemplatedYamlLibrary
and my task def
YamlTemplateApply
so far so good
e

enough-analyst-54434

11/26/2018, 8:21 PM
Why
TemplatedYamlLibrary
?
b

brief-engineer-67497

11/26/2018, 8:22 PM
I have my target def
ControllersLibrary
and my task
ControllersGen
e

enough-analyst-54434

11/26/2018, 8:22 PM
I thought the final idea was to be target agnostic in that task
b

brief-engineer-67497

11/26/2018, 8:22 PM
it actually is but I still need to declare a target
and I have a dictionary in the payload
e

enough-analyst-54434

11/26/2018, 8:24 PM
it actually is but I still need to declare a target
You don't. You can implement
is_gentarget(target)
and test the given target for
.tyml
sources.
b

brief-engineer-67497

11/26/2018, 8:25 PM
e

enough-analyst-54434

11/26/2018, 8:25 PM
Thanks - much easier to talk about something we can both look at.
b

brief-engineer-67497

11/26/2018, 8:25 PM
yes, of course
b

brief-engineer-67497

11/26/2018, 8:26 PM
yes it's a typo that it's still here
e

enough-analyst-54434

11/26/2018, 8:26 PM
Its a super-confusing api, but you can either define
gentarget_type
once or else implement
is_gentarget
which gets to say yes or no per active target in the current pants run.
b

brief-engineer-67497

11/26/2018, 8:26 PM
look at the
is_gentarget
e

enough-analyst-54434

11/26/2018, 8:27 PM
Aha
b

brief-engineer-67497

11/26/2018, 8:27 PM
yes my bad, I figured that out
e

enough-analyst-54434

11/26/2018, 8:27 PM
You did both
do one
ok
Was that the issue then?
b

brief-engineer-67497

11/26/2018, 8:27 PM
yes my bad, typo
well the issue is in the ControllersGen execution
e

enough-analyst-54434

11/26/2018, 8:27 PM
ok
b

brief-engineer-67497

11/26/2018, 8:28 PM
so back to my setup, I have a TemplatedYamlLibrary target instance, let's call it
spec
and a ControllersLibrary target instance, let's call it
controllers
e

enough-analyst-54434

11/26/2018, 8:29 PM
So your
is_gentarget
disagrees with your
execute_codegen
assertion.
b

brief-engineer-67497

11/26/2018, 8:30 PM
indeed
good catch
but actually the print in the is_gentarget never show
spec
even if
controllers
declare
spec
as a dependencie
maybe I misuse pants but my target goal is that for one
TemplatedYamlLibrary
I can output two artefacts a controller lib and route files. So I though that I needed 3 targets
I'm not yet at the route files target/task but it's to give you some context on my train of though
b

brief-engineer-67497

11/26/2018, 8:34 PM
yes ?
I think your extension is wrong.
b

brief-engineer-67497

11/26/2018, 8:35 PM
hmm indeed
e

enough-analyst-54434

11/26/2018, 8:35 PM
You want the codegen output extension type. You're telling SimpleCodegen what output files to collect into the synthetic target it generates
So, previously, you generated synthetic targets with 0 sources
Since the target workdir would presumably had no tyaml files emitted into it, just yaml files.
The main theme today is that SimpleCodegen is anything but simple 🙂
b

brief-engineer-67497

11/26/2018, 8:37 PM
yes !
^_^
ok where I'm troubled is that the more we go, the more I feel I have no need for ControllersLibrary
well it's confusing
I have a mental mapping target/task
so I setup a target to "store" the arguments I want to pass to my CLI
e

enough-analyst-54434

11/26/2018, 8:41 PM
Your mental model is pretty much spot on.
b

brief-engineer-67497

11/26/2018, 8:41 PM
but here my mental schema is broken because for the
ControllersGen
task I have on one hand my
TemplatedYamlLibrary
which contain the sources but on the other hand it's
ControllersLibrary
that contains all the other args for the CLI
e

enough-analyst-54434

11/26/2018, 8:43 PM
Why then have 2 targets? If
YamlTemplateApply
can work on any target with
.tyaml
sources, why can't it operate directly on the target with the
target.payload.dictionary
ControllersGen
needs?
b

brief-engineer-67497

11/26/2018, 8:44 PM
it has different concerns
a third output for the yaml template is a public openapi that we will publish to the client
e

enough-analyst-54434

11/26/2018, 8:45 PM
My naming: 1 target
controller
. It has sources that can be
.yaml
or
.tyaml
as wells as a
dictionary
. The
YamlTemplateApply
operates on these targets when they own
.tyaml
sources, the
ControllersGen
task always operates on these targets.
b

brief-engineer-67497

11/26/2018, 8:46 PM
so in one directory I want to declare my target responsible of the
tyaml
sources and I want to setup my controllers_library target next to my server code in another diroctory
ok …
I think I see where you're going
e

enough-analyst-54434

11/26/2018, 8:47 PM
Is the main issue then that a controllers data is split in 2 dirs right now?
Souce code in one tree, yaml in another?
But really they belong in same dir?
b

brief-engineer-67497

11/26/2018, 8:47 PM
one source directory, three targets pointing to the same input directory
well the spec and the "meat" of the code do not belong in the same directory because with the same yaml spec we will generate also the client code
e

enough-analyst-54434

11/26/2018, 8:49 PM
Will a controller always have a yaml spec? IE: does it make sense to have a
controller
target with 2 required fields:
dictionary
and
spec
, where spec is a target adress that must point to a yaml (or templated yaml)?
b

brief-engineer-67497

11/26/2018, 8:52 PM
well for me it was more "pipeline" of tasks where the second one did not need to know how its input is produce : the dictionary is a general entity related to templating, but on the other hand, it's true that the content of this dictionary is very oriented
so it could make sense yes
e

enough-analyst-54434

11/26/2018, 8:53 PM
Here's the problem I have - there is a ton of domain-specific important info. Is there any way you can present a full spec of what you need to do somewhere?
This is hard to puzzle out for me.
b

brief-engineer-67497

11/26/2018, 8:55 PM
what I cannot do is show the code I'm building but I think there is no problem with the build/tooling
and I can try to explain in a few words because I really think that what I'm tryng to achieve is not domain dependent
e

enough-analyst-54434

11/26/2018, 8:55 PM
At a minimum, a sample dir layout and what you need to be generated / compile given that dir layout would be extremely helpful
b

brief-engineer-67497

11/26/2018, 8:56 PM
yes, give me a few minutes
e

enough-analyst-54434

11/26/2018, 8:56 PM
OK - great. I really appreciate it.
b

brief-engineer-67497

11/26/2018, 9:03 PM
ok so here is a simplified version of our mono-repo layout: As I go, I'm not sure any code sample would help understanding but please say if I'm mistaken
Copy code
mono-repo/
   specs/
      service1/
         service1-api.tyaml
      service2/
         service2-api.tyaml

      dictionary.yaml   
   api/
      service1-models/
      service2-models/
   service1/
   service2/
actually we have ~ 40 services
dictionary contains the template definitions, here an exemple I gave earlier: in service1-api.tyaml
responses: $tref:DataResponse[<mics-type://myResource>]
, dictionary.yaml contains
Copy code
Data[ResourceSchema]:
  schema:
    type: object
    required:
      - status
      - data
    properties:
      status:
        type: string
        enum:
          - ok
      data: ResourceSchema

DataResponse[Resource]:
  200:
    description: Success !
    content:
      'application/json': $tref:Data[| {"$ref":"Resource"} |]
  400:
    $ref: '#/components/responses/error'
  401:
    $ref: '#/components/responses/error'
  403:
    $ref: '#/components/responses/error'
  404:
    $ref: '#/components/responses/error'
  500:
    $ref: '#/components/responses/error'
  503:
    $ref: '#/components/responses/error'
and in my output file will be
Copy code
responses:
      '400':
        $ref: '#/components/responses/error'
      '401':
        $ref: '#/components/responses/error'
      '404':
        $ref: '#/components/responses/error'
      '403':
        $ref: '#/components/responses/error'
      '500':
        $ref: '#/components/responses/error'
      '200':
        description: Success !
        content:
          application/json:
            schema:
              type: object
              required:
              - status
              - data
              properties:
                status:
                  type: string
                  enum:
                  - ok
                data:
                  $ref: <mics-type://MyResource>
      '503':
        $ref: '#/components/responses/error'
e

enough-analyst-54434

11/26/2018, 9:06 PM
OK - can you enumerate how you need to use the dictionary.yaml + .tyaml for a given service?
b

brief-engineer-67497

11/26/2018, 9:09 PM
image.png
that's an old slide Plugin means sbt plugin
so I need to apply dictionary.yaml once to a tyaml to get a "usable" spec
e

enough-analyst-54434

11/26/2018, 9:12 PM
SO, in brief, (tyaml -> yaml) -> spec -> (client + server)? Ie, 1 task to generate yaml, reason for the yaml agnostic, then 2 tasks to take that yaml - happens to be openapi spec, and turn into client and server?
b

brief-engineer-67497

11/26/2018, 9:12 PM
the "usable" spec is used to build controllers stubs, play route files (i.e two artifacts with the same end goal, building a server)
yes !
e

enough-analyst-54434

11/26/2018, 9:12 PM
OK - that is useful
b

brief-engineer-67497

11/26/2018, 9:13 PM
cool !
I'm sorry english is not my main language so I may not convey the message as clearly as I hope
e

enough-analyst-54434

11/26/2018, 9:15 PM
Understood and no worries. It is my main language and I have no clue how to use it effectively after 46 years.
So, drilling on 1 final point.
The templating task may be generic, but it needs 2 inputs: 1 - a dictionary with template values to plug in. 2 one or more template files to plug 1 into. Correct?
b

brief-engineer-67497

11/26/2018, 9:17 PM
yes
e

enough-analyst-54434

11/26/2018, 9:18 PM
Ok, and - on the client/server generation side, only yaml is needed. Neither client no server generator knows about templating, just the openapi yaml spec?
b

brief-engineer-67497

11/26/2018, 9:18 PM
yes, and client and server are generated separately
actually now the clients are still hand written and we are in the process of migrating servers to the generated stubs but that is the goal
e

enough-analyst-54434

11/26/2018, 9:21 PM
ok
So, I think
TemplatedYamlLibrary
needs to remain as the target type
YamlTemplateApply
operates on since
YamlTemplateApply
needs more than just
.yaml
files - it needs the dictionary too. I think, in full,
TemplatedYamlLibrary
needs 3 fields: + dictionary (required) + sources - 0 or more .tyaml files + target_type - the type of target to generate - defaulting to
Resources
This will allow you to write the generic
YamlTemplateApply
task but yet still have it emit a specific
Spec
target that both
ControllerGen
and
ClientGen
can look for and use. That
Spec
target can be hand-written in a BUILD file and point to a fully formed yanml file hand-written by a user, or be generated.
Does that work?
b

brief-engineer-67497

11/26/2018, 9:32 PM
yes
I think I begin to understand why I had so much trouble to grasp how pants work. I'm really used to a schema where I have task that behave like function and I can branch them together. Whereas with pants the "lifecycle" evolves around a target. And the more we go in this discussion the more I picture a target as a blueprint and tasks succeed each other to flesh out the target
is this correct ?
so the
target_type
is actually
type(self)
so to speak
ok the only drawback I see here is that for a given spec, I will have for exemple two targets
ControllerLibrary
and
ClientLibrary
that will have two differents output but the same first step (
YamlTemplateApply
) won't be shared. Or is a kind of hash used that make the two targets able to share this first step ?
e

enough-analyst-54434

11/26/2018, 9:43 PM
Not really, but I wouldn't get hung up on it either. A target is generally just files + metadata. Tasks operate on either products or else targets of interest (right file types, right metadata or a combo of both) or both, and produce new targets or else products or both. So tasks are functions with 2 possible sources of input, products or targets. They can likewise produce the same two types of output. Pants has a "new engine" that is in the process of development and it will eliminatye targets as a possible output type in favor of only products. As such, on the input side, targets will be turned into products intrinsically by the new engine.
ok the only drawback I see here is that for a given spec, I will have for exemple two targets
ControllerLibrary
and
ClientLibrary
that will have two differents output but the same first step (
YamlTemplateApply
) won't be shared.
I don't understand: YamlTemplateApply(TemplatedYamlLibrary) -> Spec ControllerLibrary(Spec) -> Contoller ClientLibrary(Spec) -> Client
Why can the last 2 task signatures use the same Spec instance generated by the 1st?
b

brief-engineer-67497

11/26/2018, 9:46 PM
Yes they absolutely can. Let me reread what you wrote above then ^_^
e

enough-analyst-54434

11/26/2018, 9:46 PM
I proposed
TemplatedYamlLibrary(dictionary, sources, output_target_type)
b

brief-engineer-67497

11/26/2018, 9:46 PM
so YamlTemplateApply is a task, TemplatedYamlLibrary and spec targets
e

enough-analyst-54434

11/26/2018, 9:48 PM
Concretely, in a BUILD:
Copy code
templated_yaml_library(
  dictionary='dictionary.yaml',
  sources=['service.tyaml'],
  target_type=spec
)
And, by the way, the .tyaml extension could now go away.
b

brief-engineer-67497

11/26/2018, 9:50 PM
I'm not clear with what is the target_type
e

enough-analyst-54434

11/26/2018, 9:51 PM
b

brief-engineer-67497

11/26/2018, 9:51 PM
is it a target(type) ie the synthetic_target type of the build graph node outputed by TemplatedYamlLibrary ?
e

enough-analyst-54434

11/26/2018, 9:51 PM
Yes
b

brief-engineer-67497

11/26/2018, 9:52 PM
ok but actually the same spec is used to describe both the client and the server
e

enough-analyst-54434

11/26/2018, 9:52 PM
Which is perfect - right?
You write 1 BUILD target like I showed above, it generates one target of type (alias)
spec
and both controller and client tasks operate only on
spec
targets
b

brief-engineer-67497

11/26/2018, 9:53 PM
ok so Spec is a synthetic target that I never declare in a build ?
e

enough-analyst-54434

11/26/2018, 9:53 PM
Correct
b

brief-engineer-67497

11/26/2018, 9:53 PM
okaaay
e

enough-analyst-54434

11/26/2018, 9:53 PM
You declare:
Copy code
templated_yaml_library(
  dictionary='dictionary.yaml',
  sources=['service.tyaml'],
  target_type=spec
)
You have the client and server code depend on this
The gen code generates a spec
The client and server gen code operate on the spec - which is in the dep graph because both client and server code depend on the original:
Copy code
templated_yaml_library(
  dictionary='dictionary.yaml',
  sources=['service.tyaml'],
  target_type=spec
)
The, if the user executes pants with only the client code in scope, only the client stubs get generated
If both server and client targets are in scope, both, etc...
b

brief-engineer-67497

11/26/2018, 9:55 PM
yes perfect
but so concretely I have in spec/service1/BUILD :
Copy code
templated_yaml_library(
 dictionary='dictionary.yaml',
 sources=['service.tyaml'],
 target_type=spec
)
so in service1/BUILD:
Copy code
controllers_library(
 dependencies = ['spec/service1']
)
e

enough-analyst-54434

11/26/2018, 9:59 PM
Yup
Ditto client code (in the future)
b

brief-engineer-67497

11/26/2018, 9:59 PM
yes
e

enough-analyst-54434

11/26/2018, 10:00 PM
And, to underscore, no more need for
.tyaml
, just yaml will do.
b

brief-engineer-67497

11/26/2018, 10:00 PM
so it's exactly what I was trying to achieve (modulo the target_type argument )
e

enough-analyst-54434

11/26/2018, 10:00 PM
The
t
is now embedded in
templated_yaml_library
.
b

brief-engineer-67497

11/26/2018, 10:00 PM
but I'm still not clear on the
ControllerGen
task
e

enough-analyst-54434

11/26/2018, 10:01 PM
The
target_type
is the only way to make the
YamlTemplateApply
as far as I can tell.
ControllerGen operates on
Spec
targets
No matter who wrote them, human or
YamlTemplateApply
.
Is there any other complication?
b

brief-engineer-67497

11/26/2018, 10:02 PM
ok but so, In one hand I have my input files in
Spec
and on the other hand I have my CLI args in
ControllersLibrary
how do I unite both ?
e

enough-analyst-54434

11/26/2018, 10:03 PM
ControllersLibrary has a mandatory
spec
argument that expects an address.
Just a sec for details on that...
Um, just a sec, more thought
b

brief-engineer-67497

11/26/2018, 10:03 PM
when
execute_codegen
is called, I operate on one target
e

enough-analyst-54434

11/26/2018, 10:05 PM
One questio before I answer - what are all the inputs needed by a
ControllersLibrary
target?
b

brief-engineer-67497

11/26/2018, 10:06 PM
the one that will be passed to the CLI:
Copy code
args = [
            '--controllers_package', target.payload.controllers_package,
            '--effect', target.payload.effect,
            
            '--base_route', sources_root,
            '--output', target_workdir
        ]
so
output
is given by
execute_codegen
,
sources_root
by
Spec
and I have two other arguments
e

enough-analyst-54434

11/26/2018, 10:09 PM
OK - and 1 more - I'm not sure which one of those args communicates the spec yaml file, but is the expectation exactly 1 or 1 or more spec yaml files as inout?
b

brief-engineer-67497

11/26/2018, 10:11 PM
1 or more yaml files
e

enough-analyst-54434

11/26/2018, 10:11 PM
OK, so, from your example above:
Copy code
controllers_library(
 dependencies = ['spec/service1']
)
b

brief-engineer-67497

11/26/2018, 10:12 PM
Copy code
controllers_library(
    package='controllers',
   effect='future',
   dependencies = ['spec/service1']
)
e

enough-analyst-54434

11/26/2018, 10:12 PM
The SimpleCodegen of
YamlTemplateApply
is alreadu expanding
dependencies = ['spec/service1']
to
dependencies = ['spec/service1', sopec.service1.prime]
b

brief-engineer-67497

11/26/2018, 10:13 PM
ok, so I search in my dependencies
e

enough-analyst-54434

11/26/2018, 10:13 PM
Yes
Look at direct dependencies
b

brief-engineer-67497

11/26/2018, 10:14 PM
ok
e

enough-analyst-54434

11/26/2018, 10:14 PM
The synthetic one will be there and you ignore the rest that are not
Spec
It would be messier if the controller gen code expected exactly one. Then you'd need to fail late when there were too many Specs / files at task runtime,
b

brief-engineer-67497

11/26/2018, 10:16 PM
ok, thank you
e

enough-analyst-54434

11/26/2018, 10:16 PM
I'm going offline here in ~10 minutes. Hopefully we've gotten this to a sensible state for the night.
b

brief-engineer-67497

11/26/2018, 10:16 PM
yeah hope so.
it's a bit late here, I really should go home 😛
anyway thanks a lot !!
6 Views