https://pantsbuild.org/ logo
#development
Title
# development
n

narrow-vegetable-37489

07/27/2022, 4:20 PM
Could use some additional brain power regarding dependencies and dependency inference in the OpenAPI backend -> ๐Ÿงต
In the current implementation there are two target types,
openapi_definition
for the main OpenAPI file, and
openapi_source
for every json/yaml that is part of the definition (aka. referenced by
$ref
). The reason a specific
openapi_definition
target was added is because OpenAPI tooling usually takes this file as its argument, making it a lot easier to add support for various tools without having to "guess" which
openapi_source
files that are "entry points". It gets quite verbose though, since you'd likely want a
file
or
resource
for each
openapi_source
in order to use the OpenAPI definition in your code. So why not just piggyback off of
file
and
resource
directly, right? You have your assets and then you can "mark" your main OpenAPI file with
openapi_definition
. Well, the problematic part is dependency inference. Let's say we have this this tree:
Copy code
.
โ”œโ”€โ”€ BUILD
โ”œโ”€โ”€ a.json
โ”œโ”€โ”€ b.json
โ””โ”€โ”€ c.json
With this BUILD file:
Copy code
resources(sources=["*.json"])
openapi_source(source="a.json")
With the JSON files referencing each other (through
$ref
) like this:
Copy code
a.json -> b.json -> c.json
Then we'd want to end up with this dependency graph:
Copy code
openapi_source(source="a.json") -> resource(source="a.json") -> resource(source="b.json") -> resource(source="c.json")
With the current
openapi_definition
and
openapi_source
implementation its easy since the OpenAPI backend owns those two target types and can do whatever it wants, but if we start using
resource
and
file
instead we're suddenly dealing with core targets that may or may not be OpenAPI-related files, and then it gets a bit more complicated. One option is to let the
openapi_source
depend on all three files directly, aka. the graph becomes
openapi_source(source="a.json") -> resource(source="a.json") and resource(source="b.json") and resource(source="c.json")
, but that makes
./pants dependencies
,
./pants dependees
,
--changed-dependees=transitive
etc. rather unreliable which is suboptimal to say the least. Another option is for the OpenAPI backend to (optionally?) provide partial dependency inference for
AssetSourceField
by simply trying to do its
$ref
resolving on any JSON and YAML asset (maybe with some sort of skip field available). This should get us the result we want and I'm leaning towards this solution, but it could potentially be a tad too intrusive. A third option is to not do any automatic inference and simply leave it up to the user to specific
dependencies
between files. (sad face) The final option is to stick to the intitial implementation, as verbose as it is. I'd appreciate any and all thoughts and ideas on how to best proceed with this ๐Ÿ––
For some additional context, in OpenAPI documents (and JSON schemas I believe) you can reference JSON/YAML objects defined elsewhere in the file (and in other files) using
$ref
, e.g.
"$ref": "b.json#schemas/foobar"
.
Maybe
$ref
resolving in JSON/YAML `files`/`resources` should be decoupled from the OpenAPI backend altogether? People having JSON schemas in their repository would benefit from it too, and we could offer "partial dependency inference" for other
files
and
resources
types down the line, like .html could have automatic inference to CSS/JS/images, .css to images etc.
Another argument for it being geared towards `files`/`resources` in general rather than OpenAPI specifically is a potential
fmt
goal that any JSON/YAML user would benefit from, not only those with the OpenAPI backend enabled. So maybe some sort of "general purpose" JSON/YAML backend target towards
files
and
resources
that includes
$ref
resolving?
h

happy-kitchen-89482

07/27/2022, 8:53 PM
I think it does make sense to have
openapi_source
targets separate from `files`/`resources`, since these aren't treated like `files`/`resources` - i.e., they are not embedded inside binaries or placed in sandboxes at test time or whatever. They have domain-specific meaning.
But I guess to avoid verbosity, what you want is a
openapi_sources
(plural) generator, that generates individual
openapi_source
(singular) targets per file
This is exactly akin to
python_sources
generating
python_source
And note that your
openapi_definition
is somewhat akin to
pex_binary
, in that it points to an "entry point" of sorts
And in that case it probably doesn't want to have that entry point be in its
sources
field (because that same file is already in the
sources
of the
openapi_source
target and it's generally better to have a file be "owned" by a single target)
Instead you want some field like `pex_binary`'s "entry_point" field
that is logically distinct from the special
source
field, which has a specific usage and meaning
pex_binary
has no
sources
field, and so I'm suggesting that
openapi_definition
shouldn't either
But that is beside your main point...
The main thing for reducing verbosity here is an
openapi_sources
generator
n

narrow-vegetable-37489

07/27/2022, 9:20 PM
Thanks for the input! Yeah, I've already implemented generators, maybe that'll be good enough as you say. I'll carry on with the current implementation :)