https://pantsbuild.org/ logo
Join Slack
Powered by
Are the tests <described here> the "only" way to i...
# development
g
Are the tests described here the "only" way to implement tests? I'm looking at structure tests for my OCI builder; but it seems like a fairly orthogonal approach to what's described there; as I'm pretty much in the situation one test target = one container = one process.
h
I don't think it's documented in the plugin API docs yet, but we now have support for batching tests. See https://www.pantsbuild.org/docs/python-test-goal#batching-tests for the user facing explanation. https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/goals/pytest_runner.py#L431 for the partitioning code What would be your ideal semantics? Note that Pants's caching is based on the batch-level you have. If you stick with one test target == one process == one container, you'll have that fine-grained of caching. If you batch, the cache will be at the level of the batch
f
Also the right way to structure a test target can also depend on the particular language / technology. For example, 
go_package
is usable with the 
test
goal because in Go it is whole packages which are tested, not individual files.
➕ 1
So that page's instruction "When creating a test target, you should usually subclass 
SingleSourceField
" would really only apply to a situation where it makes sense to test file-by-file.
👍 1
How do you see the tests being structured for the OCI builder backend?
g
So right now I'm looking at structure tests primarily; so: build the container, and validate contents of the built image. Specific files in specific locations, maybe file contents. Layer and image digests. That kind of thing. So it'll likely look something like this:
Copy code
oci_build_image(
   name="foobar",
)

oci_structure_test(
    image=[":foobar"],
    files=["/bin/app/__main__.py"],
    digest="abc123efefefef",
    ...
)
So this technically doesn't require spawning the container, but it'll be quite a lot of work per container to unpack and run the tests, so doing once-per makes most sense.
Eventually I want to do fancy things like "build a pex into a container and then run some Python tests in it". But that's a completely different workflow.
So this is more like "goldens" in terms of code-generation etc, more than traditional testing.
I.e., the code will - for the foreseeable future at least - behave more like 
run
in terms of how it's bundled and executed. Which is very different (I think) from the sharding/batching etc that the page describes. And I don't really need a test runner, and so on.
To back up a step, I think it makes me a bit confused because it seems like every new thing I want to support is "custom" and requires a wholly new abstraction. If I want a repl ("interactive run in container") registering and using that is different from run, which is different from package, which is different from publish, and then for test I need another method. So I'm wondering if this "is the way" or whether the docs are just summarizing "one method" of many.
I.e, sometimes I need a command line arg, sometimes I need a field set, sometimes a request, sometimes a subsystem, and so on. It's a bit all over the place. I just want... to do the dumb thing first.
f
I do not believe much effort has been given to ensure that all of the various goal-related plugin APIs are consistent with each other.
The docs you linked are summarizing one way of doing things: for languages where one file is the "unit of build"
The docs should probably explicitly state that assumption so it does not confuse readers.
As I mentioned earlier, the Go backend for example ignores some of that advice because the "unit of build" in Go is the package and not a file.
(and ignores because the advice is inapplicable to Go)
You are in the best position to know how to model your backend's concepts.
So model it the way you want is my vote.
one test target = one container = one process
seems fine by me
➕ 1
And if a better way to model OCI image testing is conceived in the future, then the backend can evolve at that time.
g
Thank you! Yeah, for sure, I think I can do the modelling. My confusion stems from "is one way of doing things"... I'm super-happy that any docs exist, but it seems like it's diving in at the deepest end with sharding and collation and test runner subsystems, etc. What I normally do is read the docs, then go look at some backend for practical implementations. But if the one I look at does all of the steps per the docs, I can't know the "simpler" method.
I.e, the other guides are more written like "this is the bare minimum", and if that is the bare minimum for testing... 😅
h
The plugin API docs were written when Python and Shell were the only languages, and those both have the same file-based semantics. They definitely need a refresh now that Pants has been more generalized for multiple languages / frameworks
g
Gotcha, makes sense. I'll have a peek at the go test setup and see what I can learn from there! And maybe PR back a simplification of the guide if I can skip any major steps. 🙂
❤️ 1
🙌 1
👍 2
2 Views
Open in Slack
PreviousNext
Powered by