Does pants have any native support for handling prob by igno

Question

Does pants have any native support for handling prob by ignoring directories that are git submodules Or perhaps there s a better approach to solving my problem One of my test fixtures has to be a git submodule as the code under test runs `git worktree` For other directories with similar contents I can just import something typically a PATH var to that directory generated with an test infra function But if I do that across the git submodule boundary python will fail to import if the git submodule isn t checked out and pants will complain about any dependencies on that empty directory more in the thread

proud-dentist-22844 · Answer

See the `test content version` directory in Here s the gitmodules file

proud-dentist-22844 · Answer

for the similar folders that are not git submodules I m doing this this fixture file a BUILD file makes this depend on all the relevant fixture metadata imported in the test file And here s where that imported var is used

proud-dentist-22844 · Answer

And the comparable bit in the test file that uses `test content version` git submodule directory is here With this utility test function call here the utility function here checks to make sure this submodule is checked out before running the relevant tests

proud-dentist-22844 · Answer

How can I tell pants about this dependency on a git submodule

enough-analyst-54434 · Answer

So you want to allow the submodule to be checked out or not dealer s choice and then have the tests run or not accordingly You don t want to iron fist and force folks to check out the submodule

enough-analyst-54434 · Answer

gt One of my test fixtures has to be a git submodule as the code under test runs `git worktree` Just to chase down all avenues as the the one doesn t obviously follow from the other A test can create its own ephemeral git repository with `git init` etc in a tmp dir to then exercise `git worktree` against

enough-analyst-54434 · Answer

If you must have a git submodule for this the only quick thing I can think of is it be nested 1 level below a dir that contains a BUILD file that uses recursive globs and maybe includes a dummy file if needed to stop Pants complaining When the git submodule subdir is hydrated checked out the globs capture those files When its not the globs capture nothing except perhaps the dummy file if needed to appease Pants

proud-dentist-22844 · Answer

I m trying to introduce pants without re architecting all the tests introducing as few changes as possible I explain why it is a git submodule not to say that s the only way but to say that s probably why whoever wrote the tests wrote them that way gt checked out or not dealer s choice I think the idea is that it s required you must checkout the submodule The utility function serves to raise a helpful error message before running the tests that need it if it s not there <https github com st2sandbox st2 blob pants st2tests st2tests fixturesloader py L151 L157> Next time I work on this my eyes are drooping bed is calling stuck out tongue I need to find a way to accomplish that fail nicely purpose within the context of pants

proud-dentist-22844 · Answer

I wonder if I can use a symlink to improve this somehow

proud-dentist-22844 · Answer

OK I think I found part of a decent way to handle this Add a sibling directory with the BUILD file that includes anything needed from the submodule which won t have BUILD files in it ```~ p st2sandbox st2 git st2tests st2tests fixtures packs $ cat test content version fixture BUILD pack metadata name= metadata sources= st2tests st2tests fixtures packs test content version pack yaml st2tests st2tests fixtures packs test content version yaml st2tests st2tests fixtures packs test content version icon png python library name= test content version dependencies= metadata sources= st2tests st2tests fixtures packs test content version py python library dependencies= test content version ``` That of course leads to some warnings about unmatched globs ```12 49 28 19 WARN Unmatched globs from st2tests st2tests fixtures packs test content version fixture metadata s `sources` field st2tests st2tests fixtures packs test content version fixture st2tests st2tests fixtures packs test content version yaml st2tests st2tests fixtures packs test content version fixture st2tests st2tests fixtures packs test content version icon png st2tests st2tests fixtures packs test content version fixture st2tests st2tests fixtures packs test content version pack yaml Do the file s exist If so check if the file s are in your ` gitignore` or the global `pants ignore` option which may result in Pants not being able to see the file s even though they exist on disk Refer to 12 49 28 20 WARN Unmatched glob from st2tests st2tests fixtures packs test content version fixture test content version s `sources` field st2tests st2tests fixtures packs test content version fixture st2tests st2tests fixtures packs test content version py Do the file s exist If so check if the file s are in your ` gitignore` or the global `pants ignore` option which may result in Pants not being able to see the file s even though they exist on disk Refer to ``` Is there any way to enhance that error message from within a plugin I d like to add a note for that directory that explains how to initialize the git submodule

proud-dentist-22844 · Answer

I see where the error is generated Looks like it is in rust so there s probably no way to add more context to the warning <https github com pantsbuild pants blob main src rust engine fs src glob matching rs L509 L553>

proud-dentist-22844 · Answer

Here s the additional context <https github com pantsbuild pants blob main src rust engine src nodes rs L497 L504> I wonder if it would be possible to add a field to the target like `unmatched globs additional context` so that the target can provide additional information if the glob doesn t match Or maybe a function that wraps the sources field to inject that context `sources=ctx my glob on unmatched globs= You need to do git submodule `

proud-dentist-22844 · Answer

< happy kitchen 89482> wdyt

happy-kitchen-89482 · Answer

There is a way of adding context to unmatched globs via `description of origin` Right now this is set at the call site I m not sure about setting it from target information But certainly something we could consider if we saw more general uses for it < hundreds father 404> WDYT

happy-kitchen-89482 · Answer

Basically maybe <https github com pantsbuild pants blob 0592a169979de99a31bb03250951c98c5ccc0f67 src python pants engine target py L1316|here> we could check if the target itself provides any extra verbiage

proud-dentist-22844 · Answer

ok I wonder if there s a way to use that from a plugin now Maybe a target that grabs the sources globs from all the targets in the same BUILD file and then runs them through a customized `PathGlobs` with a custom `description of origin` wdyt

proud-dentist-22844 · Answer

Creating the target is simple enough but I m not sure what rule I could register that would get triggered simply because a target exists in a BUILD file

proud-dentist-22844 · Answer

wow I just looked at the dependency graph for the `dependencies` goal wow

happy-kitchen-89482 · Answer

I m not sure I understand what you re suggesting

happy-kitchen-89482 · Answer

< hundreds father 404> knows this part of the codebase far better than I do though

hundreds-father-404 · Answer

Yeah that would be tricky to allow plugins to change what the warning error message is for unmatched source globs It s possible but would require lots of special cased code in pants s core to expose a plugin hook and consume it My first thought was Danny s WIP PR to allow plugins to try and catch exceptions in rules which would allow you to overwrite the message But that won t work the rule for hydrating sources is used in lots of places in Pants s core and you would need to add try catches to each call site Instead the only way I see this happening is adding a plugin hook of some type

proud-dentist-22844 · Answer

Here is how I attempted to inject that error message but it doesn t work stuck out tongue the rule never actually gets called <https github com st2sandbox st2 commit 4712c96fc3f6418a7a14b7bb06abff072b54e205>

proud-dentist-22844 · Answer

wdyt Is there a way to make this hacky target work

proud-dentist-22844 · Answer

maybe inject dependencies isn t the best way to do this Perhaps I can piggy back on something else that needs to happen before or with dependency resolution

proud-dentist-22844 · Answer

Maybe something that makes the rule run for some of the goals dependencies dependees test

hundreds-father-404 · Answer

Oh You can subclass the Sources field and implement the class method validate snapshot error if things aren t what you expect The warning will still trigger before unfortunately but you can at least then error with good instructions

proud-dentist-22844 · Answer

I m fine with the warning still showing up as long as I can print the instructions So with that sources subclass I won t have to add a rule at all right Just the target and fields

hundreds-father-404 · Answer

Yep exactly

proud-dentist-22844 · Answer

Hmm Using `Sources validate resolved files ` doesn t work ``` original tgt WrappedTarget = await Get ^ SyntaxError await outside async function``` <https github com st2sandbox st2 commit c366c3b0fd5b173de6f5d0c606460c17d808f73c diff ccb0f023e13c60e488a16c3232eaed422915a5bbd8c995c009b437e521df1729R35|https github com st2sandbox st2 commit c366c3b0fd5b173de6f5d0c606460c17d808f73c diff ccb0f023e13c60e488a16c3232 5a5bbd8c995c009b437e521df1729R35>

proud-dentist-22844 · Answer

How could I deal with that

hundreds-father-404 · Answer

Ah yeah it can t use the rules API Note that the globs are already resolved no need to call `PathGlobs` again I was imagining that you d check something like `if self address spec path == some dir and not files` That is if it s the problematic target s see if the sources are missing or not

proud-dentist-22844 · Answer

So I ll need to extend the targets I want to add the error message to maybe with a mixin that adds the unmatched globs behavior not a separate unmatched globs target that depends on the targets with the real sources lists

proud-dentist-22844 · Answer

OK I m working on this and I think I almost have it From within `Sources validate resolved files ` can I access another field on the same target The closest I can find is `self address` but I can t turn that into a `Target` without the rules api afaict

proud-dentist-22844 · Answer

oh I see where fields get instantiated in `Target init ` So it looks like the Field only gets the value and address There is no way to get the error message from a different field

proud-dentist-22844 · Answer

So I would have to somehow pass the error message in through the field s value which might look odd Hmm

hundreds-father-404 · Answer

Yeah unfortunately you cannot access another field without the rules API at least currently But taking a step back to the top of the thread what s the intended behavior you want I m wondering if John s solution works <https pantsbuild slack com archives C046T6T9U p1622614433212800 thread ts=1622611477 211300 amp channel=C046T6T9U amp message ts=1622614433 212800|https pantsbuild slack com archives C046T6T9U p1622614433212800 thread ts=1622611477 211300 amp channel=C046T6T9U amp message ts=1622614433 212800>

proud-dentist-22844 · Answer

I didn t write those tests and trying to rewrite them is a whole other can of worms that I d rather avoid The more I change the tests the harder it will be to get buy in to merge pants into the main StackStorm

proud-dentist-22844 · Answer

So technically changing the tests is preferrable but politically Not so much

hundreds-father-404 · Answer

Oops sorry linked to wrong message I get what you mean there Iiuc the above thread correctly you don t want to require folks to init the git submodule right And the tests will already error if they try to run them <https pantsbuild slack com archives C046T6T9U p1622614612213000 thread ts=1622611477 211300 amp cid=C046T6T9U>

proud-dentist-22844 · Answer

yeah I did something almost like that here <https github com st2sandbox st2 blob pants st2tests st2tests fixtures packs test content version fixture BUILD>

proud-dentist-22844 · Answer

I m fine requiring them to init the submodule as long as I can print a good error message with instructions on what is missing

proud-dentist-22844 · Answer

Here s my next version But the error isn t caught like I expected where I override `Target init ` here I get this ```Traceback most recent call last File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev0 py37 lib python3 7 site packages pants bin local pants runner py line 234 in run inner return self perform run goals File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev0 py37 lib python3 7 site packages pants bin local pants runner py line 173 in perform run return self perform run body goals poll=False File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev0 py37 lib python3 7 site packages pants bin local pants runner py line 195 in perform run body poll delay= 0 1 if poll else None File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev0 py37 lib python3 7 site packages pants init engine initializer py line 136 in run goal rules goal product params poll=poll poll delay=poll delay File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev0 py37 lib python3 7 site packages pants engine internals scheduler py line 530 in run goal rule self raise on error t for t in throws File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev0 py37 lib python3 7 site packages pants engine internals scheduler py line 494 in raise on error wrapped exceptions=tuple t exc for t in throws pants engine internals scheduler ExecutionError 1 Exception encountered Engine traceback in select in pants backend project info dependencies dependencies in pants engine internals graph resolve targets st2tests st2tests fixtures packs test content version fixture in pants engine internals graph resolve unexpanded targets st2tests st2tests fixtures packs test content version fixture in pants engine internals graph resolve dependencies st2tests st2tests fixtures packs test content version fixture in pants backend python dependency inference rules infer python dependencies via imports st2tests st2tests fixtures packs test content version fixture in pants backend python dependency inference module mapper map module to address in pants backend python dependency inference module mapper merge first party module mappings in pants backend python dependency inference module mapper map first party python targets to modules in pants engine internals graph resolve targets in pants engine internals graph generate subtargets st2tests st2tests fixtures packs test content version fixture test content version in pants engine internals graph resolve source paths st2tests st2tests fixtures packs test content version fixture test content version Traceback most recent call last File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev0 py37 lib python3 7 site packages pants engine internals selectors py line 654 in native engine generator send res = func send arg File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev0 py37 lib python3 7 site packages pants engine internals graph py line 645 in resolve source paths sources field validate resolved files paths files File home cognifloyd p st2sandbox st2 git pants plugins unmatched globs target types py line 37 in validate resolved files raise UnmatchedGlobsError unmatched globs target types UnmatchedGlobsError``` Is there a way to catch that error so I can define the error message in the build file which makes the purpose of that funky BUILD file much clearer

hundreds-father-404 · Answer

Hm so that mixin won t do what you expect Hydrating source files is lazy and happens through the ~Target~ Rules API It does not happen in the `Target` constructor That try catch won t ever trigger how you want There is currently no plugin hook to try catch the `hydrate sources` rule from `pants engine internals graph py` Instead I was envisioning something like ``` BUILD custom tgt sources= maybe exist ``` ```class CustomSources Sources def validate resolved files self files Sequence str gt None if files return raise AssertionError Init git submodule ``` That is error if the files were not resolved meaning we can assume the git submodule is not set up Does that make sense

proud-dentist-22844 · Answer

That is what I m doing but I was trying to pull the ` Init git submodule ` error message from the BUILD file somehow

proud-dentist-22844 · Answer

I can put the messaging in the plugin if I can t find a way to do that I just thought it would be easier for people to reason about that odd BUILD file is the message was in it instead of in the plugin

hundreds-father-404 · Answer

Ah yeah right now the plugin hook can t access other fields from the target so you d want the message hardcoded in your plugin

proud-dentist-22844 · Answer

Oh I get it Duh I was trying to catch around `Target init gt Field init ` But I m not throwing the error in `Field init ` Bummer It was worth a shot

proud-dentist-22844 · Answer

I assume there s no way to silence the traceback yet right ```11 03 21 38 ERROR Exception caught pants engine internals scheduler ExecutionError File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev0 py37 lib python3 7 site packages pants bin local pants runner py line 234 in run inner return self perform run goals File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev0 py37 lib python3 7 site packages pants bin local pants runner py line 173 in perform run return self perform run body goals poll=False File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev0 py37 lib python3 7 site packages pants bin local pants runner py line 195 in perform run body poll delay= 0 1 if poll else None File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev0 py37 lib python3 7 site packages pants init engine initializer py line 136 in run goal rules goal product params poll=poll poll delay=poll delay File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev0 py37 lib python3 7 site packages pants engine internals scheduler py line 530 in run goal rule self raise on error t for t in throws File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev0 py37 lib python3 7 site packages pants engine internals scheduler py line 500 in raise on error wrapped exceptions=tuple t exc for t in throws Exception message 1 Exception encountered UnmatchedGlobsError Instructions go here```

hundreds-father-404 · Answer

Don t use print stacktrace

proud-dentist-22844 · Answer

I m not ```$ pants dependencies st2tests st2tests fixtures packs test content version fixture 11 42 14 13 ERROR Exception caught pants engine internals scheduler ExecutionError File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev1 py37 lib python3 7 site packages pants bin local pants runner py line 234 in run inner return self perform run goals File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev1 py37 lib python3 7 site packages pants bin local pants runner py line 173 in perform run return self perform run body goals poll=False File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev1 py37 lib python3 7 site packages pants bin local pants runner py line 195 in perform run body poll delay= 0 1 if poll else None File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev1 py37 lib python3 7 site packages pants init engine initializer py line 136 in run goal rules goal product params poll=poll poll delay=poll delay File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev1 py37 lib python3 7 site packages pants engine internals scheduler py line 530 in run goal rule self raise on error t for t in throws File home cognifloyd cache pants setup bootstrap Linux x86 64 2 6 0 dev1 py37 lib python3 7 site packages pants engine internals scheduler py line 500 in raise on error wrapped exceptions=tuple t exc for t in throws Exception message 1 Exception encountered UnmatchedGlobsError Instructions go here Use print stacktrace to see more error details ```