https://pantsbuild.org/ logo
Join Slack
Powered by
I ran in to a weird bug(?) with the tagging system...
# general
d
I ran in to a weird bug(?) with the tagging system. I'm trying to build a CI pipeline in python. This CI pipeline calls out to pants to query targets and then runs the targets. However, I get different result if I run the same command in 
Popen
and just bash. 
Popen(shell=True)
has the same output as bash This is how I set up the command
Copy code
tag_filters = [f'--filter-tag-regex="{x}"' for x in test_group.filter_expression()]

filter_args = [
        "./pants",
        "filter",
        "--filter-target-type=python_tests",
        "--filter-granularity=file"
        *tag_filters,
        "::"
]
Calling with
Copy code
filter_process = subprocess.Popen(
        filter_args,
        stdout=subprocess.PIPE,
)
Pants seems to just ignore the tags that I'm trying to filter If I do
Copy code
filter_process = subprocess.Popen(
        " ".join(filter_args),
        stdout=subprocess.PIPE,
        shell=True,
)
the result is as expected Maybe I'm doing something silly?
h
Maybe try without the 
" "
escaping? With how Python arg handling works, I suspect that Pants thinks the tag includes the 
"
character
d
I tried that, same with and without
🤔 1
h
(also btw because those options are after the 
filter
goal, you can leave off the 
filter
part - 
--target-type
instead of 
--filter-target-type
)
🙌 1
Hm, different line of thought. I wonder if using 
subprocess.run()
instead of 
Popen
might make a difference? That waits for the process to finish. (And a really nice API in general!)
d
i also tried that 😂 the only thing that helped is 
shell=True
👍 1
pants didn't complain about unknown args, so I suspect the tag is somehow not understood?
Oh, maybe a red herring, but this is using 
PANTS_CONCURRENT
. CI runner is triggered by 
./pants run ci:runner
h
Okay weird. How about running 
./pants --tag=<blah blah> help global
and looking at the setting for 
--tag
*`./pants --filter-target-regex=<blah> help filter`, not 
help global
d
hmm, I just get the help message
no echo of what I passed in
h
Hmm you should see something like
Copy code
❯ ./pants --filter-tag-regex=blah help filter
...
  --filter-tag-regex="[[+-]regex1,regex2,..., [+-]regex1,regex2,..., ...]"
  PANTS_FILTER_TAG_REGEX
  tag_regex
      default: []
      current value: [
          "blah"
      ] (from command-line flag)
      Filter on targets with tags matching these regexes.
looks like that's written to stdout
d
yes, I do see that
h
so and when you run that from your script with Popen(shell=False), is it setting the value how you expect?
d
Copy code
current value: [

          "-big",

          "-integ",

          "-manual"

      ] (from command-line flag)
let me try shell = True
👍 1
Copy code
current value: [
          "-big",
          "-integ",
          "-manual"
      ] (from command-line flag)
🤔 1
It's the quotes...
🤦‍♂️
h
how so? they look the same
d
with quotes 
"\"-big\""
with shell=False
h
you have to escape with quotes when shell=False? Either way, oof to this whole thing
d
you have to escape with quotes when shell=False?
Not sure, I was playing with a few switches at the same time. No idea which one had which effect
👀 1
Moral of the story, I should have just written a sub system instead of bash in python
h
Eh, ideally both work though. I'm wondering if Pants could somehow change it's parsing to handle this more intuitively - I suspect the 
-
is what throws things off
w
you shouldn’t need the quote at all when 
shell=False
usually those quotes are stripped by the shell
i.e., when you invoke 
./pants --something="blah" …
in a shell (say, bash), bash eats the quotes (tokenizes) before invoking the child process
but if you’re invoking a process directly with 
shell=False
, you are expected to pre-tokenize everything… so passing 
"--something=blah blah blah"
for 
shell=False
is fine and with pass “blah blah blah” for the flag, because you have pre-tokenized it
Open in Slack
PreviousNext
Powered by