https://pantsbuild.org/ logo
#general
Title
# general
d

dazzling-diamond-4749

09/10/2021, 1:22 AM
I ran in to a weird bug(?) with the tagging system. I'm trying to build a CI pipeline in python. This CI pipeline calls out to pants to query targets and then runs the targets. However, I get different result if I run the same command in
Popen
and just bash.
Popen(shell=True)
has the same output as bash This is how I set up the command
Copy code
tag_filters = [f'--filter-tag-regex="{x}"' for x in test_group.filter_expression()]

filter_args = [
        "./pants",
        "filter",
        "--filter-target-type=python_tests",
        "--filter-granularity=file"
        *tag_filters,
        "::"
]
Calling with
Copy code
filter_process = subprocess.Popen(
        filter_args,
        stdout=subprocess.PIPE,
)
Pants seems to just ignore the tags that I'm trying to filter If I do
Copy code
filter_process = subprocess.Popen(
        " ".join(filter_args),
        stdout=subprocess.PIPE,
        shell=True,
)
the result is as expected Maybe I'm doing something silly?
h

hundreds-father-404

09/10/2021, 1:28 AM
Maybe try without the
" "
escaping? With how Python arg handling works, I suspect that Pants thinks the tag includes the
"
character
d

dazzling-diamond-4749

09/10/2021, 1:28 AM
I tried that, same with and without
🤔 1
h

hundreds-father-404

09/10/2021, 1:29 AM
(also btw because those options are after the
filter
goal, you can leave off the
filter
part -
--target-type
instead of
--filter-target-type
)
🙌 1
Hm, different line of thought. I wonder if using
subprocess.run()
instead of
Popen
might make a difference? That waits for the process to finish. (And a really nice API in general!)
d

dazzling-diamond-4749

09/10/2021, 1:31 AM
i also tried that 😂 the only thing that helped is
shell=True
👍 1
pants didn't complain about unknown args, so I suspect the tag is somehow not understood?
Oh, maybe a red herring, but this is using
PANTS_CONCURRENT
. CI runner is triggered by
./pants run ci:runner
h

hundreds-father-404

09/10/2021, 1:33 AM
Okay weird. How about running
./pants --tag=<blah blah> help global
and looking at the setting for
--tag
*`./pants --filter-target-regex=<blah> help filter`, not
help global
d

dazzling-diamond-4749

09/10/2021, 1:36 AM
hmm, I just get the help message
no echo of what I passed in
h

hundreds-father-404

09/10/2021, 1:38 AM
Hmm you should see something like
Copy code
❯ ./pants --filter-tag-regex=blah help filter
...
  --filter-tag-regex="[[+-]regex1,regex2,..., [+-]regex1,regex2,..., ...]"
  PANTS_FILTER_TAG_REGEX
  tag_regex
      default: []
      current value: [
          "blah"
      ] (from command-line flag)
      Filter on targets with tags matching these regexes.
looks like that's written to stdout
d

dazzling-diamond-4749

09/10/2021, 1:39 AM
yes, I do see that
h

hundreds-father-404

09/10/2021, 1:39 AM
so and when you run that from your script with Popen(shell=False), is it setting the value how you expect?
d

dazzling-diamond-4749

09/10/2021, 1:40 AM
Copy code
current value: [

          "-big",

          "-integ",

          "-manual"

      ] (from command-line flag)
let me try shell = True
👍 1
Copy code
current value: [
          "-big",
          "-integ",
          "-manual"
      ] (from command-line flag)
🤔 1
It's the quotes...
🤦‍♂️
h

hundreds-father-404

09/10/2021, 1:46 AM
how so? they look the same
d

dazzling-diamond-4749

09/10/2021, 1:46 AM
with quotes
"\"-big\""
with shell=False
h

hundreds-father-404

09/10/2021, 1:47 AM
you have to escape with quotes when shell=False? Either way, oof to this whole thing
d

dazzling-diamond-4749

09/10/2021, 1:48 AM
you have to escape with quotes when shell=False?
Not sure, I was playing with a few switches at the same time. No idea which one had which effect
👀 1
Moral of the story, I should have just written a sub system instead of bash in python
h

hundreds-father-404

09/10/2021, 1:50 AM
Eh, ideally both work though. I'm wondering if Pants could somehow change it's parsing to handle this more intuitively - I suspect the
-
is what throws things off
w

witty-crayon-22786

09/10/2021, 4:15 PM
you shouldn’t need the quote at all when
shell=False
usually those quotes are stripped by the shell
i.e., when you invoke
./pants --something="blah" …
in a shell (say, bash), bash eats the quotes (tokenizes) before invoking the child process
but if you’re invoking a process directly with
shell=False
, you are expected to pre-tokenize everything… so passing
"--something=blah blah blah"
for
shell=False
is fine and with pass “blah blah blah” for the flag, because you have pre-tokenized it