i'm invoking pants in a subprocess in a plugin in a random repo because i have a

jvm_binary()

that i'm using as a jvm tool. this works fine, but the output from the pants subprocess (compiling the jvm tool) is only visible all at once, at the end of the run, and not at all if the pants subprocess completes successfully (which seems very strange, to me). this is in a (linux) terminal -- i haven't tried it on an osx terminal. to invoke the pants subprocess, i'm using:

subprocess.check_call(
            cmd, # pants invoke command with cli args
            cwd=get_buildroot(),
            stdout=workunit.output('stdout'),
            stderr=workunit.output('stderr'),
            env=env)

env

is currently the os environment, but popping off

PANTS_ENABLE_ENTRYPOINT

from the env dict, because that's what's done in the pants subprocess bootstrap plugin inside twitter (i'm not sure why). this occurs when i use

Popen()

as well, both calls should normally not buffer the output like this, i think. i've been finding it difficult to understand how to debug this -- i don't think it would be a tty issue at all, but for some reason hooking up outputs like the above is buffering it all and writing it after the subprocess exits. i could not reproduce this behavior when invoking not-pants, and the compile and link tasks i recently wrote for c/c++ did not have this behavior. how might i start to debug this further?

What is the enclosing workunit label? https://github.com/pantsbuild/pants/blob/master/src/python/pants/reporting/plaintext_reporter.py#L75-L85

the single label is

WorkUnitLabel.BOOTSTRAP

(text block incoming):

def _build_binary(self, ensime_binary_target_spec):

    pants_config_files_args = ['"{}"'.format(f) for f in self._bootstrap_config_files]

    with temporary_dir() as tmpdir:
      cmd = [
        './pants',
        '--pants-config-files=[{}]'.format(','.join(pants_config_files_args)),
        '--pants-distdir={}'.format(tmpdir),
        'binary',
        ensime_binary_target_spec,
      ]

      env = self._get_subproc_env()

      with self.context.new_workunit(
          name='bootstrap-ensime-gen-subproc',
          labels=[WorkUnitLabel.BOOTSTRAP],
          # TODO: replace space join with safe_shlex_join() when #5493 is merged!
          cmd=' '.join(cmd),
      ) as workunit:

        try:
          subprocess.check_call(
            cmd,
            cwd=get_buildroot(),
            stdout=workunit.output('stdout'),
            stderr=workunit.output('stderr'),
            env=env)
        except OSError as e:
          workunit.set_outcome(WorkUnit.FAILURE)
          raise self.BootstrapEnsimeError(
            "Error invoking pants for the ensime-gen binary with command {} from target {}: {}"
            .format(cmd, ensime_binary_target_spec, e),
            e)
        except subprocess.CalledProcessError as e:
          workunit.set_outcome(WorkUnit.FAILURE)
          raise self.BootstrapEnsimeError(
            "Error generating the ensime-gen binary with command {} from target {}. "
            "Exit code was: {}."
            .format(cmd, ensime_binary_target_spec, e.returncode),
            e)

      dist_jar = self._collect_dist_jar(tmpdir)
      jar_fname = os.path.basename(dist_jar)
      cached_jar_path = os.path.join(self.workdir, jar_fname)
      shutil.move(dist_jar, cached_jar_path)

So that explains the suppression.

can't believe i didn't think to look at that -- will check with a different label

yw

ok, so with that change the output is coming out incrementally as well, which is great