Rate-limiting tests because when too many run toge...
# development
a
Rate-limiting tests because when too many run together they allocate too much virtual memory https://travis-ci.org/pantsbuild/pants/jobs/539430254
Copy code
=================================== FAILURES ===================================
_________________________ FSTest.test_walk_single_star _________________________
self = <pants_test.engine.test_fs.FSTest testMethod=test_walk_single_star>
    def test_walk_single_star(self):
>     self.assert_walk_files(['*'], ['4.txt'])
pants_test/engine/test_fs.py:125: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pants_test/engine/test_fs.py:59: in assert_walk_files
    self.assert_walk_snapshot('files', filespecs_or_globs, paths, **kwargs)
pants_test/engine/test_fs.py:66: in assert_walk_snapshot
    result = self.execute(scheduler, Snapshot, self.specs(filespecs_or_globs))[0]
pants_test/engine/scheduler_test_base.py:74: in execute
    return self.execute_literal(scheduler, request)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <pants_test.engine.test_fs.FSTest testMethod=test_walk_single_star>
scheduler = <pants.engine.scheduler.SchedulerSession object at 0x7d533807fd68>
execution_request = ExecutionRequest(roots=((PathGlobs(include=('*',), exclude=(), glob_match_error_behavior=GlobMatchErrorBehavior(value=...ction(value='any_match')), <class 'pants.engine.fs.Snapshot'>),), native=<cdata 'struct ExecutionRequest *' 0xef0d810>)
    def execute_literal(self, scheduler, execution_request):
      returns, throws = scheduler.execute(execution_request)
      if throws:
        with temporary_file_path(cleanup=False, suffix='.dot') as dot_file:
          scheduler.visualize_graph_to_file(dot_file)
>         raise ValueError('At least one root failed: {}. Visualized as {}'.format(throws, dot_file))
E         ValueError: At least one root failed: (((PathGlobs(include=('*',), exclude=(), glob_match_error_behavior=GlobMatchErrorBehavior(value='ignore'), conjunction=GlobExpansionConjunction(value='any_match')), <class 'pants.engine.fs.Snapshot'>), Throw(exc=Exception('Snapshot failed: Error making env for store at "/tmp/tmppc8cyu15/directories/e": Cannot allocate memory',))),). Visualized as /tmp/tmps5j2j5w4.dot
h
I have no idea what the fix is here. Note that originally we hit this OOM error just in launching the tests. Now it looks like we sometimes get OOM when the tests are executing themselves Reducing memory usage is tracked by https://github.com/pantsbuild/pants/issues/7741
a
@witty-crayon-22786 has some context where we switched us from creating a scheduler-per-test to a scheduler-per-test-class because of the VIRT usage of making a scheduler…
h
VIRT?
w
virtual memory
đź‘Ť 1
caching will not affect the memory usage of the parent process, but it would eliminate the memory usage of the children that hit.