pantsbuild #development

# Retrieve file contents -- this will be
# 'First line.\nSecond line.\n'
contents = output.getvalue()

# Close object and discard memory buffer --
# .getvalue() will now raise an exception.
output.close()

full-kilobyte-93429

04/25/2016, 9:06 PM

otherwise every getvalue call will have to make a copy

full-kilobyte-93429

04/25/2016, 9:07 PM

anyway, moving into

closing

still fail https://travis-ci.org/peiyuwang/pants/builds/125675947

witty-crayon-22786

04/25/2016, 9:21 PM

@full-kilobyte-93429: "otherwise every getvalue call will have to make a copy" ... no. if you finish using the value before close, then you're fine.

witty-crayon-22786

04/25/2016, 9:22 PM

many usecases don't need a copy of the buffer to survive the call.

witty-crayon-22786

04/25/2016, 9:22 PM

and in fact, in the lmdb case, we don't need the buffer copied (except into the db's storage)

full-kilobyte-93429

04/25/2016, 9:23 PM

unless lmdb purges the memory in time?

witty-crayon-22786

04/25/2016, 9:23 PM

think of it from the other direction: if getvalue created a copy, then there would be no way to use it without copying

full-kilobyte-93429

04/25/2016, 9:24 PM

my above build is based on

InMemoryDb

will do another test for lmdb

wide-energy-11069

04/26/2016, 12:35 AM

@witty-crayon-22786: is only the selected (https://github.com/wisechengyi/pants/blob/f83e2b0272e2ff1c464e453c1f74d9335e18fcc3/src/python/pants/engine/exp/fs.py#L404-L420) in the scope of TODO? can you also clarify what

native

means?

wide-energy-11069

04/26/2016, 12:36 AM

does native mean in the fashion of

Sources

wide-energy-11069

04/26/2016, 12:55 AM

oic it’s a bunch of concrete classes

rough-minister-58256

04/26/2016, 12:59 AM

native as in, handled natively by the engine vs something a task would have to implement

wide-energy-11069

04/26/2016, 1:01 AM

things like

Copy code

class Dirs(datatype('Dirs', ['dependencies'])):
  """A set of Dir objects.”””

means a task will have to implement?

wide-energy-11069

04/26/2016, 1:02 AM

and what’s the difference between ^ and

Copy code

class Dir(datatype('Dir', ['path']), Stat):
  """A directory."""

  def __new__(cls, path):
    return super(Dir, cls).__new__(cls, six.text_type(path))

rough-minister-58256

04/26/2016, 1:03 AM

one is singular and the other is plural?

rough-minister-58256

04/26/2016, 1:03 AM

Dirs([Dir, Dir, Dir])

etc

wide-energy-11069

04/26/2016, 1:04 AM

rough-minister-58256

04/26/2016, 2:22 AM

to make something native means to move it out of a task implementation entirely - e.g. into its own Node type

rough-minister-58256

04/26/2016, 2:23 AM

e.g. the addition of

FilesystemNode

in https://rbcommons.com/s/twitter/r/3550/diff/4#10 and the removal of the non-native task triples in https://rbcommons.com/s/twitter/r/3550/diff/4#4

👍 1

rough-minister-58256

04/26/2016, 3:22 AM

got the closed loop runner working e2e with primed

BuildGraph

via a resident `Scheduler`:

rough-minister-58256

04/26/2016, 3:22 AM

Copy code

I0425 20:19:03.592344 63467 pailgun_server.py:76] handling pailgun request: `./pants list 3rdparty::`
D0425 20:19:03.593507 63467 pailgun_service.py:51] synchronously requesting BuildGraph from SchedulerService.
D0425 20:19:03.593738 63467 scheduler_service.py:92] execution commandline: [u'./pants', 'list', '3rdparty::']
D0425 20:19:03.638525 63467 scheduler_service.py:94] parsed spec_roots: [DescendantAddresses(directory='3rdparty')]
D0425 20:19:03.638927 63467 scheduler_service.py:97] cache stats: hits=0, misses=0, total=0
D0425 20:19:03.639098 63467 scheduler_service.py:98] build_graph is: <pants.engine.exp.legacy.graph.ExpGraph object at 0x410ed59e10>
D0425 20:19:03.639349 63467 pailgun_server.py:60] running pailgun request with runner: <pants.bin.daemon_pants_runner.DaemonPantsRunner object at 0x10c4fe190>
D0425 20:19:03.640522 63467 process_manager.py:402] forking <pants.bin.daemon_pants_runner.DaemonPantsRunner object at 0x10c4fe190>

rough-minister-58256

04/26/2016, 3:22 AM

not as big of a speedup as expected - cache stats seem to stay fixed at 0, need to look into it some more

rough-minister-58256

04/26/2016, 3:27 AM

slow but functional!

full-kilobyte-93429

04/26/2016, 6:26 AM

wonder if it is because

EngineTest

extends

unittest.TestCase

as opposed to

BaseTest

or other base test class that it does not get

pants.ini

options.

rough-minister-58256

04/26/2016, 6:28 AM

pants is the test runner tho, so its own options should govern its behavior

full-kilobyte-93429

04/26/2016, 6:33 AM

right, how did your test go? did you see the

$PATH

rough-minister-58256

04/26/2016, 6:34 AM

Copy code

/home/travis/virtualenv/python2.7.11/bin:/home/travis/bin:/home/travis/.local/bin:/home/travis/.gimme/versions/go1.4.1.linux.amd64/bin:/opt/python/2.7.9/bin:/opt/python/2.6.9/bin:/opt/python/3.4.2/bin:/opt/python/3.3.5/bin:/opt/python/3.2.5/bin:/opt/python/pypy-2.5.0/bin:/opt/python/pypy3-2.4.0/bin:/usr/local/phantomjs/bin:/home/travis/.nvm/v0.10.36/bin:./node_modules/.bin:/usr/local/maven-3.2.5/bin:/usr/local/clang-3.4/bin:/home/travis/.gimme/versions/go1.4.1.linux.amd64/bin:/opt/python/2.7.9/bin:/opt/python/2.6.9/bin:/opt/python/3.4.2/bin:/opt/python/3.3.5/bin:/opt/python/3.2.5/bin:/opt/python/pypy-2.5.0/bin:/opt/python/pypy3-2.4.0/bin:/usr/local/phantomjs/bin:./node_modules/.bin:/usr/local/maven-3.2.5/bin:/usr/local/clang-3.4/bin:/home/travis/.gimme/versions/go1.4.1.linux.amd64/bin:/home/travis/.rvm/gems/ruby-1.9.3-p551/bin:/home/travis/.rvm/gems/ruby-1.9.3-p551@global/bin:/home/travis/.rvm/rubies/ruby-1.9.3-p551/bin:/opt/python/2.7.9/bin:/opt/python/2.6.9/bin:/opt/python/3.4.2/bin:/opt/python/3.3.5/bin:/opt/python/3.2.5/bin:/opt/python/pypy-2.5.0/bin:/opt/python/pypy3-2.4.0/bin:/usr/local/phantomjs/bin:./node_modules/.bin:/usr/local/maven-3.2.5/bin:/usr/local/clang-3.4/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/travis/.rvm/bin:/home/travis/.rvm/bin:/home/travis/.rvm/bin

rough-minister-58256

04/26/2016, 6:34 AM

quite the path