pantsbuild #development

Channels

welcome

github-notifications

development

witty-crayon-22786

03/08/2016, 10:07 PM

full-kilobyte-93429

03/08/2016, 10:08 PM

hmm, between engine and scheduler, the cache key would need to be computed from the inputs, not available in engine

witty-crayon-22786

03/08/2016, 10:08 PM

yes they are

full-kilobyte-93429

03/08/2016, 10:08 PM

the step batch?

witty-crayon-22786

03/08/2016, 10:08 PM

the Step object, yes.

witty-crayon-22786

03/08/2016, 10:09 PM

each input should just be a key

full-kilobyte-93429

03/08/2016, 10:09 PM

ok, let me look into that

witty-crayon-22786

03/08/2016, 10:10 PM

i'm not sure about whether the Return/Throw should be serialized in the value, or should wrap the value.

witty-crayon-22786

03/08/2016, 10:10 PM

the scheduler does need that info, so

witty-crayon-22786

03/08/2016, 10:10 PM

but it doesn't need the content

witty-crayon-22786

03/09/2016, 12:53 AM

@kind-holiday-68516: https://rbcommons.com/s/twitter/r/3540/ look alright now?

rough-minister-58256

03/09/2016, 1:39 AM

anyone care if I go ahead and break the

create_*_tasks()

functions out into a separate register.py? hit a circular import issue with the fs stuff and that was the cure - figured I’d go ahead and do it for all of them

witty-crayon-22786

03/09/2016, 1:39 AM

by all means

witty-crayon-22786

03/09/2016, 1:40 AM

hm... although what was circular? just for my reference

rough-minister-58256

03/09/2016, 1:49 AM

also I seem to have broken test_dependency_inference somehow..

rough-minister-58256

03/09/2016, 1:49 AM

Copy code

==================== FAILURES ====================
                     ____ SchedulerTest.test_dependency_inference _____
                     
                     self = <pants_test.engine.exp.test_scheduler.SchedulerTest testMethod=test_dependency_inference>
                     
                         def test_dependency_inference(self):
                           """Scala dependency inference introduces dependencies that do not exist in BUILD files."""
                           build_request = self.request(['compile'], self.inferred_deps)
                           walk = self.build_and_walk(build_request)
                         
                           # Validate the root.
                           self.assert_root(walk,
                                            SelectNode(self.key(self.inferred_deps), Classpath, None, None),
                     >                      Classpath(creator='scalac'))
                     
                     tests/python/pants_test/engine/exp/test_scheduler.py:207: 
                     _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
                     tests/python/pants_test/engine/exp/test_scheduler.py:82: in assert_root
                         self.assertEquals(Return([return_value]), root_state)
                     E   AssertionError: Return(value=[Classpath(creator=u'scalac')]) != Throw(exc=ValueError(u"No source of explicit dependency SelectNode(subject_key=Key(cc5a4693405f47d7c36bf4111c2f8e08addf2452:[src/scala/inferred_deps:inferred_deps]), product=<class 'pants.engine.exp.examples.planners.Classpath'>, variants=None, variant_key=None)",))
                     -------------- Captured stdout call --------------
                     ivy_resolve executed for (Jar(address=3rdparty/jvm:scala-library),), returned: Classpath(creator=u'ivy_resolve')
                     scalac executed for (ScalaSources(dependencies=[3rdparty/jvm:scala-library], files=[u'Scrooge.scala'], spec_path=u'src/scala/scrooge', type_alias=u'scala'), [Classpath(creator=u'ivy_resolve')]), returned: Classpath(creator=u'scalac')
                     -------------- Captured stderr call --------------
                     created 379 total nodes in 520 scheduling iterations and 856 steps, with 16 nodes in the executed path.
                      1 failed, 104 passed, 1 xfailed in 81.21 seconds

rough-minister-58256

03/09/2016, 1:49 AM

any ideas on that one?

witty-crayon-22786

03/09/2016, 1:51 AM

@rough-minister-58256: should render it probably

rough-minister-58256

03/09/2016, 2:06 AM

Copy code

if self.product is Paths and type(subject) is PathGlobs:
        return Return(file_exists(step_context.project_tree, subject))
      elif self.product is FilesContent and type(subject) is Paths:
        return Return(files_content(step_context.project_tree, subject))
      elif self.product is DirectoryListing and type(subject) is Path:
        return Return(list_directory(step_context.project_tree, subject))
      else:
        return Noop('couldnt resolve product {} for subject {} in a FilesystemNode'
                    .format(self.product, subject))

witty-crayon-22786

03/09/2016, 2:19 AM

@rough-minister-58256: https://github.com/pantsbuild/pants/issues/3024

rough-minister-58256

03/09/2016, 2:19 AM

thx!

witty-crayon-22786

03/09/2016, 2:24 AM

@rough-minister-58256: just merged https://rbcommons.com/s/twitter/r/3540/

rough-minister-58256

03/09/2016, 2:24 AM

sweet

rough-minister-58256

03/09/2016, 2:24 AM

ah and you already landed 3548, perfect

witty-crayon-22786

03/09/2016, 2:25 AM

ah, yea.

full-kilobyte-93429

03/09/2016, 5:16 PM

@witty-crayon-22786: for moving storage to engine, one idea I am exploring is to wrap

subjects

State

with a wrapper class, say

Storage.Entry

with

key

and

content

fields, engine is going to handle the translation in both ways, so scheduler only sees

key

and nodes see

content

witty-crayon-22786

03/09/2016, 5:38 PM

Could work... I think usually I'd wait until it looks like a lot of functions are receiving both the key and value before doing that though?

witty-crayon-22786

03/09/2016, 5:39 PM

One thing I was thinking of earlier too is that it's possible that StorageKey should hold the type of its value, so that you can do type checks without desearializing

full-kilobyte-93429

03/09/2016, 5:40 PM

i see, that would work for the

State

case

witty-crayon-22786

03/09/2016, 5:42 PM

Ah, yea.