https://pantsbuild.org/ logo
Powered by
#general
  • m

    mysterious-action-44812

    02/09/2017, 10:04 PM
    @ancient-stone-50795 do u know ^?
  • m

    mysterious-action-44812

    02/09/2017, 10:04 PM
    also not clear whats going on in my second question, how to build many bundles at once from single target with deps
  • f

    full-kilobyte-93429

    02/09/2017, 10:13 PM
    sorry that's controlled by a parameter, by default the safe naming is used, will need this to be set
    Copy code
    `[bundle.jvm]
use_basename_prefix: True
  • m

    mysterious-action-44812

    02/09/2017, 10:15 PM
    aha great!
  • m

    mysterious-action-44812

    02/09/2017, 10:15 PM
    any ideas on the second question, why my target of apps doesn’t work?
  • m

    mysterious-action-44812

    02/09/2017, 10:21 PM
    @full-kilobyte-93429 ^
  • f

    full-kilobyte-93429

    02/09/2017, 10:21 PM
    yea, checking
  • f

    full-kilobyte-93429

    02/09/2017, 10:27 PM
    looks like we actually create the bundle for dependencies but create symlink only for those 
    target_roots
    i.e, those passed from command line
  • f

    full-kilobyte-93429

    02/09/2017, 10:27 PM
  • m

    mysterious-action-44812

    02/09/2017, 10:28 PM
    there a way around that?
  • f

    full-kilobyte-93429

    02/09/2017, 10:28 PM
    not clear whether it's a feature or bug, but at least some inconsistencies
  • f

    full-kilobyte-93429

    02/09/2017, 10:29 PM
    workaround is to pass those 
    jvm_app
    or 
    jvm_binary
    from command line
  • f

    full-kilobyte-93429

    02/09/2017, 10:29 PM
    can pipe list -> filter -> bundle
  • p

    purple-oil-45048

    02/10/2017, 3:11 PM
    Are the slides of https://events.rainfocus.com/oow15/catalog/oracle.jsp?event=javaone&search=Monorepi&search.event=javaoneEvent posted anywhere? Thriving @monorepi Use Pants at Twitter, Foursquare, and Square [CON10370]
  • s

    straight-waiter-86090

    02/10/2017, 3:37 PM
    hi. I’m attempting to use the node contrib plugin but get an error after trying to configure it in pants.ini.
  • s

    straight-waiter-86090

    02/10/2017, 3:37 PM
    Copy code
    ➜  ~R9SRC git:(master) ./pants                                                                               
Exception caught: (<class 'pex.resolver.Untranslateable'>)
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/bin/pants", line 11, in <module>
    sys.exit(main())
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/pants_exe.py", line 44, in main
    PantsRunner(exiter).run()
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/pants_runner.py", line 57, in run
    options_bootstrapper=options_bootstrapper)
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/pants_runner.py", line 46, in _run
    return LocalPantsRunner(exiter, args, env, options_bootstrapper=options_bootstrapper).run()
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 53, in run
    self._maybe_profiled(self._run)
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 50, in _maybe_profiled
    runner()
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/local_pants_runner.py", line 59, in _run
    options, build_config = OptionsInitializer(options_bootstrapper, exiter=self._exiter).setup()
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/options_initializer.py", line 46, in __init__
    self._working_set = working_set or PluginResolver(self._options_bootstrapper).resolve()
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/plugin_resolver.py", line 49, in resolve
    for plugin_location in self._resolve_plugin_locations():
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/plugin_resolver.py", line 73, in _resolve_exact_plugin_locations
    for plugin in self._resolve_plugins():
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pants/bin/plugin_resolver.py", line 93, in _resolve_plugins
    cache_ttl=self._python_setup.resolver_cache_ttl)
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pex/resolver.py", line 346, in resolve
    return resolver.resolve(resolvables_from_iterable(requirements, builder))
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pex/resolver.py", line 200, in resolve
    dist = self.build(package, resolvable.options)
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pex/resolver.py", line 257, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pex/resolver.py", line 168, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))

Exception message: Package SourcePackage(u'file:///Users/dyoung/.cache/pants/plugins/pantsbuild.pants.contrib.node-1.2.1.tar.gz') is not translateable by ChainedTranslator(EggTranslator, SourceTranslator)
  • s

    straight-waiter-86090

    02/10/2017, 3:37 PM
    I added it like this:
  • s

    straight-waiter-86090

    02/10/2017, 3:37 PM
    Copy code
    [GLOBAL]
pants_version: 1.2.1
pythonpath: +[
    '%(buildroot)s/build/pants/src/python',
  ]
backend_packages: +[
    'kayak',
  ]
plugins: [
    'pantsbuild.pants.contrib.node==%(pants_version)s',
  ]
  • s

    straight-waiter-86090

    02/10/2017, 3:38 PM
    I tried removing the .tar.gz. it gets recreated again. it seems to be a valid .tgz. am I doing something wrong?
  • s

    straight-waiter-86090

    02/10/2017, 3:52 PM
    If I run 
    PEX_VERBOSE=1 ./pants
    I see some additional info:
  • s

    straight-waiter-86090

    02/10/2017, 3:52 PM
    Copy code
    pex: Failed to translate SourcePackage('file:///private/var/folders/v7/6nkv5c9d14ng3j4t3b2t5_cdcwtpbc/T/tmpAbUYiV/pantsbuild.pants.contrib.node-1.2.1.tar.gz')
pex: Traceback (most recent call last):
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pex/translator.py", line 101, in translate
    strict=(package.name not in ('distribute', 'setuptools')))
  File "/Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/1.2.1/lib/python2.7/site-packages/pex/installer.py", line 61, in __init__
    self._interpreter.binary, self.__class__.__name__))
IncapableInterpreter: Interpreter /Users/dyoung/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.deby4K/install/bin/python2.7 not capable of running EggInstaller
  • d

    dazzling-crayon-98253

    02/10/2017, 5:31 PM
    hi all, running into an issue with pex. I have a python script that is used as a reducer script in hive and the whole setup works nicely. Now, I am instead packaging the python file as a pex, and the same script now throws:
    Copy code
    Traceback (most recent call last):
  File ".bootstrap/_pex/pex.py", line 321, in execute
  File ".bootstrap/_pex/pex.py", line 78, in _activate
  File ".bootstrap/_pex/environment.py", line 135, in activate
  File ".bootstrap/_pex/environment.py", line 179, in _activate
  File ".bootstrap/_pex/environment.py", line 124, in update_candidate_distributions
  File ".bootstrap/_pex/environment.py", line 110, in load_internal_cache
  File ".bootstrap/_pex/environment.py", line 97, in write_zipped_internal_cache
  File ".bootstrap/_pex/util.py", line 176, in cache_distribution
  File ".bootstrap/_pex/common.py", line 124, in safe_open
  File ".bootstrap/_pex/common.py", line 112, in safe_mkdir
  File "/usr/lib64/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib64/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/home/.pex'
org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20003]: An error occurred when trying to close the Operator running your custom script.
  • d

    dazzling-crayon-98253

    02/10/2017, 5:32 PM
    seems similar to the bug that was hit in https://github.com/twitter/heron/issues/1103 but i don't understand the resolution
  • r

    rough-sundown-61224

    02/10/2017, 7:04 PM
    @dazzling-crayon-98253 Pex has to unpack itself to a directory; you can control which one with the PEX_ROOT env var. I do a similar thing for running mrjob:
    Copy code
    cmdenv:
  PEX_ROOT: /tmp/.pex
  • r

    rough-sundown-61224

    02/10/2017, 7:06 PM
    (or a more secure location if you’re worried about /tmp)
  • d

    dazzling-crayon-98253

    02/10/2017, 7:21 PM
    thanks todd, i'll try this out
  • f

    fancy-queen-20734

    02/10/2017, 7:29 PM
  • f

    fancy-queen-20734

    02/10/2017, 7:29 PM
    ?
  • d

    dazzling-crayon-98253

    02/10/2017, 11:32 PM
    @rough-sundown-61224 I ran the query like :
    Copy code
    USING 'PEX_ROOT=/tmp reducer.pex'
    but that didn't quite work for me,
    Copy code
    Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error 20000]: Unable to initialize custom script.
	at org.apache.hadoop.hive.ql.exec.ScriptOperator.process(ScriptOperator.java:439)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
	at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:236)
	... 7 more
Caused by: java.io.IOException: Cannot run program "PEX_ROOT=/tmp": error=2, No such file or directory
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
	at org.apache.hadoop.hive.ql.exec.ScriptOperator.process(ScriptOperator.java:397)
	... 10 more
Caused by: java.io.IOException: error=2, No such file or directory
	at java.lang.UNIXProcess.forkAndExec(Native Method)
	at java.lang.UNIXProcess.<init>(UNIXProcess.java:247)
	at java.lang.ProcessImpl.start(ProcessImpl.java:134)
	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
	... 11 more
  • w

    witty-crayon-22786

    02/10/2017, 11:32 PM
    processbuilder will have another way to set an environment variable
1...163016311632...2420Latest