https://pantsbuild.org/ logo
Powered by
#general
  • r

    ripe-memory-59241

    10/18/2017, 11:45 PM
    The overall use case is that I'm trying to get a pex with numpy/scipy/sklearn together. I've been able to build wheels for these outside the context of pex.
  • r

    rough-minister-58256

    10/18/2017, 11:45 PM
    ideally you could just declare the dep for the wheel in the pex build and let pex package it for you
  • r

    rough-minister-58256

    10/18/2017, 11:45 PM
    so why not just declare all 3 of those as deps for the pex?
  • r

    ripe-memory-59241

    10/18/2017, 11:46 PM
    pex keeps failing when trying to build the scipy wheel
  • r

    ripe-memory-59241

    10/18/2017, 11:46 PM
    or are you saying there's a way to declare a wheel as a dep (rather than just a package with its version)?
  • r

    rough-minister-58256

    10/18/2017, 11:47 PM
    Copy code
    [omerta ~]$ pex --disable-cache scipy scikit-learn numpy -o test.pex
[omerta ~]$ ./test.pex
Python 2.7.10 (default, Dec 16 2015, 14:09:45) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import numpy, sklearn, scipy
>>>
  • r

    rough-minister-58256

    10/18/2017, 11:48 PM
    you can’t directly declare wheels, but you can control where pex resolves them from
  • r

    rough-minister-58256

    10/18/2017, 11:48 PM
    via 
    --python-repos-repos
    for pants, 
    -f
    for pex
  • r

    rough-minister-58256

    10/18/2017, 11:49 PM
    so if you had an intermediate dir of known good wheels, you could 
    ./pants --python-repos-repos="['file:///tmp/wheels']" ...
    to make pants resolve against that dir
  • r

    rough-minister-58256

    10/18/2017, 11:49 PM
    thats how we resolve everything internally at build time
    👍 1
  • r

    rough-minister-58256

    10/18/2017, 11:50 PM
    whenever you’re running setup.py from a bdist in the midst of a build you’re sure to experience pain at any reasonable scale
  • r

    rough-minister-58256

    10/18/2017, 11:52 PM
    packages that rely on distutils are particularly painful
  • r

    ripe-memory-59241

    10/18/2017, 11:52 PM
    I've certainly been feeling some pain 🙂
  • r

    ripe-memory-59241

    10/19/2017, 12:01 AM
    Hmm... with 
    PEX_VERBOSE=5 pex --disable-cache -f /home/ec2-user/custom_numpy_wheel numpy scipy
    it seems to take forever with the last message 
    pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x26078d0>
    . Is that expected?
  • r

    ripe-memory-59241

    10/19/2017, 12:02 AM
    *the scipy wheel is also in custom_numpy_wheel, was just too lazy to rename it
  • r

    rough-minister-58256

    10/19/2017, 12:06 AM
    its probably just resolving transitive deps
  • r

    rough-minister-58256

    10/19/2017, 12:07 AM
    generally the pattern is: 
    pip wheel
    into a dir to get all transitive deps as wheels, then 
    pex -f /those/wheels --no-build --no-pypi
    to do a pure wheel resolve against them
  • r

    ripe-memory-59241

    10/19/2017, 12:12 AM
    Weird, I don't think numpy and scipy have any external (python) dependencies, but 
    pex --disable-cache -f /home/ec2-user/custom_numpy_wheel --no-build --no-pypi numpy scipy
    worked almost immediately, and I successfully imported scipy and numpy in the interpreter shell pex opened. Thanks a ton! Now to just add the sklearn wheel. Should be straightforward at this point.
  • r

    rough-minister-58256

    10/19/2017, 12:13 AM
    sweet
  • a

    alert-monitor-62116

    10/19/2017, 12:30 AM
    Hi, any idea why I out of the sudden getting this error:
    Copy code
    Exception message: Package SourcePackage(u'<https://pypi.python.org/packages/03/53/65d932f96d5aab57c21679c30e6ec33c37f8530828263d0a1d41f9faddc9/setuptools-30.0.0.zip#md5=4f7349b1d33b7c32e6f6d46f21c9b651>') is not translateable by ChainedTranslator(EggTranslator, SourceTranslator)
  • r

    rough-minister-58256

    10/19/2017, 12:36 AM
    that would be a failure on your local machine to run the equivalent of 
    python setup.py bdist_egg
    in the unzipped setuptools-30.0.0 dir
  • r

    rough-minister-58256

    10/19/2017, 12:36 AM
    there should be more output surrounding that line that would hint at why its failing.
  • a

    alert-monitor-62116

    10/19/2017, 12:43 AM
  • a

    alert-monitor-62116

    10/19/2017, 12:46 AM
    I removed ~/.cache/pants and still the same error
  • a

    alert-monitor-62116

    10/19/2017, 4:44 AM
    it's really weird everything worked great so far, and suddenly this error out of the blue
  • a

    alert-monitor-62116

    10/19/2017, 4:58 AM
    Copy code
    21:57:39 00:01     [bootstrap-jvm-tools]
21:57:39 00:01     [provide-tools-jar]
21:57:39 00:01   [imports]
21:57:39 00:01     [ivy-imports]
21:57:39 00:01   [unpack-jars]
21:57:39 00:01     [unpack-jars]
21:57:39 00:01   [deferred-sources]
21:57:39 00:01     [deferred-sources]
21:57:39 00:01   [jvm-platform-validate]
21:57:39 00:01     [jvm-platform-validate]
21:57:39 00:01   [gen]
21:57:39 00:01     [antlr-java]
21:57:39 00:01     [antlr-py]
21:57:40 00:02     [jaxb]
21:57:40 00:02     [protoc]
21:57:40 00:02     [ragel]
21:57:40 00:02     [thrift-java]
21:57:40 00:02     [thrift-py]
21:57:40 00:02     [wire]
21:57:40 00:02     [avro]
21:57:40 00:02     [avro-java]
21:57:40 00:02   [resolve]
21:57:40 00:02     [ivy]DEBUG] Using previous resolve.
DEBUG] Using previous resolve.
DEBUG] Parsing ivy report /Users/jastl001/git/github.disney.com/data-services/.pants.d/ivy/1abca6e9918764d2c80b3bc51fac752ce87b02ab/resolve-report-default.xml

21:57:40 00:02   [compile]
21:57:40 00:02     [compile-jvm-prep-command]
21:57:40 00:02       [jvm_prep_command]
21:57:40 00:02     [compile-prep-command]
21:57:40 00:02     [compile]
21:57:40 00:02     [zinc]
21:57:41 00:03     [jvm-dep-check]
21:57:41 00:03   [pyprep]
21:57:41 00:03     [interpreter]
and its fails here
  • a

    alert-monitor-62116

    10/19/2017, 5:10 AM
    a bit closer:
    Copy code
    setuptools-30.0.0-py3.6.egg') is not compatible with CPython-2.7.10 / macosx-10.12-intel
  • a

    alert-monitor-62116

    10/19/2017, 5:24 AM
    any ideas how to fix this problem ?
  • l

    lively-france-2777

    10/19/2017, 6:24 AM
    I have at times seen similar problems locally and then deleted the corresponding source archive from 
    .pants.d/python-setup/resolved_requirements/CPython-2.7.10/
    to get it to work (in my case it has usually happened when I tried to build a binary for a different platform than my current).
  • h

    high-advantage-47550

    10/19/2017, 8:54 PM
    what's the right syntax to exclude files from a 
    scala_jar
    ? e.g. 
    META-INF/*.SF
1...183918401841...2420Latest