pantsbuild #pex

Channels

welcome

github-notifications

development

mysterious-farmer-45668

08/13/2018, 10:25 PM

without the platforms clause, the build works; but when I add it, I get:

mysterious-farmer-45668

08/13/2018, 10:25 PM

Copy code

Exception caught: (<class 'pex.resolver.Untranslateable'>)
Exception message: Package SourcePackage(u'file:///Users/seolh001/projects/de-monorepo/.pants.d/python-setup/resolved_requirements/CPython-2.7.14/thrift-0.11.0.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)

mysterious-farmer-45668

08/13/2018, 10:26 PM

but thrift isn't explicitly stated as a dependency

mysterious-farmer-45668

08/13/2018, 10:35 PM

is this a better question for #general?

mysterious-farmer-45668

08/13/2018, 10:39 PM

verbosity gave me this log:

Copy code

pex: Target package WheelPackage('file:///private/var/folders/7y/9l6fcjcn6pj__cgcnk3ggdxwzs6css/T/tmpKOdQHO/thrift-0.11.0-cp27-cp27m-macosx_10_12_x86_64.whl') is not compatible with [('cp27', 'cp27u', 'linux_x86_64'), ('cp27', 'cp27m', 'manylinux1_x86_64'), ('py2', 'none', 'any'), ('cp27', 'cp27mu', 'manylinux1_x86_64'), ('cp27', 'none', 'any'), ('cp27', 'cp27dm', 'manylinux1_x86_64'), ('cp27', 'cp27u', 'manylinux1_x86_64'), ('cp27', 'cp27d', 'manylinux1_x86_64'), ('cp27', 'cp27mu', 'linux_x86_64'), ('cp27', 'none', 'linux_x86_64'), ('py2', 'none', 'linux_x86_64'), ('cp27', 'cp27dm', 'linux_x86_64'), ('cp27', 'cp27d', 'linux_x86_64'), ('cp27', 'cp27m', 'linux_x86_64'), ('py27', 'none', 'any'), ('cp27', 'cp27dmu', 'linux_x86_64'), ('cp27', 'none', 'manylinux1_x86_64'), ('cp2', 'none', 'any'), ('py2', 'none', 'manylinux1_x86_64'), ('cp27', 'cp27dmu', 'manylinux1_x86_64')]
pex:   Packaging thrift: 3206.8ms

mysterious-farmer-45668

08/13/2018, 10:51 PM

is this not a general use case? building on osx and running on linux?

enough-analyst-54434

08/13/2018, 10:54 PM

@mysterious-farmer-45668 it is a common use case, but pants does not perform magic. It can't build osx-specific wheels on linux for you or vice-versa. If you want to build multiplatform pexes, you will need pre-built wheels for the other platforms.

mysterious-farmer-45668

08/13/2018, 10:54 PM

how can i determine what is causing thrift to be included in my build map?

mysterious-farmer-45668

08/13/2018, 11:02 PM

I get this error even when I remove the 'current' part in platforms,

mysterious-farmer-45668

08/13/2018, 11:03 PM

Why would

thrfit-0.11.0-cp27-cp27m-macosx_10_12_x86_64.whl

be attempted to packaged when I specified that the platform should be

linux-x86_64

mysterious-farmer-45668

08/13/2018, 11:03 PM

also doesn't help that I can't locate which dependency is bringing in thrift in the first place

mysterious-farmer-45668

08/13/2018, 11:14 PM

ok, i progressed a little because I found the platform the wheel exists for

mysterious-farmer-45668

08/13/2018, 11:15 PM

but now there is another library that requires another platform, and I added that to the platforms section but it fails; is there a way i can get one wheel built for

cp27-cp27mu-linux_x86_64

and another for

cp27-cp27mu-manylinux1_x86_64

? my guess would be no

wide-energy-11069

08/13/2018, 11:21 PM

the manylinux wheel wasn’t supported until a recent version of pants

wide-energy-11069

08/13/2018, 11:22 PM

1.8.0 https://www.pantsbuild.org/notes-1.8.x.html

dazzling-florist-39383

08/15/2018, 3:30 PM

random q: anyone doing code-signing on .pex binaries

aloof-angle-91616

08/18/2018, 2:34 PM

I'm seeing this upon running

./pants binary src/python/pants/util:s3_log_aggregator_bin

-- is the correct way to solve this to not download a

.tar.gz

for the

s3-log-parse

package (which pex doesn't appear to know how to deal with), or define a new

EggTgzArchiveTranslator

which can understand this?

stdout

aloof-angle-91616

08/18/2018, 2:36 PM

nothing changes when i add

platforms = ['current']

to the

python_binary()

target at

src/python/pants/util:s3_log_aggregator_bin

aloof-angle-91616

08/18/2018, 2:37 PM

to be clear, i'm thinking this is probably a small pex pr, just want to know whether we're downloading the wrong thing or something is a tar gz that wasn't before

aloof-angle-91616

08/18/2018, 2:37 PM

this is on linux it reproes on osx i think too

aloof-angle-91616

08/18/2018, 2:38 PM

and to be clear:

Copy code

> tar --list -f ~/.cache/pants/python_cache/requirements/CPython-2.7.15/s3-log-parse-0.1.1.tar.gz 
s3-log-parse-0.1.1/
s3-log-parse-0.1.1/s3_log_parse.egg-info/
s3-log-parse-0.1.1/s3_log_parse.egg-info/dependency_links.txt
s3-log-parse-0.1.1/s3_log_parse.egg-info/requires.txt
s3-log-parse-0.1.1/s3_log_parse.egg-info/top_level.txt
s3-log-parse-0.1.1/s3_log_parse.egg-info/entry_points.txt
s3-log-parse-0.1.1/s3_log_parse.egg-info/SOURCES.txt
s3-log-parse-0.1.1/s3_log_parse.egg-info/PKG-INFO
s3-log-parse-0.1.1/setup.py
s3-log-parse-0.1.1/setup.cfg
s3-log-parse-0.1.1/PKG-INFO
s3-log-parse-0.1.1/s3logparse/
s3-log-parse-0.1.1/s3logparse/__init__.py
s3-log-parse-0.1.1/s3logparse/cli.py
s3-log-parse-0.1.1/s3logparse/s3logparse.py
s3-log-parse-0.1.1/README.rst

aloof-angle-91616

08/18/2018, 2:48 PM

oh wow, the version conflict in the output i posted just above this one is potentially related

aloof-angle-91616

08/18/2018, 2:56 PM

do we want

./pants run <target...>

to have access to the pants environment? this excerpt from above:

Copy code

pkg_resources.VersionConflict: (pycodestyle 2.4.0 (/home/cosmicexplorer/tools/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages), Requirement.parse('pycodestyle<2.4.0,>=2.0.0'))

seems to imply the pants venv is interacting with the

PYTHONPATH

exposed to python binaries? because the target

src/python/pants/util:s3_log_aggregator_bin

i think shouldn't be getting anything from the pants venv? i could be way mistaken. in fact, that's probably wrong in that that python binary target depends on

3rdparty/python:future

, which is in

requirements.txt

and therefore probably taken from the pants venv intentionally (an except of the BUILD file):

Copy code

python_library(
  name = 's3_log_aggregator',
  sources = ['s3_log_aggregator.py'],
  dependencies = [
    '3rdparty/python:future',
    '3rdparty/python:s3logparse'
  ]
)

python_binary(
  name = 's3_log_aggregator_bin',
  entry_point = 'pants.util.s3_log_aggregator',
  dependencies = [
    ':s3_log_aggregator'
  ]
)

aloof-angle-91616

08/18/2018, 3:15 PM

so to recap i know now why the version conflict is happening (because we depend on something that transitively depends on a version of pycodestyle incompatible with the version we have in our venv

aloof-angle-91616

08/18/2018, 3:16 PM

and there are multiple such dependencies

aloof-angle-91616

08/18/2018, 3:20 PM

this is actually also completely changed / fixed by https://github.com/pantsbuild/pants/pull/6275 too i think

enough-analyst-54434

08/21/2018, 4:47 PM

If any pexers are keen on reviews, I've got a few changes out and will post a few more once they are in: https://github.com/pantsbuild/pex/pull/540 https://github.com/pantsbuild/pex/pull/541 https://github.com/pantsbuild/pex/pull/542

rough-minister-58256

08/21/2018, 6:59 PM

🚢 🚢 🚢

enough-analyst-54434

08/21/2018, 7:16 PM

Thanks @rough-minister-58256!

👍 1

aloof-angle-91616

08/25/2018, 2:07 AM

@enough-analyst-54434 it looks like pex #543 (https://github.com/pantsbuild/pex/commit/927433a21108eb0a932af1d6a63d256729c053b7) was merged -- the https://github.com/pantsbuild/pants/pull/6275 PR will work without it (thanks to your immense help), but i'm not super familiar with how up to date pants stays with pex release cycles. would it make sense to make another pants PR to update pex and pull in those changes before getting in #6275, or is that a deed for another day?