https://pantsbuild.org/ logo
Powered by
b

bulky-evening-62934

04/08/2020, 5:23 PM
Hi all, another question, I want to build an python_binary which has tensorflow==2.1.0 as dependency, and my BUILD file is like,
Copy code
python_binary(
  name='demo',
  dependencies=[
    '3rdparty/python:tensorflow',
  ],
  platforms=[
    'current',
  ],
  source='main.py'
)
My 
current
platforms is macos, but I want to build a pex file also works on linux (ubuntu), I tried several possible values for platform but None of them work Any idea about setting a proper platform value here ?
h

hundreds-father-404

04/08/2020, 5:31 PM
I believe you will want to use something like 
linux_x86_64-cp-36-36m
. If you upgrade to Pants 1.27.0.dev2, run 
./pants target-types2 --details='python_binary'
for more information on all the fields that you can use with a 
python_binary
, including the 
platforms
field
b

bulky-evening-62934

04/08/2020, 7:15 PM
I tried 
linux_x86_64-cp-37-37m
, but it doesn’t work, from tensorflow pip page, https://pypi.org/project/tensorflow/#files, they are using platform name 
manylinux-2010
, does it make any difference ?
h

hundreds-father-404

04/08/2020, 7:16 PM
Ah, does 
manylinux2010_x86_64-cp-37-37m
work?
Also, if you are using the V2 engine (you added 
pants.backend.python
to 
backend_packages2
), be sure you are using 1.27.0.dev2. Until then, the V2 implementation was ignoring the value 
platforms
in a 
python_binary
target, but that’s now fixed.
b

bulky-evening-62934

04/08/2020, 7:19 PM
Yeah, I followed the 
pants.toml
file in that new python repo example
💯 1
manylinux2010_x86_64-cp-37-37m
doesn’t work
h

happy-kitchen-89482

04/08/2020, 7:21 PM
what python version are you on?
b

bulky-evening-62934

04/08/2020, 7:22 PM
3.7.5
h

happy-kitchen-89482

04/08/2020, 7:22 PM
Maybe those 
37
need to be 
36
or 
38
, or just omitted?
Not sure if they can be omitted
maybe there's some other way to say "any python version" or whatever
I would need to do a bit of reading
Ah OK, so 
37
is right
Note that you can only build a linux-compatible pex on MacOS if all the transitive requirements are available as prebuilt wheels. But that's probably the case here.
👍 1
h

hundreds-father-404

04/08/2020, 7:24 PM
Yeah, I think Pex (the underlying tool that creates a single binary file) used to allow the more flexible string without the version, but that was wrong behavior because the ABI changes between different Python versions, unless the wheel author is constraining to a particular ABI by flagging their wheel as 
abi3
. So, now you need to be more strict
h

happy-kitchen-89482

04/08/2020, 7:24 PM
@enough-analyst-54434 if you happen to be around, this is in your domain of expertise ...
b

bulky-evening-62934

04/08/2020, 7:25 PM
I’m quite confused by platform for python wheel, for linux, some packages use 
linux
, some use 
manylinux1
, some use 
manylinux2010
or 
manylinux2014
, are they actually same if I just specify linux as platform ?
h

hundreds-father-404

04/08/2020, 7:29 PM
Not quite - Python has had a couple changes the past few years to handle Linux wheels. It’s not safe for a distribution to say 
linux
because Linux distributions vary so much, e.g. the Linux distro on your Android phone might be very different than the server for Pants docs vs. an embedded system. So, Python devs came up with the idea of “manylinux1” in 2016 as a way to say “my wheel only depends on this constrained set of the kernel”. See https://www.python.org/dev/peps/pep-0513/ for the original PEP that introduced this. They’ve gone through a couple iterations. https://www.python.org/dev/peps/pep-0571/ added 
manylinux2010
to say that you depend on more modern symbols than 
manylinux1
. Then, Python devs realized how hard it is to maintain so many different 
manylinux
tags, so they generalized the scheme in https://www.python.org/dev/peps/pep-0600/
👍 1
Oh! I think I got it working. Try this: 
manylinux2010_x86_64-cp-37-m
this is the abbreviated version of a platform tag
b

bulky-evening-62934

04/08/2020, 7:35 PM
manylinux2010_x86_64-cp-37-m
not work for me, what’s your build file ?
Copy code
ERROR: Could not find a version that satisfies the requirement termcolor>=1.1.0 (from tensorflow==2.1.0) (from versions: none)
ERROR: No matching distribution found for termcolor>=1.1.0 (from tensorflow==2.1.0)
from pip files page https://pypi.org/project/termcolor/#files, it has only 
tar.gz
@hundreds-father-404 what if I have 2 dependencies, one has wheel prebuilt by manylinux2010 and another one by manylinux1, which platform I need to specify ?
h

hundreds-father-404

04/08/2020, 7:41 PM
Ah, bummer, termcolor is an issue 😕 It’s impossible for a mac machine to convert a 
.tar.gz
(sdist / source distribution) into a Linux-compatible wheel Is it possible to build TensorFlow without that? I’m surprised that they’re depending on a library last updated in 2011.
b

bulky-evening-62934

04/08/2020, 7:42 PM
Also I might have some dependencies have no wheel prebuilt, is possible (and how) to fetch 3rdparty dependencies (wheel) from my private PyPI server or cloud server or local ?
h

hundreds-father-404

04/08/2020, 7:42 PM
what if I have 2 dependencies, one has wheel prebuilt by manylinux2010 and another one by manylinux1, which platform I need to specify ?
I believe if you specify 
manylinux2010
, then Pex/Pip will still build because 
manylinux1
is future compatible with all new 
manylinux
tags. Conversely, if you said 
manylinux1
, it would fail to build because 
manylinux10
is not backwards compatible with 
manylinux1
👍 1
is possible (and how) to fetch 3rdparty dependencies (wheel) from my private PyPI server or cloud server or local ?
Yes, for this, you’d want to set 
repos
and/or 
indexes
in the 
python-repos
options scope, like this in `pants.toml`:
Copy code
[python-repos]
repos.add = []
indexes.add = []
I’m honestly not sure what goes in those, though. @happy-kitchen-89482 do you know the difference? FYI you can run 
./pants help python-repos
for some more information, although we need to improve the documentation here.
b

bulky-evening-62934

04/08/2020, 7:48 PM
so I could build termcolor into a linux-compatible wheel, then store it somewhere remotely and then configure pants to use them ?
h

hundreds-father-404

04/08/2020, 7:50 PM
Yes, I believe that would work! Under the hood, Pex (which runs Pip to resolve dependencies) would find that there’s no compatible 
termcolor
on PyPI but that there is one in your custom repository
Oh, I just realized for now you’ll need to define these values in the 
python_binary
target through the fields 
repositories
and 
indices
(run 
./pants target-types --details=python_binary
) We haven’t yet hooked up the global options 
--python-repo
to the V2 implementation, which is an oversight we’ll fix.
👍 1
h

happy-kitchen-89482

04/08/2020, 8:24 PM
Yes, this would work
You can use your own pypi-like repos, as Eric described. E.g., gemfury. @hundreds-father-404 Are you sure 
python_repos
doesn't work in v2? We will fix that ASAP if so. I had thought I was able to use it with gemfury...
h

hundreds-father-404

04/08/2020, 10:56 PM
Fix at https://github.com/pantsbuild/pants/pull/9495. This will go out in the dev release on Friday :)
e

enough-analyst-54434

04/10/2020, 4:45 PM
Just read all this. It looks like you all figured out the required elements. I explained the non-abbreviated platform bug detour here: https://github.com/pantsbuild/pex/issues/953
👍 1
h

hundreds-father-404

04/10/2020, 5:34 PM
@bulky-evening-62934 I gave you bad advice to use 
manylinux2010
. You’ll want to use 
linux
instead, per the explanation in https://github.com/pantsbuild/pex/issues/953.
👍 1
Powered by