Hi I m using `1 27 0 dev3` it seems that pants will always b Pants #general

Hi, I’m using `1.27.0.dev3`, it seems that pants w...

bulky-evening-62934

04/16/2020, 9:19 AM

Hi, I’m using

1.27.0.dev3

, it seems that pants will always build the binary for current platform even I set

platforms = ['linux-x86_64-cp-37-cp37m']

BUILD

, is it expected behavior ? And is it possible to pass platform as an option when running

pants binary

instead of using the value in BUILD file ?

hundreds-father-404

04/16/2020, 2:25 PM

And is it possible to pass platform as an option

Yes, it should be possible with

--python-setup-platforms

. When you define

platforms

in a BUILD file, it will override this global default. But, I realized we haven’t yet wired this to V2. Our bad. I’ll fix this. --

is it expected behavior ?

No, that is not expected behavior. In 1.27.0.dev3, if you set

platforms

in the

python_binary

, we should be passing that down to Pex (and thus Pip). Could you please paste your BUILD file?

bulky-evening-62934

04/16/2020, 2:45 PM

Copy code

python_binary(
  name = 'tf_train',
  sources = ['*.py', '!*_test.py'],
  dependencies = [
    '//:tensorflow',
  ],
  platforms = [
    'linux-x86_64-cp-37-cp37m'
  ],
  entry_point = "tf_train.main",
)

I build the binary on macOS, but it will still put all macOS wheel into .pex file

bulky-evening-62934

04/16/2020, 2:46 PM

Copy code

$unzip -l dist/tf_train.pex | grep wrapt
        4  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-macosx_10_15_x86_64.whl/wrapt-1.12.1.dist-info/INSTALLER
     1304  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-macosx_10_15_x86_64.whl/wrapt-1.12.1.dist-info/LICENSE
     6489  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-macosx_10_15_x86_64.whl/wrapt-1.12.1.dist-info/METADATA
      872  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-macosx_10_15_x86_64.whl/wrapt-1.12.1.dist-info/RECORD
      111  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-macosx_10_15_x86_64.whl/wrapt-1.12.1.dist-info/WHEEL
        6  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-macosx_10_15_x86_64.whl/wrapt-1.12.1.dist-info/top_level.txt
      658  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-macosx_10_15_x86_64.whl/wrapt/__init__.py
    57456  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-macosx_10_15_x86_64.whl/wrapt/_wrappers.cpython-37m-darwin.so
    20247  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-macosx_10_15_x86_64.whl/wrapt/decorators.py
     7896  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-macosx_10_15_x86_64.whl/wrapt/importer.py
    33768  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-macosx_10_15_x86_64.whl/wrapt/wrappers.py
        4  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-manylinux2010_x86_64.whl/wrapt-1.12.1.dist-info/INSTALLER
     1304  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-manylinux2010_x86_64.whl/wrapt-1.12.1.dist-info/LICENSE
     6489  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-manylinux2010_x86_64.whl/wrapt-1.12.1.dist-info/METADATA
      883  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-manylinux2010_x86_64.whl/wrapt-1.12.1.dist-info/RECORD
      104  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-manylinux2010_x86_64.whl/wrapt-1.12.1.dist-info/WHEEL
        6  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-manylinux2010_x86_64.whl/wrapt-1.12.1.dist-info/top_level.txt
      658  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-manylinux2010_x86_64.whl/wrapt/__init__.py
   179760  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-manylinux2010_x86_64.whl/wrapt/_wrappers.cpython-37m-x86_64-linux-gnu.so
    20247  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-manylinux2010_x86_64.whl/wrapt/decorators.py
     7896  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-manylinux2010_x86_64.whl/wrapt/importer.py
    33768  01-01-1980 00:00   .deps/wrapt-1.12.1-cp37-cp37m-manylinux2010_x86_64.whl/wrapt/wrappers.py

hundreds-father-404

04/16/2020, 2:46 PM

Hm. That all looks right, minus the

sources

field as discussed in the other thread, which is irrelevant here.

hundreds-father-404

04/16/2020, 2:46 PM

Oh, interesting, so it is at least including linux! The goal is to not also include macOS, right?

👍 1

hundreds-father-404

04/16/2020, 2:47 PM

To confirm, if you remove the

platforms

field, will the output only include macOS and not Linux?

bulky-evening-62934

04/16/2020, 2:50 PM

if I remove

platform

field, it build .pex only for macOS

👍 1

hundreds-father-404

04/16/2020, 2:52 PM

Cool, that helps a lot to know that the

platforms

field is indeed being respected, only not quite how you or I would expect. I’m investigating now.

bulky-evening-62934

04/16/2020, 2:53 PM

$./pants binary --python-setup-platforms "['linux-x86_64-cp-37-cp37m']" tf_train

Run with the option will get an error

ERROR: Missing value for command line flag --platforms in scope 'python-setup'

hundreds-father-404

04/16/2020, 2:53 PM

./pants binary --python-setup-platforms="['linux-x86_64-cp-37-cp37m']" tf_train

. I don’t think there’s a great reason that we don’t allow the lack of

, probably only oversight.

hundreds-father-404

04/16/2020, 2:55 PM

But, reminder that this is ignored currently with the V2 implementation of binary, so I don’t expect it to do anything.

👍 1

bulky-evening-62934

04/16/2020, 2:56 PM

oh, there is a

, but this option doesn’t work, it will build binary only for current platform

hundreds-father-404

04/16/2020, 3:03 PM

Huh, so I ran

pex --platform=linux-x86_64-cp-37-cp37m cryptography -o test.pex && unzip -l test.pex

and it did the right thing, i.e. only included Linux wheels. I’ll try to reproduce your issue with Pants and figure out why it wouldn’t be working as intended.

👍 1

hundreds-father-404

04/16/2020, 5:58 PM

Okay, I reproduced this issue with Pants. Trying to figure out why it’s happening now.

hundreds-father-404

04/16/2020, 6:08 PM

Interesting. This has something to do with Pex and the

--interpreter-constraint

option:

Copy code

▶ pex --platform=linux-x86_64-cp-37-cp37m --interpreter-constraint 'CPython>=3.6' --output-file test.pex cryptography && unzip -l test.pex | grep 'macosx' | wc -l
     151

▶ pex --platform=linux-x86_64-cp-37-cp37m --output-file test.pex cryptography && unzip -l test.pex | grep 'macosx' | wc -l
       0

Next step is to see if this is intentional or a bug (more likely)

👍 1

hundreds-father-404

04/16/2020, 6:51 PM

See https://github.com/pantsbuild/pex/issues/957. This is indeed intentional. We’re thinking about how Pants should behave when you specify `--platform`: 1) No longer pass

--interpreter-constraint

to Pex, but have Pants validate that the interpreter constraints we would have passed are compatible with the platform(s). 2) Current behavior of always including the

current

platform no matter what. I think #1 is more correct for Pants to do.

bulky-evening-62934

04/16/2020, 7:09 PM

nice finding of the root cause, so the solution is removing this option when current is not in the platforms list ?

hundreds-father-404

04/16/2020, 7:11 PM

Yes, close to it. For now, I’m going with “simply don’t pass

--interpreter-constraint

to Pex when

--platform

is also set”. Note that you are already embedding which interpreter you want to use in the platform string with

cp-37-cp37m

. You can’t say just

linux

anymore; you must specify which interpreter you want. So, you’re overriding whatever Pants would have normally used for the interpreter constraints.

👍 1

hundreds-father-404

04/16/2020, 8:36 PM

https://github.com/pantsbuild/pants/pull/9563

👍 1

Open in Slack

Previous Next