pantsbuild #pex

Channels

welcome

github-notifications

development

aloof-angle-91616

12/06/2019, 4:33 AM

@enough-analyst-54434 wanted to thank you immensely for the patient and thorough discussion this morning! and also wanted to reiterate -- in the future, we'll use github issues like everyone else, we'll make issues much earlier, and we'll open with a discussion of the problem like other good pex issues, instead of trying to present a solution. this was a really good learning experience for me (sorry if that sounds patronizing)!

👖 1

❤️ 1

enough-analyst-54434

12/06/2019, 8:06 AM

Pex 2.0.3 is released: + https://pypi.org/project/pex/2.0.3/ + https://github.com/pantsbuild/pex/releases/tag/v2.0.3 Posted in #releases

🎉 3

aloof-angle-91616

12/07/2019, 6:04 PM

thinking about whether a metadata-only resolve could be possible (idea from @rough-minister-58256) to download massive wheels with great parallelism by somehow sending an http byte range header in the request to download the zip. i found that most wheels will have the METADATA file at the bottom of the zip, and was thinking that it could be possible to extract the dependencies with a very small range request (can use http HEAD to get the size of the whole zip beforehand), then start downloading those wheels in parallel recursively so we can saturate the network interface. i looked through the zip file spec and i’m trying to see if the content of a zip file is specified with purely local information so i can just request the last however many bytes of a wheel, parse and find the METADATA file content, extract it, and read out the requirements. currently just trying to follow along with emacs in hexl-mode

dry-policeman-7927

12/10/2019, 12:36 PM

Hi @enough-analyst-54434! We are debugging an issue with pants release at Twitter, which is also the reason master is red right now. We are seeing the same thing that is started to happening in upstream on

Deploy unstable pants pex

Travis shard at your commit updating pex to 2.0.3. https://travis-ci.org/pantsbuild/pants/builds/622371760 We’ve run the pip command that the pex is running. In the directory of wheels some of them are linux_x86_64 and some of them are manylinux1_x86_64. It looks like we were able to resolve both of them with the same command before. Now the platform flag limits us to one or another. Is there a quick an easy fix? Could you help us, or implement it? If not, can we please revert the pex update until we have the issue solved? Thanks! 🙂

aloof-angle-91616

12/11/2019, 7:13 PM

created https://github.com/pantsbuild/pants/pull/8793 which adds the

--generate-ipex

flag to

PexBuilderWrapper.Factory

to generate a pex file which resolves its requirements when it is first executed

rapid-crayon-8232

12/11/2019, 10:01 PM

hey guys, a little question about the latest pants version (1.24.0.dev2), with pex=2.0.3, I'm trying to install a git package which needs another two packages to be installed before building it, and for that I've instinctively done in the third-party BUILD file:

Copy code

python_requirement_library(
  name='pycocotools',
  dependencies=[
    "packages-python/third_party:Cython",
    "packages-python/third_party:numpy"
  ],
  requirements=[
    python_requirement(name='pycocotools',
                       requirement='pycocotools@git+<https://github.com/cocodataset/cocoapi.git@f67e873971bcf289c994e061c5bc359d0bbe8ff7#subdirectory=PythonAPI'>),
  ])

but pip still tries to install all modules at the same time:

Copy code

ERROR: pid: 94702 -> /usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/bin/python3.7 -s -m pip --disable-pip-version-check --isolated --exists-action i -q --cache-dir /Users/XXX/.pants.d/python-setup/resolved_requirements download --dest /var/folders/n6/cx4l_npd6s39czmkmyk2p3tc0000gn/T/tmpvp9nqvlx/resolved_dists/cp-37-cp37m --index-url <https://pypi.org/simple/> Cython==0.29.14 numpy==1.17.3 pycocotools@ git+<https://github.com/cocodataset/cocoapi.git@f67e873971bcf289c994e061c5bc359d0bbe8ff7#subdirectory=PythonAPI> raised Executing /usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/bin/python3.7 -s -m pip --disable-pip-version-check --isolated --exists-action i -q --cache-dir /Users/XXX/.pants.d/python-setup/resolved_requirements download --dest /var/folders/n6/cx4l_npd6s39czmkmyk2p3tc0000gn/T/tmpvp9nqvlx/resolved_dists/cp-37-cp37m --index-url <https://pypi.org/simple/> Cython==0.29.14 numpy==1.17.3 pycocotools@ git+<https://github.com/cocodataset/cocoapi.git@f67e873971bcf289c994e061c5bc359d0bbe8ff7#subdirectory=PythonAPI> failed with 1

The full traceback of the error:

rapid-crayon-8232

12/11/2019, 10:01 PM

Copy code

22:46:20 00:00 [main]
               (To run a reporting server: ./pants server)
22:46:21 00:01   [setup]
22:46:21 00:01     [parse]
               Executing tasks in goals: gen -> pyprep -> test
22:46:21 00:01   [gen]
22:46:21 00:01     [thrift-py]
22:46:21 00:01     [py-thrift-namespace-clash-check]
22:46:21 00:01   [pyprep]
22:46:21 00:01     [interpreter]
22:46:21 00:01     [build-local-dists]
22:46:21 00:01     [requirements]
22:46:21 00:01       [cache]
                   No cached artifacts for 22 targets.
                   Invalidated 22 targets.pex: Resolving for:
pex: Isolating pex in Chroot(/var/folders/n6/cx4l_npd6s39czmkmyk2p3tc0000gn/T/tmpxvvj_a1o {fs:}): 299.4ms
    ERROR: Command errored out with exit status 1:
     command: /usr/local/opt/python/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/n6/cx4l_npd6s39czmkmyk2p3tc0000gn/T/pip-download-wa2d7k0w/pycocotools/PythonAPI/setup.py'"'"'; __file__='"'"'/private/var/folders/n6/cx4l_npd6s39czmkmyk2p3tc0000gn/T/pip-download-wa2d7k0w/pycocotools/PythonAPI/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --no-user-cfg egg_info --egg-base /private/var/folders/n6/cx4l_npd6s39czmkmyk2p3tc0000gn/T/pip-download-wa2d7k0w/pycocotools/PythonAPI/pip-egg-info
         cwd: /private/var/folders/n6/cx4l_npd6s39czmkmyk2p3tc0000gn/T/pip-download-wa2d7k0w/pycocotools/PythonAPI
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/n6/cx4l_npd6s39czmkmyk2p3tc0000gn/T/pip-download-wa2d7k0w/pycocotools/PythonAPI/setup.py", line 2, in <module>
        import numpy as np
    ModuleNotFoundError: No module named 'numpy'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

               Waiting for background workers to finish.
22:46:43 00:23   [complete]
               FAILURE

ERROR: pid: 94702 -> /usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/bin/python3.7 -s -m pip --disable-pip-version-check --isolated --exists-action i -q --cache-dir /Users/XXX/.pants.d/python-setup/resolved_requirements download --dest /var/folders/n6/cx4l_npd6s39czmkmyk2p3tc0000gn/T/tmpvp9nqvlx/resolved_dists/cp-37-cp37m --index-url <https://pypi.org/simple/> Cython==0.29.14 numpy==1.17.3 pycocotools@ git+<https://github.com/cocodataset/cocoapi.git@f67e873971bcf289c994e061c5bc359d0bbe8ff7#subdirectory=PythonAPI> raised Executing /usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/bin/python3.7 -s -m pip --disable-pip-version-check --isolated --exists-action i -q --cache-dir /Users/XXX/.pants.d/python-setup/resolved_requirements download --dest /var/folders/n6/cx4l_npd6s39czmkmyk2p3tc0000gn/T/tmpvp9nqvlx/resolved_dists/cp-37-cp37m --index-url <https://pypi.org/simple/> Cython==0.29.14 numpy==1.17.3 pycocotools@ git+<https://github.com/cocodataset/cocoapi.git@f67e873971bcf289c994e061c5bc359d0bbe8ff7#subdirectory=PythonAPI> failed with 1

enough-analyst-54434

12/28/2019, 7:28 PM

Pex 2.1.0 release prep: https://github.com/pantsbuild/pex/pull/836 Posted in #releases

enough-analyst-54434

12/28/2019, 7:54 PM

Pex 2.1.0 is released: + https://pypi.org/project/pex/2.1.0/ + https://github.com/pantsbuild/pex/releases/tag/v2.1.0 Posted in #releases

🎉 2

rapid-crayon-8232

01/20/2020, 11:00 AM

Hello @enough-analyst-54434, sorry to bother, one of my coworkers is trying to use pants in his machine, but gets a weird permission error (no bug on my end with the same repo):

Copy code

17:16:29 00:02   [pyprep]
17:16:29 00:02     [interpreter]
17:16:29 00:02     [build-local-dists]
17:16:29 00:02     [requirements]
17:16:29 00:02       [cache]
                   No cached artifacts for 20 targets.
                   Invalidated 20 targets.pex: Resolving for:
pex: Isolating pex in Chroot(/var/folders/ty/mh344wcn4z1gdf_nm_nht80h0000gn/T/tmpodrznp4n {fs:}): 187.2ms
  ERROR: Command errored out with exit status 1:
   command: /Library/Developer/CommandLineTools/usr/bin/python3 /var/folders/ty/mh344wcn4z1gdf_nm_nht80h0000gn/T/tmpodrznp4n/pex/vendor/_vendored/pip/pip install --ignore-installed --no-user --prefix /private/var/folders/ty/mh344wcn4z1gdf_nm_nht80h0000gn/T/pip-build-env-9zw3xco9/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i <https://pypi.org/simple/> -- 'numpy>=1.17.3' 'Cython>=0.29.14' setuptools wheel
       cwd: None
  Complete output (13 lines):
  Looking in indexes: <https://pypi.org/simple/>
  Collecting numpy>=1.17.3
    Using cached <https://files.pythonhosted.org/packages/2f/5b/2cc2b9285e8b2ca8d2c1e4a2cbf1b12d70a2488ea78170de1909bca725f2/numpy-1.18.1-cp37-cp37m-macosx_10_9_x86_64.whl>
  Collecting Cython>=0.29.14
    Using cached <https://files.pythonhosted.org/packages/9b/e6/6e30303d7bc17289411d82276b7338a1781964171c94f5ebe689cbecad4c/Cython-0.29.14-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl>
  Collecting setuptools
    Using cached <https://files.pythonhosted.org/packages/af/e7/02db816dc88c598281bacebbb7ccf2c9f1a6164942e88f1a0fded8643659/setuptools-45.0.0-py2.py3-none-any.whl>
  Collecting wheel
    Using cached <https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl>
  Installing collected packages: numpy, Cython, setuptools, wheel
  ERROR: Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/Library/Python/3.7/site-packages/numpy/conftest.py'
  Consider using the `--user` option or check the permissions.

aloof-angle-91616

01/21/2020, 7:24 PM

this feels like surprising behavior, and i don't understand why this command would fail to resolve a distribution with what seems like matching requirements. is it a known issue that

>=3.6

appears to not match

==3.6.4

? i expect this is an issue with the pip resolver since we don't really touch those requirements?

Copy code

> pex --interpreter-constraint='CPython==3.6.4' --interpreter-constraint='CPython>=3.6,<4' tensorflow==1.14.0
Could not satisfy all requirements for tensorflow==1.14.0:
    tensorflow==1.14.0
> pex --interpreter-constraint='CPython==3.6.4' tensorflow==1.14.0
Python 3.6.4 (default, Jan 23 2018, 10:58:43)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>>

aloof-angle-91616

01/27/2020, 9:46 PM

if i run a pex file with

PEX_PROFILE_FILENAME=out.prof

, i can get the profile of the pex process. if that process itself depends on the

pex

library and invokes

pex.resolver.resolve()

, though, all i see from the profile is how long we waited on the pip subprocess to resolve 3rdparty dependencies. i'm trying to research this now -- is there an obvious way to get the profile of the pip subprocess?

rapid-crayon-8232

01/28/2020, 5:10 PM

👋 , it's me again 😁 I've encountered a pickling error when spawning multiple processes in pex, the context is: 1. I'm trying to create a binary of the script: https://github.com/facebookresearch/detectron2/blob/master/tools/train_net.py 2. The script spawns as many processes as gpu's in the machine, using the

launch

function 3. when running the script to spawn 2 processes for example I get the following error:

Copy code

File "/home/ubuntu/git/repo/.pants.d/pyprep/requirements/CPython-3.7.4/98abeb07f7c533df9da32c92acfed24c8711ff91/.deps/detectron2-0.1-cp37-cp37m-linux_x86_64.whl/detectron2/engine/launch.py", line 49, in launch
    daemon=False,
  File "/home/ubuntu/git/repo/.pants.d/pyprep/requirements/CPython-3.7.4/98abeb07f7c533df9da32c92acfed24c8711ff91/.deps/torch-1.4.0-cp37-cp37m-manylinux1_x86_64.whl/torch/multiprocessing/spawn.py", line 162, in spawn
    process.start()
  File "/home/ubuntu/.conda/envs/py37/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/ubuntu/.conda/envs/py37/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/ubuntu/.conda/envs/py37/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/ubuntu/.conda/envs/py37/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/ubuntu/.conda/envs/py37/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/ubuntu/.conda/envs/py37/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function main at 0x7fd2ffb1edd0>: attribute lookup main on __main__ failed

👋 1

fresh-tomato-66784

01/29/2020, 8:59 PM

Hi Everyone, have question around pex, Thanks for the help in advance Is there a way to install pex file instead of bootstrap a pex environment? Here is the context, we have a lot of application build around Apache Beam . Currently we can package our application along with their dependency in a pex file and run the beam pipeline locally. However, when we want to submit this to Google Dataflow, we cannot simple run pex file inside Dataflow as it’s a managed service. The worker entry point is control by Google. It seems there is no easy way to do bootstrap when they launch the worker . Dataflow only support install dependency from

requirement.txt

tar.gz

wheel

. Any recommendations for this use case? In an ideal case, would prefer to build a single pex that can run locally and later submit to cloud.

calm-artist-46894

02/06/2020, 6:05 PM

pex version: 2.1.0 Failed to resolve requirements:

Copy code

~$ pex google-cloud-bigquery[bqstorage,pandas]~=1.17.0
Failed to execute PEX file. Needed manylinux2014_x86_64-cp-36-cp36m compatible dependencies for:
 1: grpcio<2.0dev,>=1.8.2; extra == "grpc"
    Required by:
      google-api-core==1.16.0
    But this pex only contains:

Any idea what cause the issue and possible workaround?

witty-crayon-22786

02/12/2020, 11:28 PM

@happy-kitchen-89482: a thing that might need to block upgrading pex in pants: https://github.com/pantsbuild/pex/issues/887 cc @aloof-angle-91616

aloof-angle-91616

02/13/2020, 2:16 AM

for https://github.com/pantsbuild/pex/issues/887 -- it's somewhat clear to me what the relevant change to the pip vendored code should be to fix the issue. is it preferred to make a PR against our pip vendored code first, or to make a PR against upstream pip first?

happy-kitchen-89482

02/13/2020, 3:48 AM

hrrmph, yes indeed

happy-kitchen-89482

02/13/2020, 3:48 AM

Can we fix this in pex, getting changes into pip sounds like a mess

happy-kitchen-89482

02/13/2020, 3:49 AM

I mean pex itself, not the vendored pip, although that is ok if we must

happy-kitchen-89482

02/13/2020, 3:49 AM

we already vendor a forked pip

aloof-angle-91616

02/13/2020, 3:49 AM

https://github.com/pantsbuild/pex/pull/888 is it fixed in pex via the vendored pip. it's very miniscule

aloof-angle-91616

02/13/2020, 3:50 AM

and i'm making a test for a PR against the pypa/pip repo and can add a corresponding test to the pex PR

happy-kitchen-89482

02/13/2020, 3:54 AM

Can you apply this in a PR in our fork? https://github.com/pantsbuild/pip

aloof-angle-91616

02/13/2020, 3:55 AM

yes!

aloof-angle-91616

02/13/2020, 3:55 AM

to summarize, what PRs should i be making?

aloof-angle-91616

02/13/2020, 3:55 AM

(1) make a PR in pantsbuild/pip

aloof-angle-91616

02/13/2020, 3:55 AM

oh or is that it

aloof-angle-91616

02/13/2020, 3:55 AM

witty-crayon-22786

02/13/2020, 3:56 AM

imo, we should at least open and reference an upstream (pip) PR.

👍 1