pantsbuild #pex

Channels

welcome

github-notifications

development

happy-kitchen-89482

04/02/2020, 7:46 PM

So a python_library isn't producing a pex on its own

happy-kitchen-89482

04/02/2020, 7:49 PM

when you run

./pants binary <python binary target>

, a pex gets created from all of the transitive closure of the internal (python_library) and external (3rdparty requirements) dependencies of that target.

👍 1

happy-kitchen-89482

04/02/2020, 7:52 PM

In v2 we are still working on what is the best way to introduce caching for the pex resolver. In v2 by default everything, by design, is hermetic and there can be no side-effects. A cache is a kind of side-effect. So we're still figuring out how to incorporate that safely. But we should have a solution very soon. We need this for other things as well, obviously.

happy-kitchen-89482

04/02/2020, 7:52 PM

Does this help?

😁 1

happy-kitchen-89482

04/02/2020, 10:51 PM

Following up on a private message - in v2, when you run

./pants binary <target>

today and any inputs have changed (even if only first-party code) the pex creation will today re-run pip to re-resolve the remote dependencies. This is a byproduct of pex building being completely hermetic, and therefore dependent only on its inputs. We are actively working on fixing this in two ways: 1) Add support for building a pex on top of an existing pex, so that there is one stable 3rdparty requirements pex, and we build the binary pex out of that. 2) Allow pants to use Pex's existing caching mechanism (by allowing this specific form of side-effect).

🔥 1

happy-kitchen-89482

04/02/2020, 10:51 PM

So we should have a solution within a few days.

aloof-angle-91616

04/02/2020, 10:57 PM

i’m very glad to hear we’re considering using the pex cache!!! i feel like benjy probably answered the question here but one interesting point is that as benjy mentioned, if you want to build an “ipex” file which lazily downloads requirements, it’s possible to use a pared-down form of

pip fetch

(one example is in https://github.com/pypa/pip/issues/7819) to get all the necessary info to build the pex with, without needing to download dependencies. but if we can make use of the normal pex cache, then the speedup of that approach is lessened somewhat, although i am actively investigating this. pip is working on upgrading its resolver upstream, which would potentially change further what is cached across pex runs in the future as well.

happy-kitchen-89482

04/03/2020, 12:10 AM

After chatting with @calm-balloon-91412 I realized that their concern is actually the opposite of lazy downloads

👍 1

aloof-angle-91616

04/03/2020, 12:12 AM

there’s at least one open issue i made maybe two years ago about composing pex files at build time that it sounds like your work might solve

happy-kitchen-89482

04/03/2020, 12:12 AM

Their concern is around being able to build a consistent artifact if you go back in time in Git

aloof-angle-91616

04/03/2020, 12:12 AM

oh, ah, ok

happy-kitchen-89482

04/03/2020, 12:12 AM

ideally without going back to pip

👍 1

happy-kitchen-89482

04/03/2020, 12:13 AM

Anyway, I have more context now, I think we can address their concerns

aloof-angle-91616

04/03/2020, 12:13 AM

awesome!!

witty-crayon-22786

04/03/2020, 10:15 PM

an issue that affects the 1.26.x release... maybe not a blocker, but challenging for users to resolve without arcane knowledge: https://github.com/pantsbuild/pex/issues/947

happy-kitchen-89482

04/04/2020, 1:22 AM

👍

aloof-angle-91616

04/06/2020, 5:11 PM

@happy-kitchen-89482 will review your merging pex files PR today, thanks a ton! i’m super pumped about this work

witty-crayon-22786

04/07/2020, 6:24 PM

before i dive in, has an error been resolved here recently? this is from pex

2.1.6

in the

1.26.x

pants branch:

Copy code

Traceback (most recent call last):
   File "/Users/stuhood/src/source/.pants.d/test/pytest-prep/CPython-2.7.16/9ea622545c7f4f99caa61a8eef94bfbd3992d5a7/.bootstrap/pex/pex.py", line 391, in execute
     working_set = self._activate()
   File "/Users/stuhood/src/source/.pants.d/test/pytest-prep/CPython-2.7.16/9ea622545c7f4f99caa61a8eef94bfbd3992d5a7/.bootstrap/pex/pex.py", line 89, in _activate
     for dist in env.activate():
   File "/Users/stuhood/src/source/.pants.d/test/pytest-prep/CPython-2.7.16/9ea622545c7f4f99caa61a8eef94bfbd3992d5a7/.bootstrap/pex/environment.py", line 232, in activate
     self._working_set = self._activate()
   File "/Users/stuhood/src/source/.pants.d/test/pytest-prep/CPython-2.7.16/9ea622545c7f4f99caa61a8eef94bfbd3992d5a7/.bootstrap/pex/environment.py", line 393, in _activate
     resolved = self._resolve(working_set, all_reqs)
   File "/Users/stuhood/src/source/.pants.d/test/pytest-prep/CPython-2.7.16/9ea622545c7f4f99caa61a8eef94bfbd3992d5a7/.bootstrap/pex/environment.py", line 275, in _resolve
     distributions_by_key[<http://distribution.as|distribution.as>_requirement().key].append(distribution)
 AttributeError: 'NoneType' object has no attribute 'as_requirement'

witty-crayon-22786

04/07/2020, 10:34 PM

so, i'm debugging an issue that looks like it will require getting verbose logging from the

pip

invokes running under

pex

... https://github.com/pantsbuild/pants/pull/9328 propagated pants' logging to pex, but pex does not propagate its own log level to pip

happy-kitchen-89482

04/08/2020, 12:59 AM

https://pantsbuild.slack.com/archives/C18RRR4JK/p1586307536009500

🔥 1

witty-crayon-22786

04/10/2020, 5:26 PM

what might cause the exact same pex command targeting explicit platforms to fail on one machine vs another (full disclosure: both machines have an /etc/pexrc file, but they're identical):

Copy code

pex --no-build --no-pypi --disable-cache -f /tmp/tmp.lZIeuxe9PT/dist/deploy/wheels/pantsbuild.pants/e789654e53bd346f7d3604402f13aca03f53429d/1.26.0rc1+gite789654e -f /tmp/tmp.lZIeuxe9PT/dist/deploy/wheels/3rdparty/e789654e53bd346f7d3604402f13aca03f53429d/1.26.0rc1+gite789654e -o /tmp/tmp.lZIeuxe9PT/dist/pants.1.26.0rc1+gite789654e.py36.pex -vvvvvvvvv --no-emit-warnings --no-strip-pex-env --script=pants '--interpreter-constraint=CPython==3.6.*' --platform=linux_x86_64-cp-36-m --platform=macosx_10.11_x86_64-cp-36-m pantsbuild.pants==1.26.0rc1+gite789654e pantsbuild.pants.contrib.confluence==1.26.0rc1+gite789654e pantsbuild.pants.contrib.go==1.26.0rc1+gite789654e pantsbuild.pants.contrib.mypy==1.26.0rc1+gite789654e pantsbuild.pants.contrib.node==1.26.0rc1+gite789654e pantsbuild.pants.contrib.python.checks==1.26.0rc1+gite789654e pantsbuild.pants.contrib.scalajs==1.26.0rc1+gite789654e pantsbuild.pants.contrib.scrooge==1.26.0rc1+gite789654e pantsbuild.pants.contrib.avro==1.26.0rc1+gite789654e pantsbuild.pants.contrib.googlejavaformat==1.26.0rc1+gite789654e pantsbuild.pants.testutil==1.26.0rc1+gite789654e

aloof-angle-91616

04/11/2020, 6:00 AM

@happy-kitchen-89482 i couldn't find this issue when trying to look for it earlier -- this is a pex issue that can probably be resolved very soon by the recent work on merging PEX files. super excited about the further implications of this feature shipped this week! https://github.com/pantsbuild/pex/issues/454

aloof-angle-91616

04/11/2020, 6:01 AM

that issue mostly speaks in terms of the PEX_PATH, since pants v2 and merging snapshots was not as obvious a concept back in 2018, but it may be useful background

aloof-angle-91616

04/11/2020, 6:01 AM

i just wanted to bring it to your attention because that problem is another example of something that is now much easier to attack with pants v2 >=]

aloof-angle-91616

04/11/2020, 6:02 AM

and my first immature investigations into the pex codebase were in trying to attack that issue #454 at the pex level, so it has some sentimental value for me

witty-crayon-22786

04/17/2020, 12:22 AM

ftr: we're seeing tons more network traffic with 1.26.x / pex 2 (about 10x, even with concurrency pinned lower than that)... i'll be spending time tomorrow to investigate that.

👍 1

red-jackal-13334

04/23/2020, 9:10 AM

hi everyone!

red-jackal-13334

04/23/2020, 9:17 AM

i am trying to package azure cli as pex so when i do:

Copy code

pip install azure-cli
python -m azure.cli

everything works as expected. but when i try this

Copy code

pex azure-cli -m azure.cli -o az
./az

i get this error

Copy code

Traceback (most recent call last):
  File "/az/.bootstrap/pex/pex.py", line 396, in execute
  File "/az/.bootstrap/pex/pex.py", line 328, in _wrap_coverage
  File "/az/.bootstrap/pex/pex.py", line 359, in _wrap_profiling
  File "/az/.bootstrap/pex/pex.py", line 447, in _execute
  File "/az/.bootstrap/pex/pex.py", line 544, in execute_entry
  File "/az/.bootstrap/pex/pex.py", line 551, in execute_module
  File "/usr/local/lib/python3.7/runpy.py", line 201, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/usr/local/lib/python3.7/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/root/.pex/installed_wheels/70125b7d832d1ca0e01f7578de3d97524756eb0e/azure_cli_core-2.4.0-py3-none-any.whl/azure/__init__.py", line 5, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'

how do i debug and fix this?

red-jackal-13334

04/23/2020, 9:19 AM

everything is build and tested in python:3.7-alpine docker container

red-jackal-13334

04/23/2020, 9:21 AM

Copy code

pex --version
pex 2.1.9