k... see the help. that might be ``-m flake8-pantsbuild:flake8``

we have seen a recent occurrence of the "pexrc is used from pants" issue

howdy! i have a pip command generated by pex that looks like:

pip \
  --disable-pip-version-check \
  -vvvvv \
  --isolated \
  download \
  --dest \
  blah \
  --no-index \
  --find-links \
  <https://example.com/home/third_party/source/python/bootstrap/> \
  --find-links \
  <https://example.com/home/third_party/source/python/wheels/> \
  --find-links \
  <https://example.com/home/third_party/source/python/deprecated/> \
  --platform \
  manylinux2014_x86_64 \
  --platform \
  linux_x86_64 \
  --implementation \
  cp \
  --python-version \
  27 \
  --abi \
  cp27m \
  --only-binary :all: \
  --pre \
  'pytest-cov<3,>=2.8.1' \
  'pytest<3.7,>=3.0.7' \
  'zipp==1.1.0'

more followup on the pex 2.0 upgrade: verbosity

following up on the above, now more specifically about this conflict

QQ: is there a flag/env var to force pants (then pex) to re-resolve python 3rdparty libs, ignoring the pex cache?

That's a Pants v1 question since Pants v2 disables the pex cache.

So, env var iff it propagates through v1 code - I'm not sure it does - would be PEX_ROOT=/tmp/random

Pex build-time flag is --disable-cache.

Thanks, John!

Relatedly, a high level, what does --resolver-jobs entail? https://github.com/pantsbuild/pants/blob/717e7687dfa824129c50a964b4a66f278508690b/src/python/pants/python/python_setup.py#L148-L155

Suppose we have 10 requirements to resolve, does N=5 mean pip is going to have 5 parallel connections, like coursier’s worker count, or pex will going to call 5 parallel pips?

If the latter, I don’t quite get how it works, does that mean the transitive context does not matter then, in case req A and req B needs different version of req X?

I’m profiling some pex executions… It looks like about 75% of the overhead of running a pex right now is importing things from

pex.third_party

- does this sound about right, and do people have thoughts on how to improve that?

Separate thread: I’m currently looking at shortcutting interpreter selection - changing the code to just using the current interpreter vs listing them all and filtering, drops my trivial hello-world

.pex

execution from ~1000ms to ~500ms - if I wanted to turn this into something real, any thoughts on what it would look like? A flag/env-var to specify a short-circuiting selection?

And one hopefully last one… Do we expect/desire any resolve caching for pex outside of pants?

(via Emily)

@enough-analyst-54434 I’ve noticed that importing

pyparsing

is super expensive, and we import two copies of it, one from

./.bootstrap/pex/vendor/_vendored/packaging/pyparsing.py

and another from

./.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py

. I don’t know much about this vendoring, but do you know if there’s a nice way to use one twice, rather than load them both?

There is not. Pex vendors packaging for its direct use and setuptools does for its own use. The ugly way is to use setuptools via the mangled import path.

As a general note, it would be extra nice if Tweeps hackweeking on Pex could file a tracking issue in Pex describing the problem being solved, self assign, and mark in-progress.

This is just me profiling for funsies tbh 🙂 But will do

(My goal is “I want to be able to distribute pex files which have some logic and some deps, and have them have <100ms overhead to run”, and we’re currently a ways away from there. It feels surmountable, but annoyingly fiddly because

pkg_resources

🙂)

pyoxidizer's strategy of importing from memory is another way we could swing this investigation

What would that fundamentally change?

the time to traverse and then read the filesystem to import modules from

As far as I can tell, that’s not the bottleneck I’m seeing… I’m seeing things like “We spend 50ms compiling regexes that are classproperties”

A re-read of all these threads would be useful @aloof-angle-91616 as I thnk the fundamental issue is over-importing by pkg_resources.

thanks!

Not that import itself is too slow