Another question, when changing a dependency for a test/pex binary target, we get a relatively big penalty (anything from 60-250 seconds), which rather large and really hurts the feedback loop, is this expected?

Building requirements.pex with 75 requirements: aiobotocore==1.4.2; ...

Takes most of the time

I've also noticed that building the requirements pex is a time consuming task. My impression is that it is largely pip that takes a lot of time.. wondering if there is something that could be done to help speed pip up..?

Hm, are you using a constraints file? https://www.pantsbuild.org/docs/python-third-party-dependencies#user-lockfile

Yeah, currently using a constraints file gives you a huge speedup (after the initial resolve)