https://pantsbuild.org/ logo
#general
Title
# general
r

refined-addition-53644

05/27/2022, 12:42 PM
Hi, how does pants handle VCS dependency using pex during test? My tests are taking quite a long time (~30 mins). Is it building this dependency from source for every test ? I am using multiple resolves if that's something that might be relevant.
We have a dependency on facebook prophet library which I am installing directly from github. This is until they do a new release on pypi.
h

hundreds-father-404

05/27/2022, 12:48 PM
Yeah, Pex clones the repo and builds it like it's an sdist. But it should be reusing the work from other runs. Sounds like you're not seeing that?
r

refined-addition-53644

05/27/2022, 1:24 PM
I am not sure. The only thing I saw was pants accumulating all these dependencies for each test and it took forever. Earlier I was using a global
requirements.txt
where this was a dependency and the tests used to run much quicker. But after I switched to multiple resolves using pex, these tests are taking forever
Although github is facing some issues today. So I am not sure if that's what is causing this issue. Locally on my laptop this test doesn't take so long. It's on aws where it was taking too long. I will wait and see if it goes away next week or not.
1
h

hundreds-father-404

05/27/2022, 1:32 PM
Yeah the old approach was: build a PEX once w/ everything, then extract out smaller subset PEXes from that Now the approach is: directly build the smallest PEX necessary for the task. Share Pex's cache of downloading wheels & building sdists etc across those different runs.
Share Pex's cache of downloading wheels & building sdists etc across those different runs.
It's crucial that that is working properly - so definitely please do keep us informed on this!
👍 1
r

refined-addition-53644

05/30/2022, 4:30 PM
It does take consistently ~30 minutes when running with multiple resolves on aws. Earlier without multiple resolves it used take like 5-6 minutes. All my 3rd party dependencies except this one are defined inside
pyproject.toml
which is then used by
poetry_requirements
On the other hand I use
python_requirement
for installing prophet from git. I do provide the same resolve for
poetry_requirements
and
python_requirement
w

witty-crayon-22786

05/31/2022, 4:58 PM
mm, seems like yes. will leave discussion in the other thread.