https://pantsbuild.org/ logo
Join Slack
Powered by
Hello again. I'm slowly sorting out my spaghetti m...
# general
f
Hello again. I'm slowly sorting out my spaghetti mess of dependencies (hurrah). Currently though I have a bit of a concern about speed. It's currently taking 45 seconds to run my migration script (not counting the actual DB changes!). It seems that pants is rebuilding all the dependencies every time I run it, rather than using an existing venv. I am getting some 
[WARN] Pants cannot infer owners for the following imports
- do I need to address those to fix this? ~ fixed these, it didn't change anything Here is the 
pex_binary
I'm having this caching problem with:
Copy code
pex_binary(
    name='migrate',
    entry_point='src/tools/migrate.py',
    layout='loose',
    dependencies=[
        'service/web/manage.py:lib',
    ],
)
w
Are you saying that, if you run the same command twice in a row - it rebuilds? Like, no code changes?
f
That's what it seems to be doing, yes
w
šŸ¤Æ
f
Copy code
14:45:34.76 [INFO] Initializing scheduler...
14:45:34.77 [INFO] Initializing Nailgun pool for 28 processes...
14:45:35.74 [INFO] Scheduler initialized.
ā ¤ 4.73s Resolve transitive targets
It spends a lot of time on that step
w
Can you delete the .pants.d folder - and then try again, just as a sanity? Make sure Pants isnt' doing anything sketchy
That should all get cached - after the first run. Can you also show the command you're running?
f
It's the 
pex_binary
above šŸ™‚
pants run :migrate
rm-ing the pants.d dir didn't change anything
w
Hmm, that's it... I would test deleting the .pants.d folder, and make sure that pants isn't doing anything off. Then, did you change any of the pants ignore files anywhere? If it's re-running after you delete everything, you can check out the logs in pants.d to see why it's re-running
f
Aha
Copy code
14:47:21.22 [ERROR] The scheduler was invalidated: Exception('pantsd process 34119 was using 4096.41 MiB of memory (above the `--pantsd-max-memory-usage` limit of 4096.00 MiB).')
w
Often some temp files might get picked up. After that, you can check out a 
pants peek
to make sure the dependencies are what you expect (there are other commands that are less verbose than peek, but I always peek - out of habit
Copy code
4096.41 MiB
Oh come on ... 0.41 Maybe try upping that command's memory, see what happens? Also, that shouldn't be in a log - that should be on the command line, no?
f
That was in .pants.d/workdir/pants.log
Not on the command line
w
Hmm, errors should be showing up on the command line - that feels like a problem
f
I can up the memory limit although 4 gig is... kind of a lot
w
How many targets/files do you have?
f
I am on a 36 gig mac but still šŸ˜„
In Python I have about 2,500 files
w
Hmm, that's not too bad. and then a bunch of deps I guess. You can get more information here: https://www.pantsbuild.org/stable/reference/subsystems/stats#memory_summary I have a handful of things I'm working on which should speed up runs, and reduce memory - but they're slowly rolling out over the next little while, as I have a few tools I'm building to help with these problems.
Would you mind reporting a bug for that error line not being at the command line? To me, anything that causes the scheduler to re-run every command needs to be front and centre, and I wouldn't want to lose tracking of that
Especially when the workaround is "easy"
f
Can do, where is your bug report page?
Here's the top memory usage (after upping the limit to 8G)
Copy code
1244160		51311		pants.engine.fs.PathGlobs
  2302800		28785		builtins.Snapshot
  2392176		49837		pants.engine.target.DependenciesRequest
  2392176		49837		pants.engine.target.ExplicitlyProvidedDependenciesRequest
  2438080		30479		builtins.Digest
  3624904		3504		(native) scandir
  4121185		1225		(native) process
  8734674		25114		(native) digest_file
  9775535		25391		(native) pants.engine.intrinsics.path_globs_to_digest
  11082225		28785		(native) pants.engine.intrinsics.digest_to_snapshot
  19187245		49837		(native) pants.engine.internals.graph.convert_dependencies_request_to_explicitly_provided_dependencies_request
  25293270		25391		(native) snapshot
w
f
I tried upping the limit to 8 GiB but it's still getting killed, so there's some sort of endless loop going on I think
w
Okay, that makes some sense too - because while the repo is kinda big, I didn't think it was OOM big. as a sanity, can you run 
pants peek ::
from the root dir and ensure it completes? I recall vaguely that someone had a symlink that was causing something like this (slow runs, lots of memory) but simpler commands just failed
Also, introspection will be your friend here: https://www.pantsbuild.org/stable/docs/using-pants/project-introspection You can see if one of your targets is just going wild
f
We don't have any symlinks - apart from 
node_modules/.bin
, but for the moment pants is only working on Python stuff
peek ::
works okay
w
Okay, so nothing obvious - I might try to do some introspection commands, and see if you have a dep cycle in there, or if you transitively include a lot more stuff than expected. In general, an OOM is weird though - I've never been able to cause one to occur
f
pants dependencies --transitive :migrate
outputs what I expect it to. But it is picking up all the node files so maybe that's where it's struggling
w
Oof, yeah, that could be a problem - is that expected that the node files would be included?
f
They're in a directory marked as a resource, so yes. I've just deleted node_modules for now
I haven't actually gotten as far as sorting out our JS stuff yet anyway
w
Gotcha - and for me (depending on which package manager I use), node_modules end up being a symlink hell šŸ˜¢
f
Yup clearing out node_modules has sorted the memory issue. Migrate now runs in 9s so that's definitely a bit better. Still kinda on the slow side though
w
How much of that is Pants, after the first run?
f
Using a pyenv it only takes 2.3s, so quite a chunk
w
whoa
Like, a cached run still adds like 6 seconds?
f
Roughly, yes
w
Oof, something sounds wrong - a second or maybe two, I could buy....
I'm not sure how easy this is to do - but you should have a pex_binary in your 
dist
folder, what happens if you run that directly?
f
erm, dist folder is empty
w
Oh, wait, sorry - you use 
pants run
If you run 
pants package :migrate
- it should materialize that pex_binary to a dist folder
And since it's all cached, that packaging should be fast
f
python ./dist/migrate.pex/__main__.py
is in the 2s range, so that's working right
Assuming that's the right way to run it šŸ™‚
w
Yeah, there are a few ways to run it. You could probably just run python dist/migrate.pex - but good. Did running 
package
that time take any significant amount of time? How big/number of files in that pex?
f
pants package :migrate
is about 3s or just under
There's just under 16,000 files in the pex
w
Alright, so copying those files might be the time https://www.pantsbuild.org/stable/reference/targets/pex_binaries#execution_mode https://www.pantsbuild.org/stable/reference/targets/pex_binaries#layout These two things are probably good to look into - they change the build/cache/runtime characteristics. Josh wrote a great writeup about using pex in docker, and while that's not the case here - I think it's still a great read: https://www.pantsbuild.org/blog/2022/08/02/optimizing-python-docker-deploys-using-pants And I'll take a quick look for it, but Huon wrote some great notes somewhere in the chat about pex_binary layout models a while back
I also have some numbers - there are tradeoffs: https://github.com/sureshjoshi/perfanity/tree/cached-pex
f
execution_mode='venv'
is a lot faster šŸ˜Ž
šŸŽ‰ 1
w
I think what you've ran into was what I ran into at the end, which is that with a 
loose
setup, you spend a lot of I/O time
f
layout=loose or packed seems to make it a smidge slower, so got rid of that. It is now beating the pyenv's time so that's pretty good šŸ˜„
w
šŸŽ‰
f
When cached of course šŸ™‚
w
Yeah, and essentially the trade-off in dev is what updates change, and how everything interacts. With docker, this is particularly noticeable, but even not - sometimes splitting out 3rd party dependencies (which are generally static) and 1st party code can have a ludicrous effect on caching/build time. With docker, I'd call this the standard way to use it (as per Josh's blog). Outside of docker šŸ¤· case by case
f
How would I go about splitting the third-party stuff? That should make it a lot faster if it doesn't need to re-grab the deps every time
With a changed migration file the runtime is 16-20s (depending on what I set layout to), so would be useful to try and bring that down as well
w
It doesnt' re-grab it, but yeah, this is a case of where Docker would cache that into a layer - while, pex on a machine - that's a little more nuanced. I spent some time looking into "pex in pex" kinda thing, but I can't recall where I left off. It looked something like this - https://github.com/sureshjoshi/perfanity/blob/cached-pex/multipex/BUILD But, keep in mind, that assumed everything gets unpacked into that Docker layer I mentioned. On a computer, I think there is some way to allow all of this to be unpacked into a dist - but I abandoned what I was working on, due to time reasons back then
Copy code
pex_binary(
    name="multipex",
    dependencies=[":multipex-deps", ":multipex-srcs"],
    entry_point="main.py",
    execution_mode="venv",
    layout="packed",
)
Like, I can't recall off the top of my head whether this idea in the end was actually faster, or if it was just more overhead
For rapid development iteration, I don't necessarily think needing to pack everything into a pex, and running it is necessarily optimal in the first place - so sometimes I just pants export everything into a venv, and do development - while package and all that is for releases/production
f
That multipex doesn't seem to make any difference in time taken
Looking at 
pants export
- not a lot of docs on there šŸ™‚ Would it be something like 
pants export --bin=migrate
?
w
That multipex doesn't seem to make any difference in time taken
Yeah, it seems like a lot of the time spent here anyways is on deleting and then materializing a bunch of files? But, I'm not sure I'd have to give that some thought. https://www.pantsbuild.org/stable/reference/goals/expor Re: export - generally I would export everything into a venv, and operate out of there. I think 
pants export ::
or similar, and then just work out of the dist (it's intended for IDEs and stuff - https://www.pantsbuild.org/stable/docs/using-pants/setting-up-an-ide)
f
Cool, I see. So you can use that to export a venv to disk and then use that in other tools
We're using Docker Compose atm for our dev env so I guess I'll look at adapting that existing stuff to pick up this venv for running our dev on. I was thinking we could get Pants to do everything but I guess it's better this way
Gonna be signing off for today but ty for your help :)
w
no problem Also, Pants can run the docker stuff too, or run it without docker - it's just a performance issue. There are some other ways to eject that I haven't messed around with (like workspace environments)
5 Views
Open in Slack
PreviousNext
Powered by