https://pantsbuild.org/ logo
Join Slack
Powered by
Has anyone successfully been able to <profile with...
# general
c
Has anyone successfully been able to profile with py-spy on OSX? I get the following error when trying to do so
Copy code
Error: Operation timed out (os error 60)
Reason: Operation timed out (os error 60)
c
I believe I have, but just followed the instructions in the docs, don’t recall I had to do anything special..
c
For this command, what did you use for 
<your_platform_dir>
?
Copy code
source ~/.cache/pants/pants_dev_deps/<your platform dir>/bin/activate
c
ah, if you’re using the new scie-pants, that path will be under a different path https://github.com/pantsbuild/scie-pants#caveats
It extracts the needed files into a base directory that is traditionally located in the local user cache subdirectory 
nce
(
~/.cache/nce
on Linux, 
~/Library/Caches/nce
on macOS and 
~\AppData\Local\nce
on Windows).
you see which when ever you get your pants bootstrapped on a new version
c
What’s the new scie-pants? I usually invoke pants via the deprecated command
Copy code
./pants <cmd> ...
c
ah, then you where in the right spot.. It’s this path you see when bootstrapping pants:
Installing pantsbuild.pants==2.16.0 into a virtual environment at /Users/andreas.stenius/.cache/pants/setup/bootstrap-Darwin-arm64/2.16.0_py39
regarding scie-pants, more info here: https://www.pantsbuild.org/docs/installation
oh, dev_deps…. maybe I’m mistaken…
my guess it would be this one: 
~/.cache/pants/pants_dev_deps/Darwin.arm64.arm.py39.venv/bin/activate
apologies for my confusion..
I imagined it was the Python env for pants itself that was required…
c
hmm, this is all I have in the 
~/.cache/pants/
directory…
Copy code
ls ~/.cache/pants/
lmdb_store   named_caches setup
e
@cold-soccer-63228 it's under 
~/.cache/pants/setup
- @curved-television-6568 was assuming you were hacking on Pants, but it appears you're just using a released Pants in your own repo.
The actual directory should be ~obvious given the Pants version you have configured in 
pants.toml
, although IIRC the obvious dir is a symlink to a tmpdir-style-sibling dir.
Yeah, like so - this was an old version of the example-django repo with a 
./pants
script checked in:
Copy code
$ ls -l ~/.cache/pants/setup/bootstrap-Linux-x86_64/
total 16
lrwxrwxrwx 1 jsirois jsirois   76 Aug 28 12:32 2.14.0_py39 -> /home/jsirois/.cache/pants/setup/bootstrap-Linux-x86_64/pants.m5B9ZS/install
drwx------ 3 jsirois jsirois 4096 Aug 28 12:32 pants.m5B9ZS
drwxr-xr-x 2 jsirois jsirois 4096 Aug 28 12:31 pex-2.1.103
drwxr-xr-x 2 jsirois jsirois 4096 Aug 28 12:32 virtualenv-20.4.7
c
ah, I see. thanks for that. here’s what I ran now.
Copy code
source ~/.cache/pants/setup/bootstrap-Darwin-arm64/2.15.1_py39/bin/activate
export PYTHONPATH=src/pants:$PYTHONPATH
sudo py-spy record --subprocesses -- python -m pants.bin.pants_loader --no-pantsd --changed-since=origin/master dependees --transitive --closed
still getting the first error regarding the operation timeout… really my goal is the following: we have a large repo, and have a pretty tangled dependency graph with lots of circular dependencies. we suspect that that’s the reason why the 
dependees --transitive
command takes a super long time to run. but we want to confirm if that hypothesis is true, so we wanted to profile a run of the 
dependees --transitive
command
e
Did you try the pid means?: https://pypi.org/project/py-spy/0.1.4/
I have not, but its right there in the docs.
I think to use pid you'd need to 1st run 
pants -V
to get the pantsd up, use that pid to attach to, then run the command you care to profile.
c
It was the “pants_dev_deps” that threw me.. I was on the right track to begin with 😁 https://pantsbuild.slack.com/archives/C046T6T9U/p1693249503237309?thread_ts=1693246754.732469&amp;cid=C046T6T9U
e
@cold-soccer-63228 did you google this? Is it relevant?: https://github.com/benfred/py-spy/issues/242#issuecomment-649139155
You'd want to look at the venv bin dir Python and see if it symlinks to a system Python for that to be relevant.
If it is relevant, you too can ❤️ macOS.
c
hmm, it’s a symlink to homebrew-managed Python 3.9…
this comment leads me to believe that disabling SIP won’t fix the problem (?) is there another way to profile Pants code?
e
It's just Python + native deps; so you can always google around for how to debug python to find alternatives. That said, did you try the pid thing?
c
yeah, tried the PID thing as well and got the same error :/
e
So, wait @cold-soccer-63228 - your command above just says 
-- python ...
- A bare 
python
is dangerous! What is 
python
? System, brew, other?
I would not trust activate here - use explicit paths.
It should be the absolute path to the venv python I'd think.
Yeah, right - you run under sudo, root has a different PATH; so 
python
certainly doesn't mean what you think it means.
I think you'd need 
sudo -E ...
, but still, better an abs path.
c
Ah, amazing. I got it to run with this command
Copy code
sudo -E py-spy record --subprocesses -- /Users/hughhan/.local/share/virtualenvs/front-porch-2eoLkmA-/bin/python -m pants.bin.pants_loader --no-pantsd --changed-since=origin/master dependees --transitive --closed
but now, it looks like I’m getting a separate issue
Copy code
/Users/hughhan/.local/share/virtualenvs/front-porch-2eoLkmA-/bin/python: Error while finding module specification for 'pants.bin.pants_loader' (ModuleNotFoundError: No module named 'pants')
I think I need to export Pants’ code?
Copy code
export PYTHONPATH=src/pants:$PYTHONPATH
in this case, how would I do that, since I’m not developing my own version of Pants, and I’m just using the binary?
e
So, why are you using a venv that is not the Pants venv? That's the issue here. The Pants venv should be in 
~/.cache/pants/setup/...
and certainly not 
~/.local/...
.
👍 1
c
ah, ok. I was using the venv for my local project. let me use the Pants venv
e
So, @cold-soccer-63228 IIUC, this means SIP was in fact the issue.
c
okay, seems like it. when I try to use the Pants venv, I get the same error as above
e
But the general lesson is don't use 
python
or unqualified paths in general when debugging.
👍 1
Control everything you can.
👍 1
c
got it. is there any way to somehow tell Pants to use a venv that uses a custom version of Python instead of system Python?
e
Is the Pants venv actually using the system Python?
c
I don’t think so. It tells me that it’s pointing to one managed by Homebrew…
oh, just kidding
it’s using system Python
Copy code
ls -l ~/.cache/pants/setup/bootstrap-Darwin-arm64/2.15.1_py39/bin
[...]
lrwxr-xr-x  1 hughhan  staff    63 May 26 14:45 python -> /Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9
e
Looks like you can by exporting `PYTHON`: https://github.com/pantsbuild/setup/blob/gh-pages/pants
So 1st kill the setup cache.
✅ 1
c
re-bootstrapping now…
Copy code
Bootstrapping Pants using /opt/homebrew/opt/python@3.9/libexec/bin/python
Installing pantsbuild.pants==2.15.1 into a virtual environment at /Users/hughhan/.cache/pants/setup/bootstrap-Darwin-arm64/2.15.1_py39
e
Looks better.
c
Copy code
[ERROR] Absolute spec path /opt/homebrew/Cellar/python@3.9/3.9.16/Frameworks/Python.framework/Versions/3.9/bin/python3.9 does not share build root /Users/hughhan/src/front-porch
do I need to 
cd
into the directory of my Python?..
e
No clue. You need to burn your Mac though.
I don't know Brew. @curved-television-6568 may. I don't use Mac.
c
😆
ok, I got it to work somehow by passing the symlinked Python (from my virtualenv) instead of the actual Homebrew Python. not sure why that worked
thanks for all of the help @enough-analyst-54434… this was truly really helpful
e
You're welcome, but keep in mind this was all just generic debugging techniques. I don't do Mac or py-spy or even Pants any longer.
c
when using python, I shy both system python and brew python and rely solely on pythons I install using 
pyenv
…
e
Aha, right. @cold-soccer-63228 that last error was from Pants. It did not like the path.
w
soooo, ime, attaching 
py-spy
to a running process is much easier (and we should probably update those instructions). in that case, you: 1. start 
pantsd
(e.g. using 
pants help
), 2. find the pid of 
pantsd
using e.g. 
ps -ef | grep pantsd
3. attach to 
pantsd
using 
sudo py-spy record --format flamegraph -o ~/Desktop/pantsd-help.svg -p $PID
4. execute the command you want to profile 5. ctrl+c out of 
py-spy
👍 1
that doesn’t allow for profiling all of the startup time of 
pantsd
itself, but
e
w
weird. sorry I missed that
e
I would guess it was the same SIP issue.
If the interpreter running pants(d) is system and SIP'ed, you're out of luck.
👍 2
c
thanks for all of the help y’all! i’m all resolved here now 🙂
n
o
3 Views
Open in Slack
PreviousNext
Powered by