https://pantsbuild.org/ logo
Join Slack
Powered by
Hi team, I am facing the following error while I r...
# general
g
Hi team, I am facing the following error while I run tests that use jupyter notebook in pants. The test looks like
Copy code
@testbook(PATH_NOTEBOOKS / "prep.ipynb", execute=True)
def test_prep_notebook(tb: TestbookNotebookClient):
    print_notebook_output(tb)
The error is like
Copy code
/.cache/pants/named_caches/pex_root/venvs/s/86f08450/venv/lib/python3.9/site-packages/jupyter_client/client.py:202: in _async_wait_for_ready
    raise RuntimeError("Kernel died before replying to kernel_info")
E   RuntimeError: Kernel died before replying to kernel_info
How to solve this issue? Thanks you!
Anyone has any suggestion to this problem?
e
@gray-shoe-19951 that problem looks very domain specific. It looks like it involves a 
@testbook
decorator for one - I have never heard of such a thing, and a suspicious looking constant 
PATH_NOTEBOOKS
- and it's semi-well-known Pants does not play well with certain paths, namely because Pants executes nothing except the 
run
(and maybe 
repl
?) goals in the CWD. Everything else executes from an ephemeral 
/tmp/*
dir where the files needed by the process - and only those files - are copied there. This includes tests. Given that ... can you say more about those test lines?
g
hi @enough-analyst-54434, thanks for helping me. We use this lib https://testbook.readthedocs.io/en/latest/ to run notebook in our tests. The 
PATH_NOTEBOOKS
is just a dir constants. We have declare the resource for it and add it to the dependencies into our tests. I believe Pants is able to find them. Here the issue seems to me is that Pants is not running notebook well for some issue on subprocess (e.g. the error message seems complaining the kernel from running notebook). Btw, the test is passed if I just run it using pytest instead of pants test.
e
@gray-shoe-19951 the interesting bit is the value of the `PATH_NOTEBOOKS`constant. Is it a relative path or an absolute path?
g
Something like below PATH_SOURCE = UPath(os.path.dirname(os.path.abspath(file))) PATH_PROJECT_ROOT = PATH_SOURCE.parent.parent PATH_NOTEBOOKS = PATH_PROJECT_ROOT / "notebooks"
e
What is a UPath ?
g
I don't think it matters :)
e
There is a theme here I hope you're sensing. There is a ton of context, that may or may not matter, that only you have. When providing remote support, you are our eyes and its hard to diagnose blind.
Can you maybe print out the value of 
PATH_NOTEBOOKS / "prep.ipynb"
and report that?
Best - can you provide an open source example repo with steps for repro?
g
I am just curious on what do you feel the problem is atm?
If pants can not locate the file, wouldn't it be a different error message? I don't quite understand that error message about the kernel :(
e
@gray-shoe-19951 I have no clue what the problem is, but with no hands on repro case all I can do is flail or drop everything and try to build a repro case myself and learn notebooks, this test framework, etc.
👌 1
So, for one example, that page you linked talks about sharing a kernel and there is a link for how to do that with pytest fixtures. If sharing a kernel means a kernel runs on a port, for example, that will be a huge problem with Pants since Pants runs each test file in parallel and they'd all try to grab the same port and only 1 test file would win and all others would die.
But that is a wild guess! I have no context. You'll need to dump a ton of context or come up with an open source repro case. That really is the best path forward.
g
Got you. I will dig into it further. Otherwise, I will get a repo
e
Excellent. Thank you.
g
@enough-analyst-54434 I prepared this repo to demo my issue. Appreciate your help and time.
e
I need alot of help. Where do I get conda from? How do you get conda in other words? I'm on Linux.
g
e
Ok, thanks. I'll take a look later today and report back.
g
sure. take your time
e
That works. Did fail, now works.
g
Wow, I will take a look later today. Thank you!
e
I will say, you do not need to do the abspath stuff, just 
Path(__file__).parent / ".." / "notebooks"
is enough for Pants and pytest both. Also, it would be good to hear why you use the pants PEX. Those things are huge and ... generally to be avoided.
Thanks for providing the repo. That is much simpler. I don't know how much remote support you've done in the past, but its gold. Words generally fail.
g
You mean the downloaded binary file?
e
Correct - bad news unless you have a very good reason.
g
We are behind a proxy
e
Gotcha.
g
:(
e
That makes sense then. Easier to copy 1 file behind proxy than 1304 files.
g
Yep
e
We'll be changing the install method in the new year, It will involve 2 files in the standard use. 1 for a 
pants
that goes on your PATH instead of checked into the repo and 1 more per version of Pants you use (it will be a Pex, but targeted per-OS).
The `pants`you install once ever, the 2nd file is whenever you want to switch Pants versions.
g
Ok interesting
I will watch out the changes
Yep it seems atm the pants has to been in the repo
If I put pants in to other place and include it into path, it won't work
e
Correct. The PATH thing will be new. Still working on the 1st alpha release here: https://github.com/pantsbuild/scie-pants
❤️ 1
g
One follow-up question on the notebook, if the codes in the notebook important some src files, I believe atm pants cannot detect it automatically. We need to manually specify the dependencies, am I right?
e
Correct. Pants does not have a dependency inference parser for ipynb files yet, only for .py files.
👌 1
5 Views
Open in Slack
PreviousNext
Powered by