https://pantsbuild.org/ logo
#general
Title
# general
g

gray-shoe-19951

12/14/2022, 12:32 AM
Hi team, I am facing the following error while I run tests that use jupyter notebook in pants. The test looks like
Copy code
@testbook(PATH_NOTEBOOKS / "prep.ipynb", execute=True)
def test_prep_notebook(tb: TestbookNotebookClient):
    print_notebook_output(tb)
The error is like
Copy code
/.cache/pants/named_caches/pex_root/venvs/s/86f08450/venv/lib/python3.9/site-packages/jupyter_client/client.py:202: in _async_wait_for_ready
    raise RuntimeError("Kernel died before replying to kernel_info")
E   RuntimeError: Kernel died before replying to kernel_info
How to solve this issue? Thanks you!
Anyone has any suggestion to this problem?
e

enough-analyst-54434

12/14/2022, 2:39 PM
@gray-shoe-19951 that problem looks very domain specific. It looks like it involves a
@testbook
decorator for one - I have never heard of such a thing, and a suspicious looking constant
PATH_NOTEBOOKS
- and it's semi-well-known Pants does not play well with certain paths, namely because Pants executes nothing except the
run
(and maybe
repl
?) goals in the CWD. Everything else executes from an ephemeral
/tmp/*
dir where the files needed by the process - and only those files - are copied there. This includes tests. Given that ... can you say more about those test lines?
g

gray-shoe-19951

12/14/2022, 2:57 PM
hi @enough-analyst-54434, thanks for helping me. We use this lib https://testbook.readthedocs.io/en/latest/ to run notebook in our tests. The
PATH_NOTEBOOKS
is just a dir constants. We have declare the resource for it and add it to the dependencies into our tests. I believe Pants is able to find them. Here the issue seems to me is that Pants is not running notebook well for some issue on subprocess (e.g. the error message seems complaining the kernel from running notebook). Btw, the test is passed if I just run it using pytest instead of pants test.
e

enough-analyst-54434

12/14/2022, 3:55 PM
@gray-shoe-19951 the interesting bit is the value of the `PATH_NOTEBOOKS`constant. Is it a relative path or an absolute path?
g

gray-shoe-19951

12/14/2022, 4:01 PM
Something like below PATH_SOURCE = UPath(os.path.dirname(os.path.abspath(file))) PATH_PROJECT_ROOT = PATH_SOURCE.parent.parent PATH_NOTEBOOKS = PATH_PROJECT_ROOT / "notebooks"
e

enough-analyst-54434

12/14/2022, 4:06 PM
What is a UPath ?
g

gray-shoe-19951

12/14/2022, 4:07 PM
I don't think it matters :)
e

enough-analyst-54434

12/14/2022, 4:08 PM
There is a theme here I hope you're sensing. There is a ton of context, that may or may not matter, that only you have. When providing remote support, you are our eyes and its hard to diagnose blind.
Can you maybe print out the value of
PATH_NOTEBOOKS / "prep.ipynb"
and report that?
Best - can you provide an open source example repo with steps for repro?
g

gray-shoe-19951

12/14/2022, 4:12 PM
I am just curious on what do you feel the problem is atm?
If pants can not locate the file, wouldn't it be a different error message? I don't quite understand that error message about the kernel :(
e

enough-analyst-54434

12/14/2022, 4:30 PM
@gray-shoe-19951 I have no clue what the problem is, but with no hands on repro case all I can do is flail or drop everything and try to build a repro case myself and learn notebooks, this test framework, etc.
👌 1
So, for one example, that page you linked talks about sharing a kernel and there is a link for how to do that with pytest fixtures. If sharing a kernel means a kernel runs on a port, for example, that will be a huge problem with Pants since Pants runs each test file in parallel and they'd all try to grab the same port and only 1 test file would win and all others would die.
But that is a wild guess! I have no context. You'll need to dump a ton of context or come up with an open source repro case. That really is the best path forward.
g

gray-shoe-19951

12/14/2022, 4:35 PM
Got you. I will dig into it further. Otherwise, I will get a repo
e

enough-analyst-54434

12/14/2022, 4:35 PM
Excellent. Thank you.
g

gray-shoe-19951

12/16/2022, 6:52 PM
@enough-analyst-54434 I prepared this repo to demo my issue. Appreciate your help and time.
e

enough-analyst-54434

12/16/2022, 6:54 PM
I need alot of help. Where do I get conda from? How do you get conda in other words? I'm on Linux.
g

gray-shoe-19951

12/16/2022, 6:56 PM
e

enough-analyst-54434

12/16/2022, 6:56 PM
Ok, thanks. I'll take a look later today and report back.
g

gray-shoe-19951

12/16/2022, 6:56 PM
sure. take your time
e

enough-analyst-54434

12/16/2022, 7:55 PM
That works. Did fail, now works.
g

gray-shoe-19951

12/16/2022, 7:56 PM
Wow, I will take a look later today. Thank you!
e

enough-analyst-54434

12/16/2022, 7:59 PM
I will say, you do not need to do the abspath stuff, just
Path(__file__).parent / ".." / "notebooks"
is enough for Pants and pytest both. Also, it would be good to hear why you use the pants PEX. Those things are huge and ... generally to be avoided.
Thanks for providing the repo. That is much simpler. I don't know how much remote support you've done in the past, but its gold. Words generally fail.
g

gray-shoe-19951

12/16/2022, 8:00 PM
You mean the downloaded binary file?
e

enough-analyst-54434

12/16/2022, 8:00 PM
Correct - bad news unless you have a very good reason.
g

gray-shoe-19951

12/16/2022, 8:01 PM
We are behind a proxy
e

enough-analyst-54434

12/16/2022, 8:01 PM
Gotcha.
g

gray-shoe-19951

12/16/2022, 8:01 PM
:(
e

enough-analyst-54434

12/16/2022, 8:01 PM
That makes sense then. Easier to copy 1 file behind proxy than 1304 files.
g

gray-shoe-19951

12/16/2022, 8:01 PM
Yep
e

enough-analyst-54434

12/16/2022, 8:03 PM
We'll be changing the install method in the new year, It will involve 2 files in the standard use. 1 for a
pants
that goes on your PATH instead of checked into the repo and 1 more per version of Pants you use (it will be a Pex, but targeted per-OS).
The `pants`you install once ever, the 2nd file is whenever you want to switch Pants versions.
g

gray-shoe-19951

12/16/2022, 8:04 PM
Ok interesting
I will watch out the changes
Yep it seems atm the pants has to been in the repo
If I put pants in to other place and include it into path, it won't work
e

enough-analyst-54434

12/16/2022, 8:38 PM
Correct. The PATH thing will be new. Still working on the 1st alpha release here: https://github.com/pantsbuild/scie-pants
❤️ 1
g

gray-shoe-19951

12/17/2022, 3:36 AM
One follow-up question on the notebook, if the codes in the notebook important some src files, I believe atm pants cannot detect it automatically. We need to manually specify the dependencies, am I right?
e

enough-analyst-54434

12/17/2022, 3:53 AM
Correct. Pants does not have a dependency inference parser for ipynb files yet, only for .py files.
👌 1
5 Views