https://pantsbuild.org/ logo
Join Slack
Powered by
Hello, I want to work with jupyter notebooks for s...
# general
a
Hello, I want to work with jupyter notebooks for some data science stuff in pants. I followed this work-around and it almost solves everything. However, how do I access local code that is shared? My folder structure looks like the following structure:
Copy code
src/
├─ notebooks/
├─ libraries/
   ├─ library1/
I would like to access 
library1
in my notebook. I tried to set source roots but that did not work out for me. Do you have any hints?
e
Can you please provide the output of 
pants roots
?
a
> pants roots
.
src
e
Should be:
Copy code
src
src/libraries
You want to 
import library1
- correct?
a
doing 
import libraries.library1...
is also fine
e
No, it shouldn't be. You should 1st know what your package structure should be / is intended to be. Once you have that settled, you just need to tell Pants the source roots.
Does that make sense?
a
I actually would prefer doing 
import libraries.library1
since that mimicks how it works elsewhere in the repo.
e
Ok, once you know what you intend the imports to be, the source roots should cut there. If that's what you have already, the issue is elsewhere.
👍 1
So, you're saying with roots of 
.
and 
src
you cannot 
import libraries.library1
?
a
Correct
Copy code
ModuleNotFoundError: No module named 'libraries'
Is the error I am getting
e
And from where? Within src/notebooks? Is your repo `pants tailor`'d?
👀 1
a
you mean executing 
pants tailor ::
right?
e
Correct - after source roots are correct.
a
should 
pants tailor ::
create any output at all? it does not for me. I might have wrecked it 😮‍💨
e
I suggest not being so coy for the remote support. Full output of commands (the command you ran plus all its output), is very very helpful always.
👍 1
a
> pants roots
.
src
> pants tailor ::
>
e
Thanks. But, 
git status
should show some BUILD files?
The 
tailor
goal creates 
BUILD
files in various directories - Pants needs these to function.
a
No, no BUILD files created but I already have BUILD files in the directories where I need them.
e
Ok - great.
Back to 
ModuleNotFoundError: No module named 'libraries'
- can you provide command line + full output for that?
a
Copy code
> import libraries
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[3], line 1
----> 1 import libraries

ModuleNotFoundError: No module named 'libraries'
e
Ah, right - but ... how did you get there? 
pants run
a notebook or 
pants package
a PEX that you run or ... ?
a
pants run //src/notebooks/lab.py:python-default_main -- --notebook-dir src/notebooks
I am following how it is done in this repo.
e
What does 
pants filedeps --transitive src/notebooks/lab.py
say?
a
> pants filedeps --transitive src/notebooks/lab.py
src/notebooks/BUILD
src/notebooks/lab.py
e
Ok - getting somewhere. That implies lab.py has 0 transitive dependencies (via 
import
statements) on libraries - is that correct?
a
yes, I think that is correct because lab.py is only the entry point. the dependencies are defined in the BUILD
e
Well, not correctly probably due to the evidence of the 
filedeps
output above. Can you share the BUILD?
a
jupyter_lab(name="python-default", requirements=[
"numpy==1.24.2",
"pandas==1.5.3",
"structlog==22.3.0",
"python-dateutil==2.8.2",
"jupyterlab==3.4.8"
])
e
Where does 
jupyter_lab
come from?
That is not a Pants thing IIRC.
a
It is this file
e
Aha - Ok, so you want support on a non-Pants thing.
You made that semi-clear I guess. I did not page that in.
I'll let others chime in then. I don't have enough time this am before I leave here to go read that repo's code.
a
I don't exactly know where pants starts and it ends. Sorry. This repo has been shared by "officials(?)" here on slack so I thought it was deemed as THE jupyter solution
Thanks John I appreciate your help!
e
Its a third-party provided plugin. So some Pants Slack folks have developed it, but not the core Pants maintainers. I am of the latter group and ignorant of this stuff I'm afraid.
👍 1
I'm pretty sure da-tubi is @bored-energy-25252.
r
I feel like this is more python/jupyter issue than pants. Pants is just providing a virtual env for jupyter to be run inside it. Are you able to import those local modules outside of pants by starting a jupyter server?
a
If I install in the same root folder jupyter via pip and then start it I can import the 
src
folder via 
import src
.
I also tried to use python_sources on the 
libraries
folder but I since this is outside the folder the BUILD file resides in, it tells me I am not allowed to include it.
(I am pretty sure that I missed to put a basic configuration somewhere. So it's probably neither a pants, nor python/jupyter issue but I am the issue.)
r
You don't need to use pants targets and hence BUILD files for the code you are writing if I understand correctly the purpose of this plugin. The plugin gives you just a virtual env generated from pants provided lockfile. Bit hackish way is to append 
src
to your sys.path at the top just before you import these libraries. I have had issues with importing such local libraries .
b
If you want to work with shared code, here is a sample repo: https://github.com/da-tubi/pants-mininal-lab
2 Views
Open in Slack
PreviousNext
Powered by