https://pantsbuild.org/ logo
Join Slack
Powered by
Hey there! :slightly_smiling_face: first of all th...
# general
p
Hey there! 🙂 first of all thanks for the amazing work with pants! i'm just getting started, i would like it to use it to handle a python monorepo. I am currently trying to create two libraries, each with their own set of dependencies, where one can import the other. You can see the repository here: https://github.com/pietrobolcato/pants-monorepo-test/ If I run: 
pants test kittl/libs/base/tests/
it works, because the library "base" doesn't depend on any other If I run: 
pants test kittl/libs/fancy/tests/
it gives me the error: 
ModuleNotFoundError: No module named 'kittl.libs.base'
because it doesn't have access to 
kitt.libs.base
but only to 
kittl.libs.fancy
How can I have 
fancy
to import 
base
, while keeping dependencies separated? Thanks so much! Really appreciate your help! 🙂
c
Hi! Excellent with a public repo! First off, what’s the motivation to have dedicated resolves for each lib? I think you’re making it more difficult than it needs to be. Simply having a single resolve with a single lockfile will do just fine as long as your different libs don’t require conflicting versions of the same 3rd party dependency, which seems unlikely if you want one to be able to import the other. Pants will keep track of which dependencies are actually used on a module level, so you don’t end up with all 3rd party deps every where, only the subset actually required.
another drive by observation, your 
files
target in 
kittl/libs
seems to include all sources, and pants usually complains about files having multiple owners, so not sure if that will cause issues as well.
p
Thanks so much for the answer @curved-television-6568 ! The rationale between different resolves was to be able to specify in an easier fashion dependencies for each package, but it's a good point! one will probably do 🙂
But I am still unable, to import 
base
from 
fancy
- any suggestions there?
👀 1
c
you may still have multiple 
requirements.txt
files contributing to a shared resolve, if you really want to. not sure what happens if you put duplicate entries in them though.. so might be a hassle to track common deps 😬
Copy code
❯ pants test ::
13:46:24.66 [INFO] waiting for pantsd to start...
13:46:25.09 [INFO] pantsd started
13:46:25.49 [INFO] Initializing scheduler...
13:46:29.00 [INFO] Scheduler initialized.
13:46:29.10 [WARN] Pants cannot infer owners for the following imports in the target kittl/libs/fancy/tests/test_fancy.py:

  * kittl.libs.base.adder2.add2 (line: 5)

These imports are not in the resolve used by the target (`fancy`), but they were present in other resolves:

  * kittl.libs.base.adder2.add2: 'base' from kittl/libs/base/adder2.py

If you do not expect an import to be inferrable, add `# pants: no-infer-dep` to the import line. Otherwise, see <https://www.pantsbuild.org/v2.17/docs/troubleshooting#import-errors-and-missing-dependencies> for common problems.
13:46:29.11 [WARN] Pants cannot infer owners for the following imports in the target kittl/libs/base/tests/conftest.py:../base:

  * pytest (line: 5)

These imports are not in the resolve used by the target (`base`), but they were present in other resolves:

  * pytest: 'python_default' from //:reqs#pytest

If you do not expect an import to be inferrable, add `# pants: no-infer-dep` to the import line. Otherwise, see <https://www.pantsbuild.org/v2.17/docs/troubleshooting#import-errors-and-missing-dependencies> for common problems.
13:46:29.11 [WARN] Pants cannot infer owners for the following imports in the target kittl/libs/fancy/tests/conftest.py:../fancy:

  * pytest (line: 5)

These imports are not in the resolve used by the target (`fancy`), but they were present in other resolves:

  * pytest: 'python_default' from //:reqs#pytest

If you do not expect an import to be inferrable, add `# pants: no-infer-dep` to the import line. Otherwise, see <https://www.pantsbuild.org/v2.17/docs/troubleshooting#import-errors-and-missing-dependencies> for common problems.
13:46:29.11 [WARN] Pants cannot infer owners for the following imports in the target kittl/libs/fancy/adder3.py:

  * kittl.libs.base.adder2.add2 (line: 3)

These imports are not in the resolve used by the target (`fancy`), but they were present in other resolves:

  * kittl.libs.base.adder2.add2: 'base' from kittl/libs/base/adder2.py

If you do not expect an import to be inferrable, add `# pants: no-infer-dep` to the import line. Otherwise, see <https://www.pantsbuild.org/v2.17/docs/troubleshooting#import-errors-and-missing-dependencies> for common problems.
13:46:35.75 [INFO] Canceled: Building pytest.pex from <resource://pants.backend.python.subsystems/pytest.lock>
13:46:36.98 [INFO] Completed: Building requirements.pex
13:46:46.92 [INFO] Completed: Building pytest.pex from <resource://pants.backend.python.subsystems/pytest.lock>
13:46:48.25 [INFO] Completed: Building local_dists.pex
13:46:52.63 [INFO] Completed: Building pytest_runner.pex
13:46:55.90 [INFO] Completed: Run Pytest - kittl/libs/base/tests/test_base.py - succeeded.
13:46:56.03 [ERROR] Completed: Run Pytest - kittl/libs/fancy/tests/test_fancy.py - failed (exit code 2).
============================= test session starts ==============================
platform darwin -- Python 3.11.1, pytest-7.0.1, pluggy-1.0.0
rootdir: /private/var/folders/y2/6t5bbk554wv5h6w_qlgpcpnm0000gr/T/pants-sandbox-IS7APv
plugins: xdist-2.5.0, forked-1.6.0, cov-3.0.0
collected 0 items / 1 error

==================================== ERRORS ====================================
____________ ERROR collecting kittl/libs/fancy/tests/test_fancy.py _____________
ImportError while importing test module '/private/var/folders/y2/6t5bbk554wv5h6w_qlgpcpnm0000gr/T/pants-sandbox-IS7APv/kittl/libs/fancy/tests/test_fancy.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/Users/andreas.stenius/.pyenv/versions/3.11.1/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
kittl/libs/fancy/tests/test_fancy.py:5: in <module>
    from kittl.libs.base.adder2 import add2
E   ModuleNotFoundError: No module named 'kittl.libs.base'
- generated xml file: /private/var/folders/y2/6t5bbk554wv5h6w_qlgpcpnm0000gr/T/pants-sandbox-IS7APv/kittl.libs.fancy.tests.test_fancy.py.xml -
=========================== short test summary info ============================
ERROR kittl/libs/fancy/tests/test_fancy.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.08s ===============================



✓ kittl/libs/base/tests/test_base.py succeeded in 3.21s.
✕ kittl/libs/fancy/tests/test_fancy.py failed in 3.34s.
errno:2
zsh  ~/src/github/pietrobolcato/pants-monorepo-test    main ?  13:46 
❯ pants generate-lockfiles
13:49:34.16 [INFO] Completed: Generate lockfile for base
13:49:48.06 [INFO] Completed: Generate lockfile for python_default
13:49:48.07 [INFO] Wrote lockfile for the resolve `base` to kittl/libs/base/base.lock
13:49:48.07 [INFO] Wrote lockfile for the resolve `fancy` to kittl/libs/fancy/fancy.lock
13:49:48.07 [INFO] Wrote lockfile for the resolve `python_default` to python_default.lock

Lockfile diff: kittl/libs/base/base.lock [base]

==                     Removed dependencies                     ==

  numpy                          1.25.2

Lockfile diff: python_default.lock [python_default]

==                      Added dependencies                      ==

  numpy                          1.25.2



zsh  ~/src/github/pietrobolcato/pants-monorepo-test    main !?  13:49 
❯ pants test ::
13:49:56.13 [INFO] Canceled: Building 1 requirement for requirements.pex from the kittl/libs/base/base.lock resolve: pytest~=7.1.3
13:49:56.13 [INFO] Canceled: Building 1 requirement for requirements.pex from the kittl/libs/fancy/fancy.lock resolve: pytest~=7.1.3
13:49:59.27 [INFO] Completed: Building 1 requirement for requirements.pex from the python_default.lock resolve: pytest~=7.1.3
13:50:02.82 [INFO] Completed: Building pytest_runner.pex
13:50:05.81 [INFO] Completed: Run Pytest - kittl/libs/base/tests/test_base.py - succeeded.
13:50:05.90 [INFO] Completed: Run Pytest - kittl/libs/fancy/tests/test_fancy.py - succeeded.

✓ kittl/libs/base/tests/test_base.py succeeded in 2.98s.
✓ kittl/libs/fancy/tests/test_fancy.py succeeded in 3.06s.

zsh  ~/src/github/pietrobolcato/pants-monorepo-test    main !?  13:50 
❯
Copy code
diff --git a/kittl/libs/base/BUILD b/kittl/libs/base/BUILD
index b955ff8..8012b86 100644
--- a/kittl/libs/base/BUILD
+++ b/kittl/libs/base/BUILD
@@ -1,10 +1,10 @@
 python_sources(
-    resolve="base",
+    # resolve="base",
     sources=["**/*.py"],
 )
 
 python_requirements(
     name="reqs",
     source="requirements.txt",
-    resolve="base",
-)
\ No newline at end of file
+    # resolve="base",
+)
diff --git a/kittl/libs/base/tests/BUILD b/kittl/libs/base/tests/BUILD
index 4c25759..ad53575 100644
--- a/kittl/libs/base/tests/BUILD
+++ b/kittl/libs/base/tests/BUILD
@@ -1,5 +1,5 @@
 python_tests(
     name="tests",
-    resolve="base",
+    # resolve="base",
     sources=["test_*.py"],
 )
diff --git a/kittl/libs/fancy/BUILD b/kittl/libs/fancy/BUILD
index 7eb7647..dc81270 100644
--- a/kittl/libs/fancy/BUILD
+++ b/kittl/libs/fancy/BUILD
@@ -1,10 +1,10 @@
 python_sources(
-    resolve="fancy",
+    # resolve="fancy",
     sources=["**/*.py"],
 )
 
 python_requirements(
     name="reqs",
     source="requirements.txt",
-    resolve="fancy",
-)
\ No newline at end of file
+    # resolve="fancy",
+)
diff --git a/kittl/libs/fancy/tests/BUILD b/kittl/libs/fancy/tests/BUILD
index 350d228..c5e8085 100644
--- a/kittl/libs/fancy/tests/BUILD
+++ b/kittl/libs/fancy/tests/BUILD
@@ -1,5 +1,5 @@
 python_tests(
     name="tests",
-    resolve="fancy",
+    # resolve="fancy",
     sources=["test_*.py"],
 )
also, I don’t think your 
conftest.py
files will be included in the test runs…
p
You're great @curved-television-6568 this works, thanks so much! 🙂 and good point for 
conftest.py!
🙏 1
Just a general question for me to understand in depth, why having the two resolves would make it not working whereas not having them it does?
c
you need to treat each resolve as completely isolated from all other resolves. So in order for it to work the way you had it, you’d need to have one of them be part of two resolves, in order to be able to “interact”.
which you can do, using 
parametrize
also realize that my comment earlier regarding multiple owners doesn’t apply in this case, as it was a 
files
target, not a 
python_sources
one, so that’s fine.
although I’m not sure of the purpose of it in this case… 😉
p
makes a lot of sense! thanks a lot, very much appreciated @curved-television-6568 🙏
👍 1
@curved-television-6568 so sorry to bother again, I have another question: I have a bash script that runs "chalice local" and another one that run "chalice deploy" to start and deploy an aws lambda function. I tried to create a BUILD file in the same directory where the bash script is: 
shell_command(
command="bash run_chalice_local.sh",
tools=["chalice"],
)
how can i run this through pants? appreciate your help!
thank you so much 🙂
c
To run your shell command as the primary goal, I think you want to use the 
run_shell_command
target. https://www.pantsbuild.org/docs/run-shell-commands#the-run_shell_command-target something like:
Copy code
run_shell_command(
  name="chalice",
  command="./run_chalice_local.sh",
  tools=["chalice"],
  execution_dependencies=[":run_chalice_local.sh"],
)
then:
Copy code
$ pants run ../dir:chalice -- <chalice script args>
🙏 1
editing…
p
Sounds great thanks so much! Super helpful appreciate it a lot 🙂 will try and let you know, thanks again!
c
you’re most welcome 🙂
p
Hey there dear @curved-television-6568 so sorry to bother again, the last step that is missing for us to adopt pants, would be to have a functioning jupyter notebook integration - we use them a lot to iterate over the code rather than static python files I found this plugin: https://github.com/pantsbuild/pants-jupyter-plugin But having some troubles setting it up. What I am trying to achieve is to basically load into the notebook the requirements for a specific resolve or combination of resolves (eg: how it's done with 
parametrize
) - or in other words it would be great to have the jup notebook that loads dependencies like pants run, so eg when I execute: 
pants run kittl/projects/my_project/main.py
I tried adding to my BUILD `pex_binary`:
Copy code
python_sources(
    name="my_project",
    resolve="my_project",
)

python_requirements(
    name="reqs",
    source="requirements.txt",
    resolve="my_project",
)

pex_binary(
    name="my_project_bin"
)
And loading with the magic: 
%pants_load kittl/projects/my_project
But it fails because it can't find the right version of python in the path of the notebook, specifically it's not looking at the 
pyenv root
folder Would you have any suggestion to fix this? Thanks so much once again! Really appreciate your help! 🙂
❤️ 1
c
That’s a new topic, so would suggest starting a new thread for it, to get more 👀 on it 😉 Do we have any jupyter experienced folks in here ? ( I know there’s been some discussions about jupyter in the past..)
🙏 1
b
I'll chime in on new thread 😉
👍 2
p
Thank you so much @bitter-ability-32190! create a new thread here: https://pantsbuild.slack.com/archives/C046T6T9U/p1694615600674339
@curved-television-6568 regarding the point above of 
run_shell_command
it works in itself so thanks so much for that! but unfortunately it doesn't do what we need - we are trying to run "chalice local" and "chalice deploy" (docs) to build and deploy a lambda function These commands are based on python and need to access all the dependencies of the resolve, exactly as if i was running pytest in a sense Using run 
run_shell_command
with chalice doesn't work because dependencies are not injected - does any solution come to mind by any chance? Appreciate it a lot! 🙏🙏
c
will have to defer to those more well versed on these targets 😉 cc @ancient-vegetable-10556 @broad-processor-92400 @bitter-ability-32190 maybe..?
❤️ 1
🙏 1
b
Can you push a reproduction repo with the minimum of what you got? I'm a visual person, so that helps me immensely 😛
a
What do you mean "based on python"?
Can you invoke it with 
python -m
?
p
@bitter-ability-32190 totally make sense! @ancient-vegetable-10556 nope unfortunately, will make a repo so that you can see what I mean easily 🙂
@bitter-ability-32190 @ancient-vegetable-10556 created the repo here: https://github.com/pietrobolcato/pants-chalice/ There's no pants on it to show the most minimal example possible - let me know if this works for you, or if I can do anything else to make this more clear 🙂
Thank you so much, you guys are great! Really appreciate the support 🙏
b
Can you add enough Pants to it to get to the failure point? (Also I may not be the one helping you fully, I'm not fully back from being OoO until Monday)
👍 1
p
@bitter-ability-32190 sure! I added pants until my failure point, hope this helps! If not let me know and happy to iterate further! (and enjoy your ooo! 🙂)
3 Views
Open in Slack
PreviousNext
Powered by