https://pantsbuild.org/ logo
Join Slack
Powered by
Is there a way to run shell commands easily with p...
# general
a
Is there a way to run shell commands easily with pants while using a virtual environment defined by a 
poetry_requirements
target? Normally I would use 
poetry run <shell command>
, but I would like to start using pants for this of course.
r
a
I’ve tried to use this, but I can’t get it to work, as I would keep getting 
ModuleNotFoundError
s.
e
What do you mean by 
<shell command>
? Do you mean a script / console script?: https://python-poetry.org/docs/cli/#run I think, unfortunately, no. The closest we come is with the venv we construct on the fly for the 
repl
goal in Python:
Copy code
$ ./pants repl 3rdparty/python/::
Python 3.7.13 (default, May 29 2022, 10:31:40) 
[GCC 12.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import subprocess
>>> subprocess.run(["pytest", "--version"])
pytest 7.0.1
CompletedProcess(args=['pytest', '--version'], returncode=0)
>>> 
now exiting InteractiveConsole...
But there is no option to change the entry point of 
repl
to 
pytest
in this case. Of course there is the obvious 
./pants run
- but that only works for 
pex_binary
targets for Python and not for arbitrary sets of dependencies:
Copy code
$ ./pants run 3rdparty/python/::
03:59:20.17 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.run.run
  in pants.engine.internals.specs_rules.find_valid_field_sets_for_target_roots
Traceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/pants/src/python/pants/engine/internals/selectors.py", line 653, in native_engine_generator_send
    res = func.send(arg)
  File "/home/jsirois/dev/pantsbuild/pants/src/python/pants/engine/internals/specs_rules.py", line 536, in find_valid_field_sets_for_target_roots
    raise no_applicable_exception
pants.engine.internals.specs_rules.NoApplicableTargetsException: No applicable files or targets matched. The `run` goal works with these target types:

  * deploy_jar
  * docker_image
  * experimental_run_shell_command
  * go_binary
  * pex_binary
  * pyoxidizer_binary
  * python_source
  * python_test

However, you only specified glob arguments with these target types:

  * _generator_sources_helper
  * python_requirement

Please specify relevant file and/or target arguments. Run `./pants --filter-target-type=deploy_jar,docker_image,experimental_run_shell_command,go_binary,pex_binary,pyoxidizer_binary,python_source,python_test list ::` to find all applicable targets in your project, or run `./pants --filter-target-type=deploy_jar,docker_image,experimental_run_shell_command,go_binary,pex_binary,pyoxidizer_binary,python_source,python_test filedeps ::` to find all applicable files.
So all we can offer today is not very useful - it's slow, requires 2 steps and produces venvs you don't care about:
Copy code
$ ./pants export 3rdparty/python/::
Wrote virtualenv for the resolve 'python-default' (using Python 3.7.13) to dist/export/python/virtualenvs/python-default
Wrote virtualenv for the tool 'pytest' to dist/export/python/virtualenvs/tools
Wrote virtualenv for the tool 'autoflake' to dist/export/python/virtualenvs/tools
Wrote virtualenv for the tool 'black' to dist/export/python/virtualenvs/tools
Wrote virtualenv for the tool 'docformatter' to dist/export/python/virtualenvs/tools
Wrote virtualenv for the tool 'flake8' to dist/export/python/virtualenvs/tools
Wrote virtualenv for the tool 'isort' to dist/export/python/virtualenvs/tools
Wrote virtualenv for the tool 'mypy' to dist/export/python/virtualenvs/tools
$ dist/export/python/virtualenvs/python-default/3.7.13/bin/pytest --version
pytest 7.0.1
I can't think of a reason why we couldn't support 
./pants run <requirement targets> -- pytest
and I don't think we have a feature request issue for this yet.
a
Ok, I’ll share some more details. We deploy resources in AWS using CDK in Python. We use commands such as 
poetry run cdk diff
and 
poetry run cdk deploy --require-approval never --all
in CI to deploy our resources. If I want to use pants I indeed need to use 
./pants export
first, then activate the correct virtual environment and then do 
cdk diff
, which is way slower.
But your suggestion for 
./pants run <requirement targets> -- pytest
would indeed be an interesting feature if that’s possible. Is there anything I can do to help?
e
Filing the feature request is the start, but before that I'd like to narrow in on your stated case. For that, you could (I'm using pytest here since Pants own depedencies include that but not the AWS CDK):
Copy code
$ grep pytest 3rdparty/python/requirements.txt 
# This should be compatible with pytest.py, although it can be looser so that we don't
#  E.g., as default_version = "pytest>=6.2.4,<8,!=7.1.0,!=7.1.1"
pytest>=6.2.4,<7.1.0
$ git diff
diff --git a/3rdparty/python/BUILD b/3rdparty/python/BUILD
index 9803dba97..9fea7e535 100644
--- a/3rdparty/python/BUILD
+++ b/3rdparty/python/BUILD
@@ -15,3 +15,11 @@ python_requirements(
 # Useful when using IntelliJ/PyCharm remote debugging. Importing `pydevd_pycharm` at
 # the breakpoint will cause dep inference to add this dep on the remote debugger client.
 python_requirement(name="pydevd-pycharm", requirements=["pydevd-pycharm==203.5419.8"])
+
+pex_binary(
+    name="pytest-script",
+    script="pytest",
+    dependencies=[
+        "#pytest"
+    ]
+)
$ ./pants run 3rdparty/python:pytest-script -- --version
pytest 7.0.1
So, define a 
pex_binary
target for the `cdk`and 
./pants run
that.
So, still not ad-hoc friendly like 
./poetry run
or the proposed 
./pants run <reqs> -- <script> ...
but, after the 1 time step of defining the 
pex_binary
target for the 3rdparty script, just as easy.
a
Thanks for thinking along with me 🙂 Let me give it a go
cdk
is an executable that is on my 
$PATH
instead of being a 
python_requirement
somewhere. So I think that makes it a different case right? Not sure if I can apply your suggestion then right?
e
Well, if `poetry run cdk`works, that implies the 
cdk
script come from some Python distribution / requirement. What requirements do you have that seem likely to provide that script? I've poked around but it is not this one: https://pypi.org/project/aws-cdk.core/
Ok, as I poke around it appears 
cdk
is a js script (npm binary). Gotcha, I think.
So it must be the case the 
cdk
javascript code looks for the 1st 
python
on the path, and if that 
python
is the Python in a venv that contains your code and "constructs", "stacks" or whatever AWS-CDK gadgets; then that's how this all works. Since 
poetry run
activates a venv, that would explain what's going on. If I have that all right, what you need Pants to be ale to do is activate a venv and run arbitrary shell commands (as you said!) in that context - which basically just means with the venv bin dir 1st on the PATH.
a
Yes, and then what would make it really cool if the 
resolve
is also easily selected. With poetry you have to start navigating the filesystem to use 
poetry run
effectively if you have multiple resolves
e
If so, then, yeah, I think this will require `./pants run <requirements> -- <shell command>`as a new feature. There is no way I can think of to hack something similar using any of the tools Pants provides today.
Yeah, resolve is easily selected in Pants. In fact, you should throw away your ideas about this from Poetry. In Pants you can pick out just 1 thing from a resolve.
a
when you write 
<requirements>
, what do you mean by that? any requirements target?
e
Yes.
a
ah ok, cool!
e
Try 
./pants repl 3rdparty/python#ansicolors
or 
./pants repl 3rdparty/python#ansicolors 3rdparty/python#requests
- in each case you get exactly what you asked for - no more and no less.
Of course these targets need to be adjusted for your repo, they are taken as examples from the Pants repo itself just now.
a
Nice. I am not used to thinking about dependencies so granularly, so this is very cool 🙂
❤️ 1
Do you need anything from me for the feature request?
e
Just filing one: https://github.com/pantsbuild/pants/issues/new?assignees=&amp;labels=enhancement&amp;template=feature_request.md&amp;title= It's always best to get these from a User in need and especially with details. It makes sure we're not too abstract, which we tend to be, and we meet the concrete need at the very least.
a
Thanks for the help @enough-analyst-54434 🙏
h
Double checking if y'all considered this idiom of creating a 
pex_binary
target and setting 
script
? This is how you can create a Black binary, for example: https://www.pantsbuild.org/docs/python-package-goal#the-entry_point-and-script-fields That will work with the 
run
goal
e
It was buried in the details, but 
cdk
is not Python
👍 1
Its an npm app on the PATH that calls a Python in a venv.
h
Got it. Thanks, I did miss that
e
Also buried up above, I suggested exactly that with examples using `pytest`from the Pants repo before I dug to understand the AWS-CDK and read.
👍 1
4 Views
Open in Slack
PreviousNext
Powered by