https://pantsbuild.org/ logo
#general
Title
# general
a

abundant-leather-17386

08/09/2022, 8:59 AM
Is there a way to run shell commands easily with pants while using a virtual environment defined by a
poetry_requirements
target? Normally I would use
poetry run <shell command>
, but I would like to start using pants for this of course.
r

refined-addition-53644

08/09/2022, 9:39 AM
a

abundant-leather-17386

08/09/2022, 11:02 AM
I’ve tried to use this, but I can’t get it to work, as I would keep getting
ModuleNotFoundError
s.
e

enough-analyst-54434

08/09/2022, 11:06 AM
What do you mean by
<shell command>
? Do you mean a script / console script?: https://python-poetry.org/docs/cli/#run I think, unfortunately, no. The closest we come is with the venv we construct on the fly for the
repl
goal in Python:
Copy code
$ ./pants repl 3rdparty/python/::
Python 3.7.13 (default, May 29 2022, 10:31:40) 
[GCC 12.1.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import subprocess
>>> subprocess.run(["pytest", "--version"])
pytest 7.0.1
CompletedProcess(args=['pytest', '--version'], returncode=0)
>>> 
now exiting InteractiveConsole...
But there is no option to change the entry point of
repl
to
pytest
in this case. Of course there is the obvious
./pants run
- but that only works for
pex_binary
targets for Python and not for arbitrary sets of dependencies:
Copy code
$ ./pants run 3rdparty/python/::
03:59:20.17 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.run.run
  in pants.engine.internals.specs_rules.find_valid_field_sets_for_target_roots
Traceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/pants/src/python/pants/engine/internals/selectors.py", line 653, in native_engine_generator_send
    res = func.send(arg)
  File "/home/jsirois/dev/pantsbuild/pants/src/python/pants/engine/internals/specs_rules.py", line 536, in find_valid_field_sets_for_target_roots
    raise no_applicable_exception
pants.engine.internals.specs_rules.NoApplicableTargetsException: No applicable files or targets matched. The `run` goal works with these target types:

  * deploy_jar
  * docker_image
  * experimental_run_shell_command
  * go_binary
  * pex_binary
  * pyoxidizer_binary
  * python_source
  * python_test

However, you only specified glob arguments with these target types:

  * _generator_sources_helper
  * python_requirement

Please specify relevant file and/or target arguments. Run `./pants --filter-target-type=deploy_jar,docker_image,experimental_run_shell_command,go_binary,pex_binary,pyoxidizer_binary,python_source,python_test list ::` to find all applicable targets in your project, or run `./pants --filter-target-type=deploy_jar,docker_image,experimental_run_shell_command,go_binary,pex_binary,pyoxidizer_binary,python_source,python_test filedeps ::` to find all applicable files.
So all we can offer today is not very useful - it's slow, requires 2 steps and produces venvs you don't care about:
Copy code
$ ./pants export 3rdparty/python/::
Wrote virtualenv for the resolve 'python-default' (using Python 3.7.13) to dist/export/python/virtualenvs/python-default
Wrote virtualenv for the tool 'pytest' to dist/export/python/virtualenvs/tools
Wrote virtualenv for the tool 'autoflake' to dist/export/python/virtualenvs/tools
Wrote virtualenv for the tool 'black' to dist/export/python/virtualenvs/tools
Wrote virtualenv for the tool 'docformatter' to dist/export/python/virtualenvs/tools
Wrote virtualenv for the tool 'flake8' to dist/export/python/virtualenvs/tools
Wrote virtualenv for the tool 'isort' to dist/export/python/virtualenvs/tools
Wrote virtualenv for the tool 'mypy' to dist/export/python/virtualenvs/tools
$ dist/export/python/virtualenvs/python-default/3.7.13/bin/pytest --version
pytest 7.0.1
I can't think of a reason why we couldn't support
./pants run <requirement targets> -- pytest
and I don't think we have a feature request issue for this yet.
a

abundant-leather-17386

08/09/2022, 11:23 AM
Ok, I’ll share some more details. We deploy resources in AWS using CDK in Python. We use commands such as
poetry run cdk diff
and
poetry run cdk deploy --require-approval never --all
in CI to deploy our resources. If I want to use pants I indeed need to use
./pants export
first, then activate the correct virtual environment and then do
cdk diff
, which is way slower.
But your suggestion for
./pants run <requirement targets> -- pytest
would indeed be an interesting feature if that’s possible. Is there anything I can do to help?
e

enough-analyst-54434

08/09/2022, 11:41 AM
Filing the feature request is the start, but before that I'd like to narrow in on your stated case. For that, you could (I'm using pytest here since Pants own depedencies include that but not the AWS CDK):
Copy code
$ grep pytest 3rdparty/python/requirements.txt 
# This should be compatible with pytest.py, although it can be looser so that we don't
#  E.g., as default_version = "pytest>=6.2.4,<8,!=7.1.0,!=7.1.1"
pytest>=6.2.4,<7.1.0
$ git diff
diff --git a/3rdparty/python/BUILD b/3rdparty/python/BUILD
index 9803dba97..9fea7e535 100644
--- a/3rdparty/python/BUILD
+++ b/3rdparty/python/BUILD
@@ -15,3 +15,11 @@ python_requirements(
 # Useful when using IntelliJ/PyCharm remote debugging. Importing `pydevd_pycharm` at
 # the breakpoint will cause dep inference to add this dep on the remote debugger client.
 python_requirement(name="pydevd-pycharm", requirements=["pydevd-pycharm==203.5419.8"])
+
+pex_binary(
+    name="pytest-script",
+    script="pytest",
+    dependencies=[
+        "#pytest"
+    ]
+)
$ ./pants run 3rdparty/python:pytest-script -- --version
pytest 7.0.1
So, define a
pex_binary
target for the `cdk`and
./pants run
that.
So, still not ad-hoc friendly like
./poetry run
or the proposed
./pants run <reqs> -- <script> ...
but, after the 1 time step of defining the
pex_binary
target for the 3rdparty script, just as easy.
a

abundant-leather-17386

08/09/2022, 11:51 AM
Thanks for thinking along with me 🙂 Let me give it a go
cdk
is an executable that is on my
$PATH
instead of being a
python_requirement
somewhere. So I think that makes it a different case right? Not sure if I can apply your suggestion then right?
e

enough-analyst-54434

08/09/2022, 12:45 PM
Well, if `poetry run cdk`works, that implies the
cdk
script come from some Python distribution / requirement. What requirements do you have that seem likely to provide that script? I've poked around but it is not this one: https://pypi.org/project/aws-cdk.core/
Ok, as I poke around it appears
cdk
is a js script (npm binary). Gotcha, I think.
So it must be the case the
cdk
javascript code looks for the 1st
python
on the path, and if that
python
is the Python in a venv that contains your code and "constructs", "stacks" or whatever AWS-CDK gadgets; then that's how this all works. Since
poetry run
activates a venv, that would explain what's going on. If I have that all right, what you need Pants to be ale to do is activate a venv and run arbitrary shell commands (as you said!) in that context - which basically just means with the venv bin dir 1st on the PATH.
a

abundant-leather-17386

08/09/2022, 12:57 PM
Yes, and then what would make it really cool if the
resolve
is also easily selected. With poetry you have to start navigating the filesystem to use
poetry run
effectively if you have multiple resolves
e

enough-analyst-54434

08/09/2022, 12:57 PM
If so, then, yeah, I think this will require `./pants run <requirements> -- <shell command>`as a new feature. There is no way I can think of to hack something similar using any of the tools Pants provides today.
Yeah, resolve is easily selected in Pants. In fact, you should throw away your ideas about this from Poetry. In Pants you can pick out just 1 thing from a resolve.
a

abundant-leather-17386

08/09/2022, 12:58 PM
when you write
<requirements>
, what do you mean by that? any requirements target?
e

enough-analyst-54434

08/09/2022, 12:58 PM
Yes.
a

abundant-leather-17386

08/09/2022, 12:59 PM
ah ok, cool!
e

enough-analyst-54434

08/09/2022, 12:59 PM
Try
./pants repl 3rdparty/python#ansicolors
or
./pants repl 3rdparty/python#ansicolors 3rdparty/python#requests
- in each case you get exactly what you asked for - no more and no less.
Of course these targets need to be adjusted for your repo, they are taken as examples from the Pants repo itself just now.
a

abundant-leather-17386

08/09/2022, 1:01 PM
Nice. I am not used to thinking about dependencies so granularly, so this is very cool 🙂
❤️ 1
Do you need anything from me for the feature request?
e

enough-analyst-54434

08/09/2022, 1:18 PM
Just filing one: https://github.com/pantsbuild/pants/issues/new?assignees=&amp;labels=enhancement&amp;template=feature_request.md&amp;title= It's always best to get these from a User in need and especially with details. It makes sure we're not too abstract, which we tend to be, and we meet the concrete need at the very least.
a

abundant-leather-17386

08/09/2022, 1:38 PM
Thanks for the help @enough-analyst-54434 🙏
h

hundreds-father-404

08/09/2022, 1:41 PM
Double checking if y'all considered this idiom of creating a
pex_binary
target and setting
script
? This is how you can create a Black binary, for example: https://www.pantsbuild.org/docs/python-package-goal#the-entry_point-and-script-fields That will work with the
run
goal
e

enough-analyst-54434

08/09/2022, 1:41 PM
It was buried in the details, but
cdk
is not Python
👍 1
Its an npm app on the PATH that calls a Python in a venv.
h

hundreds-father-404

08/09/2022, 1:42 PM
Got it. Thanks, I did miss that
e

enough-analyst-54434

08/09/2022, 1:42 PM
Also buried up above, I suggested exactly that with examples using `pytest`from the Pants repo before I dug to understand the AWS-CDK and read.
👍 1
4 Views