I’m getting an ugly `UnicodeEncodeError` when runn...
# general
I’m getting an ugly
when running
./pants dependencies --type=3rdparty ::
Copy code
Traceback (most recent call last):
  File "./__parse_python_imports.py", line 113, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 38-40: ordinal not in range(128)
👀 1
Copy code
% ./pants dependencies --type=3rdparty :: --print-stacktrace
13:45:03.92 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 246, in run
    engine_result = self._perform_run(goals)
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 167, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 189, in _perform_run_body
    poll_delay=(0.1 if poll else None),
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/init/engine_initializer.py", line 131, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 568, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 532, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),

Exception message: 1 Exception encountered:

Engine traceback:
  in select
  in pants.backend.project_info.dependencies.dependencies
  in pants.engine.internals.graph.resolve_targets (st2api/tests/unit/controllers/v1:tests)
  in pants.engine.internals.graph.resolve_unexpanded_targets (st2api/tests/unit/controllers/v1:tests)
  in pants.engine.internals.graph.resolve_dependencies (st2api/tests/unit/controllers/v1:tests)
  in pants.backend.python.dependency_inference.rules.infer_python_dependencies_via_imports (st2api/tests/unit/controllers/v1:tests)
  in pants.backend.python.dependency_inference.import_parser.parse_python_imports
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 267, in fallible_to_exec_result_or_raise
pants.engine.process.ProcessExecutionFailure: Process 'Determine Python imports for st2api/tests/unit/controllers/v1:tests' failed with exit code 1.
Copy code

Traceback (most recent call last):
  File "./__parse_python_imports.py", line 113, in <module>
UnicodeEncodeError: 'ascii' codec can't encode characters in position 38-40: ordinal not in range(128)
Oof, I had hoped we were done with those after the Py3 migration. Thank you for taking the time to report Are you able to share the files in
with us, i.e. all the test files in that folder? Totally fine if over DM
Here’s the branch I’m working on: https://github.com/st2sandbox/st2/tree/pants
🙌 1
I’m also working on a Mac (OS X 10.15 Catalina), with python installed using MacPorts.
👍 1
@happy-kitchen-89482 I got a minimal reproduction. In pantsbuild/pants, apply this diff:
Copy code
diff --git a/testprojects/src/python/hello/main/BUILD b/testprojects/src/python/hello/main/BUILD
index c83f898cc..f0e17d2f2 100644
--- a/testprojects/src/python/hello/main/BUILD
+++ b/testprojects/src/python/hello/main/BUILD
@@ -1,5 +1,5 @@
 # Copyright 2020 Pants project contributors (see CONTRIBUTORS.md).
 # Licensed under the Apache License, Version 2.0 (see LICENSE).

+python_library(name="lib", interpreter_constraints=['==3.6.*'])
 pex_binary(name="main", entry_point='main.py')
diff --git a/testprojects/src/python/hello/main/main.py b/testprojects/src/python/hello/main/main.py
index 0b569ea48..ba528a21b 100644
--- a/testprojects/src/python/hello/main/main.py
+++ b/testprojects/src/python/hello/main/main.py
@@ -9,3 +9,7 @@ if __name__ == "__main__":
     greetees = sys.argv[1:] or ["world"]
     for greetee in greetees:
Then run
./pants dependencies testprojects/src/python/hello/main/main.py
. The weird thing is using
, then running the
script, works fine. I suspect this has something to do with environment variables, I'm not sure how else the __run.sh script would diverge from Pants running the process
Oh also I couldn't get this to happen with Py37. @proud-dentist-22844 that could be a workaround while we fix this, set
on the
target so that Py36 isn't used to parse the imports Pardon the bug and thank you for the report!
I'll poke at it, thanks @hundreds-father-404
So this is due to non-ascii in a requirement string?
Yeah it looks like it. The
code path, specifically trying to call
on the collection (see line # from Jacob's stack trace) Might be worth iterating on that collection and asserting
isinstance(x, str)
Hmm. I’m explicitly using 3.6 because that is the only officially supported version for StackStorm… I’ll be changing a ton to get this working, and bumping the python version has a lot of ramifications that I’m not ready for.
Ah okay, no worries. To be clear, this is a bug and high priority. We'll cherry-pick it into prior releases
also, these are not python imports:
Why are they “requirement strings”?
There's an option called
, where we attempt to infer deps on strings that look like imports, e.g.
. It's off by default, but can be useful in repos like Django ones
Hmm. So, it’s collecting the possible string_imports even though that option is off?
Exactly, but yeah that could be a good optimization to make that parsing more intelligent
Hmm. How is it getting the 3.6 interpreter_constraints now?
I tried building the venv with 3.7, but that didn’t make a difference (ie I changed this line: https://github.com/st2sandbox/st2/blob/pants/scripts/generate_constraints.sh#L6 )
Based on your repo's
option, which sets the default for every Python target. You can then override it by setting
field on individual targets. https://www.pantsbuild.org/docs/python-interpreter-compatibility But, you're right that the workaround could cause issues if your repo is very strictly Py36. If you set those specific tests to be ==3.7, all the code they depend on would need to be compatible with 3.7 too, e.g.
. Which it sounds like you do not want to do yet. Checkout this tip to see what would need to be updated by activating this backend and then running
./pants py-constraints st2api/tests/unit/controllers/v1:tests
(To reiterate, I'm not suggesting you will need to make this change to work with Pants. This is a bug that we will fix w/ high priority. Only offering a temporary workaround while we fix it, which may be too much work to justify the workaround, not sure)
Well, I can at least play with the 3.6 vs 3.7 knob in this case. Weird that this is broken.
I made https://github.com/st2sandbox/st2/pull/6 bases of your branch.
the only issue I am seeing (when running
pants lint ::
) is something to do with a schema json file
Copy code
File "/Users/asher/.cache/pants/named_caches/pex_root/venvs/short/2149e2c0/lib/python3.8/site-packages/astroid/transforms.py", line 57, in _visit
    return self._transform(node)
  File "/Users/asher/.cache/pants/named_caches/pex_root/venvs/short/2149e2c0/lib/python3.8/site-packages/astroid/transforms.py", line 40, in _transform
    ret = transform_func(node)
  File "pylint_plugins/api_models.py", line 49, in transform
    module = __import__(module_name, fromlist=[class_name])
  File "/private/var/folders/hv/p6g7p3p95d19gtm5cfkrk5w00000gn/T/process-execution0GJVND/st2common/st2common/models/api/pack.py", line 25, in <module>
    from st2common.util import schema as util_schema
  File "/private/var/folders/hv/p6g7p3p95d19gtm5cfkrk5w00000gn/T/process-execution0GJVND/st2common/st2common/util/schema/__init__.py", line 53, in <module>
    "draft4": jsonify.load_file(os.path.join(PATH, "draft4.json")),
  File "/private/var/folders/hv/p6g7p3p95d19gtm5cfkrk5w00000gn/T/process-execution0GJVND/st2common/st2common/util/jsonify.py", line 140, in load_file
    with open(path, "r") as fd:
FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/hv/p6g7p3p95d19gtm5cfkrk5w00000gn/T/process-execution0GJVND/st2common/st2common/util/schema/draft4.json'
could probably be fixed by adding some explicit dependencies somewhere... but I am not familiar with the code base to do that.
(I ran it with python 3.8)
i.e. I did
eval "$(pyenv init -)" && pyenv global 3.8.6
to make sure the correct python is available to pants.
ah cool. Yeah, there are going to be several dependencies like that.
What's weird here is that there's a regex that is supposed to only match on ASCII strings that look like imports, so I'm not sure how these non-ascii strings are even being picked up
But I can repro that unexpected match in a repl
Ah, this is because of the
OK so that makes sense
To clarify the bug, the problem isn't the
part, that is joining unicode strings just fine. The issue is trying to
a unicode string without explicitly encoding it, so python tries to encode it with the default encoding, which is
in some cases.
🙌 1
So the fix is to explicitly encode
nice fix
@polite-garden-50641 I added a few things based on your PR, but I still can’t get pylint to run. 😞
If you push a branch I can take a look at pylint issues
I commented and then ran off for a family thing. 🙂 I just pushed those changes a few minutes ago.
So, it’s collecting the possible string_imports even though that option is off?
Now fixed in 2.5, which gives an ever so slight perf boost: https://github.com/pantsbuild/pants/pull/11975
💯 1