https://pantsbuild.org/ logo
Join Slack
Powered by
I’m getting an ugly `UnicodeEncodeError` when runn...
# general
p
I’m getting an ugly 
UnicodeEncodeError
when running 
./pants dependencies --type=3rdparty ::
Copy code
Traceback (most recent call last):
  File "./__parse_python_imports.py", line 113, in <module>
    print("\n".join(sorted(string_imports)))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 38-40: ordinal not in range(128)
👀 1
Copy code
% ./pants dependencies --type=3rdparty :: --print-stacktrace
13:45:03.92 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 246, in run
    engine_result = self._perform_run(goals)
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 167, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 189, in _perform_run_body
    poll_delay=(0.1 if poll else None),
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/init/engine_initializer.py", line 131, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 568, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 532, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),

Exception message: 1 Exception encountered:

Engine traceback:
  in select
  in pants.backend.project_info.dependencies.dependencies
  in pants.engine.internals.graph.resolve_targets (st2api/tests/unit/controllers/v1:tests)
  in pants.engine.internals.graph.resolve_unexpanded_targets (st2api/tests/unit/controllers/v1:tests)
  in pants.engine.internals.graph.resolve_dependencies (st2api/tests/unit/controllers/v1:tests)
  in pants.backend.python.dependency_inference.rules.infer_python_dependencies_via_imports (st2api/tests/unit/controllers/v1:tests)
  in pants.backend.python.dependency_inference.import_parser.parse_python_imports
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/jafloyd/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 267, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Determine Python imports for st2api/tests/unit/controllers/v1:tests' failed with exit code 1.
stdout:
[snip]
Copy code
--

stderr:
Traceback (most recent call last):
  File "./__parse_python_imports.py", line 113, in <module>
    print("\n".join(sorted(string_imports)))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 38-40: ordinal not in range(128)
h
Oof, I had hoped we were done with those after the Py3 migration. Thank you for taking the time to report Are you able to share the files in 
st2api/tests/unit/controllers/v1:tests
with us, i.e. all the test files in that folder? Totally fine if over DM
p
Here’s the branch I’m working on: https://github.com/st2sandbox/st2/tree/pants
🙌 1
I’m also working on a Mac (OS X 10.15 Catalina), with python installed using MacPorts.
👍 1
h
@happy-kitchen-89482 I got a minimal reproduction. In pantsbuild/pants, apply this diff:
Copy code
diff --git a/testprojects/src/python/hello/main/BUILD b/testprojects/src/python/hello/main/BUILD
index c83f898cc..f0e17d2f2 100644
--- a/testprojects/src/python/hello/main/BUILD
+++ b/testprojects/src/python/hello/main/BUILD
@@ -1,5 +1,5 @@
 # Copyright 2020 Pants project contributors (see CONTRIBUTORS.md).
 # Licensed under the Apache License, Version 2.0 (see LICENSE).

-python_library(name="lib")
+python_library(name="lib", interpreter_constraints=['==3.6.*'])
 pex_binary(name="main", entry_point='main.py')
diff --git a/testprojects/src/python/hello/main/main.py b/testprojects/src/python/hello/main/main.py
index 0b569ea48..ba528a21b 100644
--- a/testprojects/src/python/hello/main/main.py
+++ b/testprojects/src/python/hello/main/main.py
@@ -9,3 +9,7 @@ if __name__ == "__main__":
     greetees = sys.argv[1:] or ["world"]
     for greetee in greetees:
         print(greet(greetee))
+
+"dummy_pack_1.st2.dummy.action_unicode_我爱狗"
+"st2.dummy.action_unicode_我爱狗"
+
Then run 
./pants dependencies testprojects/src/python/hello/main/main.py
. The weird thing is using 
--no-process-execution-local-cleanup
, then running the 
__run.sh
script, works fine. I suspect this has something to do with environment variables, I'm not sure how else the __run.sh script would diverge from Pants running the process
Oh also I couldn't get this to happen with Py37. @proud-dentist-22844 that could be a workaround while we fix this, set 
interpreter_constraints=['==3.7.*']
on the 
python_tests()
target so that Py36 isn't used to parse the imports Pardon the bug and thank you for the report!
h
I'll poke at it, thanks @hundreds-father-404
So this is due to non-ascii in a requirement string?
h
Yeah it looks like it. The 
string_imports
code path, specifically trying to call 
"\n".join()
on the collection (see line # from Jacob's stack trace) Might be worth iterating on that collection and asserting 
isinstance(x, str)
p
Hmm. I’m explicitly using 3.6 because that is the only officially supported version for StackStorm… I’ll be changing a ton to get this working, and bumping the python version has a lot of ramifications that I’m not ready for.
h
Ah okay, no worries. To be clear, this is a bug and high priority. We'll cherry-pick it into prior releases
p
also, these are not python imports: 
dummy_pack_1.st2.dummy.action_unicode_我爱狗"
st2.dummy.action_unicode_我爱
Why are they “requirement strings”?
h
There's an option called 
[python-infer].string_imports
, where we attempt to infer deps on strings that look like imports, e.g. 
importlib.import_module("<http://path.to|path.to>.module")
. It's off by default, but can be useful in repos like Django ones
p
Hmm. So, it’s collecting the possible string_imports even though that option is off?
h
Exactly, but yeah that could be a good optimization to make that parsing more intelligent
p
Hmm. How is it getting the 3.6 interpreter_constraints now?
I tried building the venv with 3.7, but that didn’t make a difference (ie I changed this line: https://github.com/st2sandbox/st2/blob/pants/scripts/generate_constraints.sh#L6 )
h
Based on your repo's 
[python-setup].interpreter_constraints
option, which sets the default for every Python target. You can then override it by setting 
interpreter_constraints
field on individual targets. https://www.pantsbuild.org/docs/python-interpreter-compatibility But, you're right that the workaround could cause issues if your repo is very strictly Py36. If you set those specific tests to be ==3.7, all the code they depend on would need to be compatible with 3.7 too, e.g. 
>=3.6,<3.8
. Which it sounds like you do not want to do yet. Checkout this tip to see what would need to be updated by activating this backend and then running 
./pants py-constraints st2api/tests/unit/controllers/v1:tests
(To reiterate, I'm not suggesting you will need to make this change to work with Pants. This is a bug that we will fix w/ high priority. Only offering a temporary workaround while we fix it, which may be too much work to justify the workaround, not sure)
p
Well, I can at least play with the 3.6 vs 3.7 knob in this case. Weird that this is broken.
¯\_(ツ)_/¯
p
I made https://github.com/st2sandbox/st2/pull/6 bases of your branch.
the only issue I am seeing (when running 
pants lint ::
) is something to do with a schema json file
Copy code
File "/Users/asher/.cache/pants/named_caches/pex_root/venvs/short/2149e2c0/lib/python3.8/site-packages/astroid/transforms.py", line 57, in _visit
    return self._transform(node)
  File "/Users/asher/.cache/pants/named_caches/pex_root/venvs/short/2149e2c0/lib/python3.8/site-packages/astroid/transforms.py", line 40, in _transform
    ret = transform_func(node)
  File "pylint_plugins/api_models.py", line 49, in transform
    module = __import__(module_name, fromlist=[class_name])
  File "/private/var/folders/hv/p6g7p3p95d19gtm5cfkrk5w00000gn/T/process-execution0GJVND/st2common/st2common/models/api/pack.py", line 25, in <module>
    from st2common.util import schema as util_schema
  File "/private/var/folders/hv/p6g7p3p95d19gtm5cfkrk5w00000gn/T/process-execution0GJVND/st2common/st2common/util/schema/__init__.py", line 53, in <module>
    "draft4": jsonify.load_file(os.path.join(PATH, "draft4.json")),
  File "/private/var/folders/hv/p6g7p3p95d19gtm5cfkrk5w00000gn/T/process-execution0GJVND/st2common/st2common/util/jsonify.py", line 140, in load_file
    with open(path, "r") as fd:
FileNotFoundError: [Errno 2] No such file or directory: '/private/var/folders/hv/p6g7p3p95d19gtm5cfkrk5w00000gn/T/process-execution0GJVND/st2common/st2common/util/schema/draft4.json'
could probably be fixed by adding some explicit dependencies somewhere... but I am not familiar with the code base to do that.
(I ran it with python 3.8)
i.e. I did 
eval "$(pyenv init -)" && pyenv global 3.8.6
to make sure the correct python is available to pants.
p
ah cool. Yeah, there are going to be several dependencies like that.
h
What's weird here is that there's a regex that is supposed to only match on ASCII strings that look like imports, so I'm not sure how these non-ascii strings are even being picked up
But I can repro that unexpected match in a repl
Ah, this is because of the 
\w*
part
OK so that makes sense
To clarify the bug, the problem isn't the 
"\n".join()
part, that is joining unicode strings just fine. The issue is trying to 
print()
a unicode string without explicitly encoding it, so python tries to encode it with the default encoding, which is 
ascii
in some cases.
🙌 1
So the fix is to explicitly encode
p
nice fix
@polite-garden-50641 I added a few things based on your PR, but I still can’t get pylint to run. 😞
h
If you push a branch I can take a look at pylint issues
p
I commented and then ran off for a family thing. 🙂 I just pushed those changes a few minutes ago.
h
So, it’s collecting the possible string_imports even though that option is off?
Now fixed in 2.5, which gives an ever so slight perf boost: https://github.com/pantsbuild/pants/pull/11975
💯 1
5 Views
Open in Slack
PreviousNext
Powered by