https://pantsbuild.org/ logo
Join Slack
Powered by
I'm experiencing an error determining python impor...
# general
r
I'm experiencing an error determining python imports after setting 
interpreter_constraints
to 
CPython==2.7.*
(this because our project uses a mix of 2.7 and 3.6)
Copy code
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa0 in position 0: invalid start byte
Pants determines imports fine when I omit 
interpreter_constraints
from 
pants.toml
. Any ideas why this would occur?
Our goal is to test python 2.7 code using pants. This is a basic setup after using 
tailor
command. I may have some misconfiguration but not sure... Full stack trace:
Copy code
./pants dependees --changed-since=origin/master
15:43:57.93 [INFO] initializing pantsd...
15:43:58.27 [INFO] pantsd initialized.
15:45:08.34 [ERROR] Exception caught: (pants.engine.internals.scheduler.ExecutionError)
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 246, in run
    engine_result = self._perform_run(goals)
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 167, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 189, in _perform_run_body
    poll_delay=(0.1 if poll else None),
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/init/engine_initializer.py", line 131, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 568, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 532, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),

Exception message: 1 Exception encountered:

Engine traceback:
  in select
  in pants.backend.project_info.dependees.dependees_goal
  in pants.backend.project_info.dependees.find_dependees
  in pants.backend.project_info.dependees.map_addresses_to_dependees
  in pants.engine.internals.graph.resolve_dependencies (app/payroll/imports/parsers)
  in pants.backend.python.dependency_inference.rules.infer_python_dependencies_via_imports (app/payroll/imports/parsers)
  in pants.backend.python.dependency_inference.import_parser.parse_python_imports
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 267, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Determine Python imports for app/payroll/imports/parsers' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "./__parse_python_imports.py", line 106, in <module>
    visitor.visit(tree)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 251, in generic_visit
    self.visit(value)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 251, in generic_visit
    self.visit(value)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 251, in generic_visit
    self.visit(value)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "./__parse_python_imports.py", line 61, in visit_Call
    self.generic_visit(node)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "./__parse_python_imports.py", line 67, in visit_Str
    val = node.s.decode("utf-8") if isinstance(node.s, bytes) else node.s
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa0 in position 0: invalid start byte


Traceback (most recent call last):
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 246, in run
    engine_result = self._perform_run(goals)
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 167, in _perform_run
    return self._perform_run_body(goals, poll=False)
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/bin/local_pants_runner.py", line 189, in _perform_run_body
    poll_delay=(0.1 if poll else None),
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/init/engine_initializer.py", line 131, in run_goal_rules
    goal_product, params, poll=poll, poll_delay=poll_delay
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 568, in run_goal_rule
    self._raise_on_error([t for _, t in throws])
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/engine/internals/scheduler.py", line 532, in _raise_on_error
    wrapped_exceptions=tuple(t.exc for t in throws),
pants.engine.internals.scheduler.ExecutionError: 1 Exception encountered:

Engine traceback:
  in select
  in pants.backend.project_info.dependees.dependees_goal
  in pants.backend.project_info.dependees.find_dependees
  in pants.backend.project_info.dependees.map_addresses_to_dependees
  in pants.engine.internals.graph.resolve_dependencies (app/payroll/imports/parsers)
  in pants.backend.python.dependency_inference.rules.infer_python_dependencies_via_imports (app/payroll/imports/parsers)
  in pants.backend.python.dependency_inference.import_parser.parse_python_imports
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/njgrisafi/.cache/pants/setup/bootstrap-Darwin-x86_64/2.3.0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 267, in fallible_to_exec_result_or_raise
    description.value,
pants.engine.process.ProcessExecutionFailure: Process 'Determine Python imports for app/payroll/imports/parsers' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "./__parse_python_imports.py", line 106, in <module>
    visitor.visit(tree)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 251, in generic_visit
    self.visit(value)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 251, in generic_visit
    self.visit(value)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 251, in generic_visit
    self.visit(value)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "./__parse_python_imports.py", line 61, in visit_Call
    self.generic_visit(node)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "./__parse_python_imports.py", line 67, in visit_Str
    val = node.s.decode("utf-8") if isinstance(node.s, bytes) else node.s
  File "/Users/njgrisafi/.pyenv/versions/2.7.18/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa0 in position 0: invalid start byte
h
Huh, cc @happy-kitchen-89482. Relates to an issue Jacob experienced today too: https://pantsbuild.slack.com/archives/C046T6T9U/p1617990831398500 These are both very high priority bugs for us to fix. Are you able to share the contents of the folder 
app/payroll/imports/parsers
? Totally okay if it's over DM - it's really helpful for us to come up with a minimal repo of where the UTF is failing
h
Hmm that is bad, I will fix this today
🙏 1
r
discussed with @hundreds-father-404 in PM and we found it was failing due to this line in a module file 
xyz = [[y.replace('\xa0', '') for y in x] for x in csv_stream if x]
. Removing 
\xa0
fixed the issue for the target
h
Thanks @rapid-bird-79300! What Pants version are you? We'll cherry-pick probably back to 2.2, but want to make sure you're covered
r
No problem at all and I'm using version 
2.3.0
👍 1
h
This should fix it, assuming it's the same issue (which is a reasonable assumption): https://github.com/pantsbuild/pants/pull/11879
✅ 1
r
Just an update. I tested using 
2.3.1rc2
and still seeing the same issue here:
Copy code
stderr:
Traceback (most recent call last):
  File "./__parse_python_imports.py", line 106, in <module>
    visitor.visit(tree)
  File "/usr/local/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/usr/local/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/usr/local/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/usr/local/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/ast.py", line 251, in generic_visit
    self.visit(value)
  File "/usr/local/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/ast.py", line 251, in generic_visit
    self.visit(value)
  File "/usr/local/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/usr/local/lib/python2.7/ast.py", line 251, in generic_visit
    self.visit(value)
  File "/usr/local/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "./__parse_python_imports.py", line 61, in visit_Call
    self.generic_visit(node)
  File "/usr/local/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/usr/local/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "./__parse_python_imports.py", line 67, in visit_Str
    val = node.s.decode("utf-8") if isinstance(node.s, bytes) else node.s
  File "/usr/local/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa0 in position 0: invalid start byte


ripplinguser@765a5912d2f0:~/rippling-main-local$ ./pants --version
2.3.1rc2
👀 1
h
Oh @happy-kitchen-89482 note that this is the breaking line:
val = node.s.decode("utf-8") if isinstance(node.s, bytes) else node.s
not the printing line
bump that this is the minimal repro:
we found it was failing due to this line in a module file xyz = [[y.replace('\xa0', '') for y in x] for x in csv_stream if x]. Removing \xa0 fixed the issue for the target
r
Some background on our use case: We are running 
./pants test app/tests:tests0 -- --collect-only
to get all pytest targets. The project interpreter is set to 2.7.17. Our current work around for this error is to remove 
interpreter_constraints
and run 
./pants dependees --changed-since=origin/master --python-infer-string-imports --dependees-output-file=pants-dependees.txt
to get all changed file dependees. Then we parse the output in separate script and and add the test files to the pytest command. This seems to work, however, I think it would be ideal to have pants run pytest directly.
h
For sure, will fix this asap
@hundreds-father-404 This is a different error than the previous one, which was on the print line, and happened in python3
@rapid-bird-79300 are your source files encoded in something other than utf8?
r
@happy-kitchen-89482 I don't believe so. We have some occurrences where we parse unicode streams and replace for some characters. I can create a repo for this tomorrow as a working example.
h
Thanks, that would be super helpful
I have a handle on this, but would be good to verify the fix against an example
The code that is failing is looking at literal strings in your source files
so I'm not sure parsing unicode streams etc. is pertinent
r
Yes that looks like it should do it!
@happy-kitchen-89482 here's that example repo I promised: https://github.com/njgrisafi/pants-example You can run 
./pants dependees ::
to see the unicode error
🙌 1
h
Thanks! Will check that my fix works
👍 1
Confirmed!
🚀 1
🙌 2
Will cherry-pick
8 Views
Open in Slack
PreviousNext
Powered by