https://pantsbuild.org/ logo
Powered by
d

dry-analyst-73584

01/21/2022, 5:54 PM
I set 
[python].interpreter_constraints
to CPython>=2.7,<3 and pants instructed me to generate a new lockfile for pytest. Running 
/v2pants generate-lockfiles --resolve=pytest
fails on what looks like a py2/3 compatibility error in parsing python imports.
Copy code
ProcessExecutionFailure: Process 'Determine Python imports for src/python/fsluigi/main.py' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "./__parse_python_imports.py", line 114, in <module>
    main(sys.argv[1])
  File "./__parse_python_imports.py", line 105, in main
    visitor.visit(tree)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 249, in generic_visit
    self.visit(item)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 251, in generic_visit
    self.visit(value)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ast.py", line 241, in visit
    return visitor(node)
  File "./__parse_python_imports.py", line 57, in visit_Call
    elif isinstance(node.args[0], ast.Constant):
AttributeError: 'module' object has no attribute 'Constant'
Pants itself still appears to be running python3.9, but maybe part of the import parsing is being affected by my new interpreter constraint?
h

hundreds-father-404

01/21/2022, 5:56 PM
What Pants version? Can you please send 
flsuigi/main.py
to us, even if over DM? I'm curious if we can narrow down what's causing the parse error and fix it. That's a critical bug
d

dry-analyst-73584

01/21/2022, 5:56 PM
pantsd command is: PANTS_DAEMON_ENTRYPOINT=pants.pantsd.pants_daemon:launch_new_pantsd_instance PYTHONPATH=/Users/tansy/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.KynvEQ/install/bin:/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python39.zip:/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9:/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload:/Users/tansy/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.KynvEQ/install/lib/python3.9/site-packages /Users/tansy/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.KynvEQ/install/bin/python /Users/tansy/.cache/pants/setup/bootstrap-Darwin-x86_64/2.9.0_py39/bin/pants --pants-bin-name=./v2pants --pants-version=2.9.0 --level=debug generate-lockfiles --resolve=pytest
I think the file it's failing on just happens to be the first one it runs, but I'll DM it to you. Do you need the whole thing or only the imports?
h

hundreds-father-404

01/21/2022, 5:57 PM
Wait do you expect this file to work with Py2.7?
Pants itself still appears to be running python3.9, but maybe part of the import parsing is being affected by my new interpreter constraint?
Yep, each file is parsed using the interpreter constraints you specify for that file
d

dry-analyst-73584

01/21/2022, 5:58 PM
Yes, I expect it to run in 2.7
👍 1
h

hundreds-father-404

01/21/2022, 6:00 PM
Do you need the whole thing or only the imports?
If you have the time, it would be super helpful to try to find the minimal reproduction. Meaning, delete some things and run again to see if it still fails. We want to find the specific line(s) that cause the parse error and remove all the other noise
d

dry-analyst-73584

01/21/2022, 6:02 PM
I'll see what I can do. Is there a way to limit 
generate-lockfiles
to only run on this file?
h

hundreds-father-404

01/21/2022, 6:02 PM
You should be able to run 
./pants dependencies path/to/fsluigi/main.py
-- it's failing at dependency inference time
d

dry-analyst-73584

01/21/2022, 6:02 PM
👍
Well, it's definitely happening in the code itself, not in the imports. If I delete all the code the imports are parsed fine. So it's weirdness on our end.
h

hundreds-father-404

01/21/2022, 6:08 PM
Ohhhhh it's a bug in our code, not yours. Dang
d

dry-analyst-73584

01/21/2022, 6:08 PM
Yep. If I comment out the three places this file is using
Copy code
__import__()
h

hundreds-father-404

01/21/2022, 6:08 PM
Ah and we didn't catch it because it's specifically when using 
__import__
, okay
d

dry-analyst-73584

01/21/2022, 6:08 PM
everything works fine
Oh, I really thought it was our code behaving badly, but if you wanna take credit for it, I'm fine with that 😂
git blame:
Copy code
a61fb7681862 (Eric Arellano      2018-08-21 16:22:48 -0700  32)       __import__("{}.{}".format(module_prefix, modname), fromlist="dummy")
😂
❗ 1
h

hundreds-father-404

01/21/2022, 6:11 PM
No not at all. Thanks for reporting it! I'll get out a bug fix soon and cherry-pick to 2.9
d

dry-analyst-73584

01/21/2022, 6:11 PM
👍 Thanks Eric!
h

hundreds-father-404

01/21/2022, 6:11 PM
omg what commit was that? yikes
d

dry-analyst-73584

01/21/2022, 6:11 PM
Copy code
commit a61fb76818627a80c1865d0ab00af0b820c9904f
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Aug 21 16:22:48 2018 -0700

    Modernize/string interpolation luigi (#8312)
👍 1
Not your fault 🙂
That file also does
Copy code
if profile_path:
    import cProfile
which pants appears to handle correctly.
Humm. I added 
src/fsluigi
to pants_ignore so I could keep going. I'm now running into a new error:
Copy code
The current project's Python requirement (>=2.7,<3 || >=3.6,<4) is not compatible with some of the required packages Python requirement:
    - pytest requires Python >=3.6, so it will not be satisfied for Python >=2.7,<3
My python requirement is only 
>=2,<3
, I'm not sure where the other version constraint is coming from, or why it claims pytest requires >=3.6.
h

hundreds-father-404

01/21/2022, 6:56 PM
see this tip to debug: https://www.pantsbuild.org/docs/python-interpreter-compatibility#tip-activate-pantsbackendpythonmixed_interpreter_constraints Written with what I wish I had when working on Foursquare's Py3 migration 🙂
d

dry-analyst-73584

01/21/2022, 6:57 PM
👍
h

hundreds-father-404

01/21/2022, 7:08 PM
Thanks again, I'll cherry-pick to 2.9: https://github.com/pantsbuild/pants/pull/14232
d

dry-analyst-73584

01/21/2022, 7:20 PM
Thank you!
I think the pytest version issue is the result of the PANTS_PYTEST_VERSION default.
h

hundreds-father-404

01/21/2022, 7:29 PM
wait what pytest vesrion issue? but yeah, we use a modern version of pytest by default that doesn't work with Py27
d

dry-analyst-73584

01/21/2022, 7:33 PM
Fixed it, i think. I generated lockfiles, but I had to upgrade our version of pytest from 
==3.6
to 
<=4.6
and I have yet to see if that will break anything.
We just had a meeting to discuss updating our requirements more frequently. Some of these are really old.
➕ 1
h

happy-kitchen-89482

01/21/2022, 11:07 PM
You'll need a python 2.7-compatible version of pytest, but the resolver should do the right thing if you don't constrain
d

dry-analyst-73584

01/22/2022, 1:28 AM
👍
Things seem to be working now, in that I have a new error instead (Pants doesn't know about where we host our custom wheels)
h

happy-kitchen-89482

01/22/2022, 3:31 AM
d

dry-analyst-73584

01/22/2022, 3:38 AM
Yeah, I found that. Any idea how I get credentials working? It's a private artifactory repo. I'm extremely unclear on how our V1 pants is accessing it right now.
h

happy-kitchen-89482

01/22/2022, 3:13 PM
You can set up a custom CA bundle and/or pass auth-related env vars through to the process, as described here: https://www.pantsbuild.org/docs/proxies#setting-up-a-certificate-authority
But you'll have to figure out how that artifactory's auth is supposed to work first
d

dry-analyst-73584

01/22/2022, 5:48 PM
Thanks Benjy!
7 Views
Powered by