https://pantsbuild.org/ logo
Join Slack
Powered by
Hi all. I've encountered a problem after upgradin...
# general
w
Hi all. I've encountered a problem after upgrading from 2.18 to 2.21.1. Did something change in python-infer to cause it to stop picking up dependencies when a package's 
__init__.py
contains imports like, 
from . my_package_component import *
? I've tried running with use_rust_parser set to true and false, but no difference. Preserving the sandboxes shows that the dependency's 
__init__.py
file is imported, but none of the files it imports are being pulled in to the sandbox. Neither 
pants peek __init__.py
nor 
pants dependencies __init__.py
list any dependencies aside from another 
__init__.py
in a package subfolder. I've been struggling with this for about a week now, but I can't find any solutions in the github issues or from google searches. Any suggests would be greatly appreciated. Thanks!
f
Are you able to try the latest version of 2.19 and 2.20 as well to bisect when the change occurred?
w
Sure! Give me a few minutes to step through the version.
👍 1
Looks like this will take a few minutes.
So I left use_rust_parser = true and worked all the back to the original 2.18 version and it still didn't work. However, setting use_rust_parser = false has things working again (though it didn't work until I regenerated-lockfiles after the change)
I'm going to try stepping forward without rust parser and see what I learn.
updated to 2.19.0, regenerated lockfiles and it does not work.
w
This is the first instance I've seen where 
use_rust_parser
made a difference for anyone, that's strange
Could this be a rust_parser + source roots issue or something?
Can you also run this and output the results?
Copy code
pants peek :: > before.json
pants --python-infer-use-rust-parser=False peek :: > after.json
And, if those are different, could you also try:
Copy code
--python-infer-init-files=always
https://www.pantsbuild.org/stable/reference/subsystems/python-infer#init_files
w
@wide-midnight-78598, thank you for these ideas. I will try them out. I am also trying to build a test repo in which I hope to recreate the issues I'm seeing. If I succeed, I'll share the repo with you.
@wide-midnight-78598, I think I have it. We have a old libraries with some non-standard formatting. Specifically the package files used imports that had spaces between the "." and the package filenames. For whatever reason, this worked for us in 2.18.0, but beyond that it produces inference errors. For example:
Copy code
# This works
from .my_class import MyClass

# This causes inference error
from . my_class import MyClass
When we cleaned up the code to get rid of the inappropriate spaces then things began working under 2.21.1 just as they did under 2.18.0. It does not matter whether user_rust_parser is true or false, the inference errors are consistent.
w
Great to hear - also interesting that the second one is valid, I never knew that (but I've also never tried it 🤷 )
w
So it seems that Python accepts "from .somefile ..." imports with or without a space between the dot and somefile as both valid. Pants' inference parser, however, does not accept spaces after the dot as valid. I propose that the inference parser grammar should be updated to match this otherwise valid Python code, even if it is not the norm. Otherwise, it should raise an error so the user understands why their dependency inferences are failing.
f
It uses the TreeSitter library for parsing and a Python grammar from 
tree-sittter-python
crate.
Maybe the parsing issue needs to be fixed upstream in 
tree-sitter-python
first?
Alternatively, maybe it was fixed already in a newer (if any) version of 
tree-sitter-python
than in use by Pants?
Open in Slack
PreviousNext
Powered by