https://pantsbuild.org/ logo
Join Slack
Powered by
Any ideas why `./pants lint ::` would succeed loca...
# general
f
Any ideas why 
./pants lint ::
would succeed locally and then fail on some files on the same branch in CI? This has happened once or twice on my team. Recloning the branch into a new directory and running 
./pants fmt ::
properly catches the files that need reformatting. Could it be a local cache issue?
🎉 1
b
Can you be more specific? What tools are failing? Whats the failure message? etc...
f
Failure probably isn’t the right word. We have a Python monorepo and use pants to run 
black
and 
isort
. We’ve had occasions where folks will run 
./pants fmt ::
and some files get reformatted. Our pre-commit hook runs 
./pants lint ::
and it passes. But then our 
./pants lint ::
check in CI flags several files as needing to be reformatted. The only way we can get the files reformatted is to make a fresh clone of the branch in a new directory and run 
./pants fmt ::
there.
b
Do you have 
isort
configured to use `black`'s style?
f
Yep
b
Can you provide a minimal diff of the reformat/error?
GitHub repo is easiest to use for reproduction
f
This is an example of what we’ll see in CI:
Copy code
$ ./pants lint ::
22:38:53.59 [INFO] Completed: Lint with Flake8 - flake8 succeeded.
22:38:53.59 [INFO] Completed: Format with Black - black made no changes.
22:38:53.60 [INFO] Completed: Format with Autoflake - autoflake made no changes.
22:38:53.60 [INFO] Completed: Format with isort - isort made no changes.
22:38:56.32 [WARN] Completed: Format with Autoflake - autoflake made changes.
  directory/config.py
22:38:57.22 [WARN] Completed: Format with Black - black made changes.
  directory1/__init__.py
  directory1/config.py
  directory2/__init__.py
22:38:58.00 [INFO] Completed: Format with Autoflake - autoflake made no changes.
22:39:00.13 [WARN] Completed: Format with Black - black made changes.
  directory3/__init__.py
22:39:03.94 [INFO] Completed: Lint with Flake8 - flake8 succeeded.
22:39:04.29 [INFO] Completed: Lint with Flake8 - flake8 succeeded.
22:39:05.01 [WARN] Completed: Format with isort - isort made changes.
  another_directory/trigger.py
22:39:05.06 [WARN] Completed: Format with isort - isort made changes.
  file1.py
  file2.py
  file3.py
22:39:05.06 [INFO] Wrote lint report files to dist/lint/flake8/__CPython<3.9,>=3.8__.
22:39:05.06 [INFO] Wrote lint report files to dist/lint/flake8/__CPython<3.9,>=3.8___.
22:39:05.06 [INFO] Wrote lint report files to dist/lint/flake8/__CPython<3.9,>=3.8____.
✕ autoflake failed.
✕ black failed.
✓ flake8 succeeded.
✕ isort failed.
(One or more formatters failed. Run `./pants fmt` to fix.)
It is proprietary code in a private GitLab repo so I can’t really share a reproducible version.
b
What do the diffs look like? IS it changing base don line lengths? Just imports? Line endings?
f
It’s doing full reformats. Import reordering, changes for line length, single to double quotes, adding newlines between functions, etc.
b
Is 
lint
running black locally?
f
Yep. This is all we have in our 
pants.ci.toml
that I guess would cause any differences in behavior:
Copy code
# This file defines settings to use in CI
# in addition to those in pants.toml

[GLOBAL]
# cache pants files in working dir
local_store_dir=".cache/pants/lmdb_store"
named_caches_dir=".cache/pants/named_caches"

# Colors often work in CI, but the shell is usually not a TTY so Pants
# doesn't attempt to use them by default.
colors = true

# enable section below to show cache hits
# [stats]
# log = true

[coverage-py]
report = ["xml"]
output_dir = "."

[pytest]
args = ["-vv", "--no-header"]
As I mentioned, to fix it all we do is make a fresh clone and run 
./pants fmt ::
as before. No other changes.
b
What Pants version is this?
f
2.12.0
b
What happens if you just run one of those tools in 
lint
as well as 
fmt
in the 'bad" case? 
./pants lint --only=black ::
then 
./pants fmt --only=black ::
f
Hard for me to say… Unfortunately, I’m just the messenger as I haven’t encountered this myself yet. And I haven’t been able to reproduce the issue because a fresh clone of the problematic branch “works.”
b
You might try running it locally and in CI with 
-ldebug
and inspecting the log (or scrubbing it and posting here). Otherwise it's hard to say without more info. Seems like a nasty bug for sure, but Pants has so many moving parts finding the problematic one is challenging. The daemon might be to blame perhaps. Or the way we "reuse" the process execution between 
fmt
and 
lint
So dumb question, are you sure the CI hook is "failing" appropriately? Like if the user does 
lint ::
themselves, do they see it passing?
f
Yep, the users have run 
./pants lint ::
locally after the lint step fails in CI. The formatters pass locally.
And yes, I wish I had a reproducible case for you! I’ll try running with 
-ldebug
. Is there anything in particular I should look out for?
b
I think I'd try and compare the nodes that have the description of something like "running 
black
on 22 files", or something along those lines. They likely won't be identical and thats a clue
👀 1
It also helps to understand the underlying difference between 
fmt
and `lint`: • For 
lint
, we create batches of files, then "materialize" a sandbox from a "snapshot" of those files. We do this for all batches of files across all tools in parallel. Under the hood, we run the tool on those files and capture the "snapshot" of files after. If they changed 💥 we error and print which files changed. • For 
fmt
however its a bit different. We still batch, but we have to run tools sequentially one-after-the-other to make sure two tools can re-format the same line (otherwise we'd have to try and "merge" results). So we feed the output of tool 1 as an input to tool 2, and so on... Meaning if the tools are making changes in 
fmt
, tool 2 and so on won't have a matching node to their respective 
lint
process since the input will be different (the unformatted repo file vs. the formatted output of tool 1). So the helpful node to look at is the node for running the first tool in 
fmt
(i believe they use the order of 
backend_packages
👍 1
f
Update: I just encountered this issue myself. While working I made changes to a number of files in my repo, all of which should need reformatting. I then ran 
./pants fmt ::
and 
autoflake
, 
isort
, and 
black
incorrectly said everything looks good. So I added a bunch of blank lines to the end of one file to see if that would trigger the formatters, but nope “no changes needed.” I then tried deleting the pants cache (
~/.cache/pants
) just to see what would happen, same result. I then ran 
./pants -ldebug fmt ::
and the formatters DID run. Very odd
b
(pro tip, 
mv
the cache instead of deleting so you can 
mv
it back)
Do you have the debug logs from the runs? Feel free to uniqueify filenames
f
Ok, just went into stand up so I’ll respond in a bit.
I scrubbed the file paths entirely. Hopefully this is still enough info to go on. If not, let me know.
pants-fmt-log.txt
b
1. We'd need to compare the logs from the "good" case to that of the "bad" case to see if the inputs are the same 2. I don't see the log(s) for running the tools (E.g. "run black on..."_
f
The “Run <formatter> on x files” are at the bottom of the file. Slack is truncating the display.
And I would provide the logs for the “bad” case, but I mentioned that running 
./pants -ldebug fmt ::
caused the formatters to pick up the file changes.
In other words, my attempt to get the logs for the “bad” case resulted in things working as expected and thus produced logs of the “good” case. Nothing changed between running 
./pants fmt ::
and 
./pants -ldebug fmt ::
, but the result differed.
b
@witty-crayon-22786 I'm no expert on the daemon but could it not be seeing the files be invalidated?
w
yea, it should be. the log should report all file edits
f
I have a theory. I’m working on an Azure Machine Learning Studio compute instance which mounts a fileshare for local storage that may be the cause of the issue I described. This setup has some other side effects we’ve noticed like slow 
git
performance. Periodically VS Code will show a notification to the effect of 
VS Code is unable to watch for file changes
which made me think that perhaps 
pants
is similarly not seeing any changed files and saying “no changes? no formatting needed.” At some point the changes are noticed and 
pants
runs the formatter as expected. TLDR; laggy inotify on a fileshare? (I am out of my depth at this level, but figured I’d share my theory.)
b
Oh wow yeah thats a huge piece of info lol
I bet VSCode is also using 
inotify
(whoch is what the daemon uses)
So you're likely spot-on. The Pants daemon isn't being notified about file changes
🙌 2
f
I probably should have mentioned my environment earlier on…!
b
Thank goodness I can pop this off my mental stack 😛
😅 2
w
Open in Slack
PreviousNext
Powered by