@billowy-minister-38128 : the other potentially relevant bit is that zinc has been refactored and upgraded in the 1.3.x dev series.

we used to see this, but we haven't in a while

we're currently on 1.3.0dev7 internally

(which was the reason for the peiyu ping)

@witty-crayon-22786 i have tried 1.3.0dev7 and believe i’ve encountered the error. as it is non-deterministic, i cannot say with 100% certainty though.

i’m now writing up a little on the actual issue… one sec.

just to confirm the problem you see is

java.io.IOException: Analysis at...has changed since startup!

, and usually rerun a second time would pass

exactly

ok, the

non-deterministic

part is probably the hardest, will wait for your details

another thing to confirm: you don't override the

//:zinc

definition, do you? it's the default for the release you're using?

Due to lack of versions in ivy resolution, i cannot do a build all, i have to build each target in quick succession. Currently, I have 6 targets (and 6 tests) in a bash script. Many of these targets have common dependencies, all within the monorepo. For example, cdh5 depends on hadoop-common which depends on jdbc. In the bash script, I was creating an artifact for hadoop-common, so the collision occurred there. I removed hadoop-common as we don’t actually need to publish an artifact, the next down the chain, jdbc, started colliding. We do need to publish an artifact so removal isn’t possible. I reordered the build targets in bash, putting jdbc first then the version of hadoop-common from cdh5 began colliding with hadoop-common from mapr. Adding sleep between test and bundle has mitigated the issue a bit, but not entirely. I’ve removed the sleep and added a retry which seems to fix the issue for now.

@witty-crayon-22786 i do not override the

//:zinc

definition.

i am currently (for similar reasons) running a bash loop like

for t in targets; do ./pants test $t; done

across about 130 targets, and have for a while today

and i haven't seen this in months.

so something is... still fishy

i agree. i used to be able to repro this every time on my local machine.. now i cannot repro no matter what.

where does it repro?

i’m using the CI agent to repro… and it only happens if tests are turned on.

turn tests off and it’s fine.

if I created a test step, then a bundle step, this may alleviate the issue.

my big question was

why are there multiple versions of the analysis for each target in .pants.d/compile/zinc/?

- `shouldn’t the binary be the same?`…

but looking closer at the analyses, there are diffs in transitive deps.

correct.

my other question is how to clear invalid analysis. i tried adding this to pants.ini:

[compile.zinc]
incremental: False
clear_invalid_analysis = True

but it does not seem to help.

clear_invalid_analysis: True

... applies to a pretty specific case where it is literally corrupted on disk

which should be hella rare

...but which has happened a few times in the past.

re multiple versions: analysis file also contains timestamps, which could change

well, that wouldn't cause multiple target fingerprints

ok, so i’m running

./pants test x

./pants bundle x

six times. 9 times out of 10, i get the error on much faster machine (CI agent). 1 out of 20 on my local machine, cannot repro in a docker container.