#general
- c
cuddly-accountant-94274
09/03/2015, 2:43 AMYes, i was gonna submit a patch for that, it's myh first confusion. simple smile - e
enough-analyst-54434
09/03/2015, 2:43 AMIf you're willing to file an issue it's at least an enhancement - c
cuddly-accountant-94274
09/03/2015, 2:43 AMIt says "fetched" when really it probably just means "http response headers received". - c
cuddly-accountant-94274
09/03/2015, 2:44 AMI will, atm I'm exploring around. Eventually I'll do the account setup stuff and suggest the change. - e
enough-analyst-54434
09/03/2015, 2:45 AMSg - ty - b
bored-art-40741
09/03/2015, 2:46 AMI love streaming when we can, but it's worth noting that it frequently adds significantly to the complexity of the code involved - c
cuddly-accountant-94274
09/03/2015, 2:46 AMin this case go_fetch just has a yield binary.read(), should be easy to turn that into a loop. but may be overlooking something. - b
bored-art-40741
09/03/2015, 2:46 AMIf it's causing problems then it's time to properly stream - c
cuddly-accountant-94274
09/03/2015, 2:46 AMno problem - c
cuddly-accountant-94274
09/03/2015, 2:46 AMjust happened to notice while trying to debug my actual problem - c
cuddly-accountant-94274
09/03/2015, 2:47 AM(it downloads the file but there's no sign of the post-rename; not sure why yet; wondering if an exception is swallowed; in any case the file doesn't exist a bit later when the untar happens) - b
bored-art-40741
09/03/2015, 2:47 AMIs it trying to do an atomic download + rename? - b
bored-art-40741
09/03/2015, 2:47 AMThat's a common pattern - c
cuddly-accountant-94274
09/03/2015, 2:48 AMYes, though AFAIK there's no fsync so it's not actually correct - c
cuddly-accountant-94274
09/03/2015, 2:48 AM(but better certainly than no atomic rename at all) - b
bored-art-40741
09/03/2015, 2:49 AMIt's guarding in general against multi-process races - c
cuddly-accountant-94274
09/03/2015, 2:49 AMThe temporary file is there (zero bytes), but there's no post-rename file. It's just gone. So the delete happened, but possibly not the rename. But there's no rename error. - c
cuddly-accountant-94274
09/03/2015, 2:49 AMYes; i get that. Just not crash-safe. - b
bored-art-40741
09/03/2015, 2:49 AMHmm, I'm not sure about that - e
enough-analyst-54434
09/03/2015, 2:49 AMNB - it's using python stdlib instead of requests as well, would be nice to move to Fetcher: https://github.com/pantsbuild/pants/blob/master/src/python/pants/net/http/fetcher.py#L202 - c
cuddly-accountant-94274
09/03/2015, 2:50 AMopen + write + rename is not crash-safe without fsync unless you are able to make assumptions about the file system or validate the checksum of the file - b
bored-art-40741
09/03/2015, 2:50 AMThe pattern for atomic file placement does a "rename" at the end - c
cuddly-accountant-94274
09/03/2015, 2:50 AMI know. - c
cuddly-accountant-94274
09/03/2015, 2:50 AMBut it's not crash-safe without fsync() - e
enough-analyst-54434
09/03/2015, 2:50 AMPeter basically crushes us all on FS knowledge - b
bored-art-40741
09/03/2015, 2:50 AMWhat exactly is not crash safe? - e
enough-analyst-54434
09/03/2015, 2:50 AMyou want to concede - c
cuddly-accountant-94274
09/03/2015, 2:50 AMHaha. - b
bored-art-40741
09/03/2015, 2:50 AMThe existence of the final file in the proper state? - c
cuddly-accountant-94274
09/03/2015, 2:50 AMIt may not be part of the purpose here for pants.