pantsbuild #general

I have a task that according to pants server is never ending while the log and the result says otherwise. It's a simple-gen task that use a jvm cli. I print the code returned by self.runjava (and see the generated sources) but the server shows that the corresponding workunit is not stoping. Any guess of where I should look to solve this ?

brief-engineer-67497

07/24/2019, 7:57 AM

(using pants 1.15.0 )

hundreds-breakfast-49010

07/25/2019, 5:22 PM

https://travis-ci.org/pantsbuild/pants/builds/563234513?utm_source=github_status&utm_medium=notification flaked on shard 0

hundreds-breakfast-49010

07/26/2019, 10:40 PM

I think I encountered a bug in how pants does HTTP fetching, and wanted to get a second opinion

hundreds-breakfast-49010

07/26/2019, 10:41 PM

https://github.com/pantsbuild/pants/blob/7dea441933a894b084de43fe1764889e32506dbf/src/python/pants/net/http/fetcher.py#L351 throws an error if the number of bytes read from a response and the size from the header are not the same

hundreds-breakfast-49010

07/26/2019, 10:42 PM

but we're calculating the size by directly turning the

Content-Length

header into an int: https://github.com/pantsbuild/pants/blob/7dea441933a894b084de43fe1764889e32506dbf/src/python/pants/net/http/fetcher.py#L259

hundreds-breakfast-49010

07/26/2019, 10:42 PM

and I think this is the wrong behavior if we're getting a gzip'd HTTP response, since in that case

Content-Length

will be the compressed size, but we're comparing it against the uncompressed size

hundreds-breakfast-49010

07/26/2019, 10:44 PM

I noticed this because when I was creating a local version of the cloc v1.80 binary to test against at https://raw.githubusercontent.com/gshuflin/binhost/master/bin/cloc/1.80/cloc , I was hitting that error in

fetcher.py

, and also my browser was doing gzip compression and had a

Content-Length

header about 1/5 the size of the actual downloaded binary

hundreds-breakfast-49010

07/26/2019, 11:40 PM

ah, yeah, I think that's exactly what's going on: https://2.python-requests.org/en/master/user/quickstart/#response-content

hundreds-breakfast-49010

07/26/2019, 11:42 PM

iter_content

decodes gzip for us, so that code is comparing the uncompressed size of the file to the compressed size in the content-length header, which is not correct

witty-crayon-22786

07/26/2019, 11:43 PM

neat... haven't looked deeply, but sounds like a good find

witty-crayon-22786

07/26/2019, 11:43 PM

if you feel like porting some of that code to rust, feel free!

hundreds-breakfast-49010

07/26/2019, 11:45 PM

I'm not sure what the correct thing to do here is. it looks like @enough-analyst-54434 wrote most of this code, so maybe he has some insight about what the "right" check that we want to be doing here is

hundreds-breakfast-49010

07/26/2019, 11:45 PM

I'd be more than happy to port some of this code to rust, but I want to make sure I understand all the places this gets used first

👍 2

hundreds-breakfast-49010

07/26/2019, 11:46 PM

I would imagine the entire

Fetcher

class should be rust code, if we do this, and that would involve making every client of it FFI

hundreds-breakfast-49010

07/26/2019, 11:46 PM

(and I'm not sure how much of a priority doing that is)

aloof-angle-91616

07/26/2019, 11:46 PM

if so, i would really, really, really want to design it from the start to address network flakiness

💯 1

hundreds-breakfast-49010

07/26/2019, 11:46 PM

whereas to just fix this bug, we could do a one-liner that would just remove the check in

fetcher.py

hundreds-breakfast-49010

07/26/2019, 11:46 PM

(and then we wouldn't have that check anymore, but the check is broken right now, so maybe that's okay?)

witty-crayon-22786

07/26/2019, 11:47 PM

yea, sorry. this is a friday afternoon "one line bug? rewrite the whole thing!"... type suggestion

😂 1

witty-crayon-22786

07/26/2019, 11:47 PM

don't take me too seriously =P

hundreds-breakfast-49010

07/26/2019, 11:47 PM

but certainly it's a separate issue from the question of when and how to rewrite the entire class in rust

witty-crayon-22786

07/26/2019, 11:47 PM

but... see https://github.com/pantsbuild/pants/issues/7790

hundreds-breakfast-49010

07/26/2019, 11:48 PM

definitely happy to start working on that ticket

hundreds-breakfast-49010

07/26/2019, 11:49 PM

also I mentioned this to toolchain people but I don't think I mentioned it here, I'm going to be on vacation for the next two weeks

early-kangaroo-39629

07/27/2019, 12:10 AM

Hey all, can anyone restart these shards pls: https://travis-ci.org/pantsbuild/pants/jobs/564212871 https://travis-ci.org/pantsbuild/pants/jobs/564212879. Thanks

miniature-dawn-49992

07/31/2019, 6:17 PM

can somebody please provide me with an example of using

resources()

in a python context? i have a few openapi specs i want to include but when they make it into the pythonpath, they are packageless so

pkgutil.get_data

can't find them.

proud-yak-54773

08/01/2019, 3:10 PM

I’m trying to determine which pants caches are beneficial for my CI environment. Not entirely sure if I need to be caching the build cache as well as the artifact cache. Is there a guide for pants/CI integration?

witty-crayon-22786

08/01/2019, 3:57 PM

there is https://www.pantsbuild.org/orgs.html#continuous-integration ... but it doesn't mention caches.