#21163 query paramters ignored in http_source cache key Issue created by cburroughs on <!date^1720730417^{date_short}|2024-07-11T20:40:17Z> The context here was trying to use S3 object versioning which looks something like a

versionId=Tx4Yxz3_aALevSHA5iesgg8vnZPK7Rkb

parameter on the url. Something like

?versionId=FAKE_TESTING

appeared to do nothing. With this handy test program running: from http.server import BaseHTTPRequestHandler, HTTPServer from urllib.parse import urlparse, parse_qs import hashlib class RequestHandler(BaseHTTPRequestHandler): def do_GET(self): parsed_url = urlparse(self.path) query_params = parse_qs(parsed_url.query) self.send_response(200) if 'badger' in query_params: resp = ''' Badger, badger, badger, badger Badger, badger, badger, badger Mushroom, mushroom '''.encode() else: resp = query_params['val'][0].encode() self.send_header('X-pants-sha256', hashlib.sha256(resp).hexdigest()) self.send_header('X-pants-len', len(resp)) self.end_headers() print(query_params) self.wfile.write(resp) host_name = 'localhost' host_port = 8080 httpd = HTTPServer((host_name, host_port), RequestHandler) print(f'Started HTTP server on {host_name}:{host_port}') httpd.serve_forever() And this BUILD file

file(
    name="hello",
    source=http_source(
        url='<http://localhost:8080/hello?val=world>',
        #url='<http://localhost:8080/hello?val=galaxy>',
        #url='<http://localhost:8080/hello?val=world&badger=true>'                
        
        sha256="486ea46224d1bb4fb680f34f7c9ad96a8f24ec88be73ea8e5a6c65260e9cb8a7",
        len=5,
    ),
)

run_shell_command(
    name="meow",
    execution_dependencies=[":hello"],
    command="cat {chroot}/hello",
)

Running:

$ pants --local-store-dir=/tmp/gh-issue  --no-pantsd run :meow
world

Now change the BUILD file to use

url='<http://localhost:8080/hello?val=galaxy>

$ pants --local-store-dir=/tmp/gh-issue  --no-pantsd run :meow
world

When instead the url should be fetched again and cause a digest error for the content/sha256 mismatch. I don't know Rust well but in my debugging I was suspicious of this line: pants/src/rust/engine/src/nodes/downloaded_file.rs Line 30 in</pantsbuild/pants/commit/7b2dcad1b9649b9005ba9683d1a0e09e29ef5127|7b2dcad> | url: url.path().to_owned(), | | --------------------------- |

fn url_key(url: &Url, digest: Digest) -> CacheKey {
        let observed_url = ObservedUrl {
            url: url.path().to_owned(),  // <---------
            observed_digest: Some(digest.into()),
        };
        CacheKey {
            key_type: CacheKeyType::Url.into(),
            digest: Some(Digest::of_bytes(&observed_url.to_bytes()).into()),
        }
    }

Versions

pants_version = "2.21.0"

While debugging this I also tripped over #21162 but I think they are distinct issues. pantsbuild/pants