So we are currently running into an issue where running the archive command is giving us this error:

16:54:26  /usr/bin/tar: torch-1.13.1-cp39-cp39-manylinux1_x86_64.whl: file changed as we read it

The steps prior to is are doing a pip download, followed by the archive. The relevant portions of the config are as follows:

shell_command(
    name = "textcat_server_dependency_wheel_collection",
    execution_dependencies = [ ":root_build_tools_scripts",
        "resolves/core-libs/python:requirements@resolve=core_libs",
        "resolves/text-classifier/python:requirements@resolve=text_classifier",
        "resolves/textcat-server/python:requirements",
        "resolves/flask-server/python:requirements@resolve=flask_server"],
    command = "build-scripts/constraints-from-lock \
        -l resolves/core-libs/python/core-libs-requirements.lock \
        -l resolves/text-classifier/python/text-classifier-requirements.lock \
        -l resolves/textcat-server/python/textcat-server-requirements.lock \
        -l resolves/flask-server/python/flask-server-requirements.lock \
        constraints.txt && \
        pip3 download --only-binary :all: --python-version 39 --platform linux_x86_64 --platform manylinux1_x86_64 --platform manylinux2010_x86_64 --platform manylinux2014_x86_64 -c constraints.txt -r resolves/core-libs/python/requirements.txt -r resolves/text-classifier/python/requirements.txt -r resolves/textcat-server/python/requirements.txt -r resolves/flask-server/python/requirements.txt",
    tools = [ "env", "python3", "bash", "basename", "find", "pip3", "dirname" ],
    workdir = "/",
    log_output = True,
    output_files = [ "*.whl" ],
    timeout = 2400
)

archive(
    name = "textcat_server_source_wheels",
    files = [ ":textcat_server_dependency_wheel_collection" ],
    packages = [ "src/python:textcat_server_wheel" ],
    format = "tar.gz",
    output_path = "source-wheels/textcat_server_source_wheels.tar.gz"
)

Any ideas on why the downloaded file would mutate while the archive is being created? Is this somehow related to the files being moved between sandboxes?

This looks like the same symptoms as https://github.com/pantsbuild/pants/issues/19740, which, as you suspected, is apparently caused by the tar program getting confused when a file has a new hard link added (even though that doesn’t change contents) which is what happens when creating sandboxes

based on what I'm seeing it's because the mtime is being modified while the tar is being created. This is probably due to the hard link being created

@blue-city-97042 can you add your observation to that ticket? Will help prioritize it

updated as a comment to the ticket.