Hi, my

generate-lockfiles

was taking forever (>=10 mins) and won’t even finish. I figured it out that it’s because of adding s3fs which depends on aiobotocore. It’s the aiobotocore which has some very restrictive dependencies. How long is pants/pex supposed to keep trying? I tried it with pip and pip throws an error about this.

Hello, I had an issue quite similar and it was related to pants calling pip with the new resolver. pip was taking an eternity to resolve the dependencies. You can try to run the same pip command pants is running to confirm.

I have somewhat relaxed dependencies. This rigid pinned dependency is coming from a different package which I can't modify myself. I will have to abandon using this package because of that.

Some very long backtracking resolves are faster in newer versions of pip, which pex now supports, but we haven’t yet upgraded in Pants.

Do you have boto3 listed as well? S3FS + Boto3 takes forever to resolve - When I want S3FS and boto3 on the same project I’ve started leaving out boto3 entirely and using the aiobotocore extra instead to get it which has been helpful:

s3fs
aiobotocore[boto3]

yeah I do have both. I am actually going other way around. Just trying to use boto3 for now since there is also type stubs for mypy and moto(pytest mock for boto3) available.

I’m not actually sure about mocking s3fs, but theoretically (🤞) moto should handle that by mocking the underlying call 🤷

@refined-addition-53644 which version of pip throws an error? Can you try with pip 22.3 vs 22.2.2 vs 20.3.4?

Didn’t get time to try different pip but the error was thrown by 22.3.1

@refined-addition-53644 I forgot to follow back up here, I got s3fs and moto working for mocking after a lot of help from this thread: https://github.com/aio-libs/aiobotocore/issues/755 I’m currently pinned on

aiohttp==3.8.3

and

moto[s3]==4.1.0

in my dev dependencies to get this to work - and using

s3fs==2023.1.0

`conftest.py`:

from unittest.mock import MagicMock

import aiobotocore.awsrequest
import aiobotocore.endpoint
import aiohttp
import aiohttp.client_reqrep
import aiohttp.typedefs
import botocore.awsrequest
import botocore.model
import pytest

class MockAWSResponse(aiobotocore.awsrequest.AioAWSResponse):
    """
    Mocked AWS Response.

    <https://github.com/aio-libs/aiobotocore/issues/755>
    <https://gist.github.com/giles-betteromics/12e68b88e261402fbe31c2e918ea4168>
    """

    def __init__(self, response: botocore.awsrequest.AWSResponse):
        self._moto_response = response
        self.status_code = response.status_code
        self.raw = MockHttpClientResponse(response)

    # adapt async methods to use moto's response
    async def _content_prop(self) -> bytes:
        return self._moto_response.content

    async def _text_prop(self) -> str:
        return self._moto_response.text


class MockHttpClientResponse(aiohttp.client_reqrep.ClientResponse):
    """
    Mocked HTP Response.

    See <MockAWSResponse> Notes
    """

    def __init__(self, response: botocore.awsrequest.AWSResponse):
        """
        Mocked Response Init.
        """

        async def read(self: MockHttpClientResponse, n: int = -1) -> bytes:
            return response.content

        self.content = MagicMock(aiohttp.StreamReader)
        self.content.read = read
        self.response = response

    @property
    def raw_headers(self) -> Any:
        """
        Return the headers encoded the way that aiobotocore expects them.
        """
        return {
            k.encode("utf-8"): str(v).encode("utf-8")
            for k, v in self.response.headers.items()
        }.items()


@pytest.fixture(scope="session", autouse=True)
def patch_aiobotocore() -> None:
    """
    Pytest Fixture Supporting S3FS Mocks.

    See <MockAWSResponse> Notes
    """

    def factory(original: Callable[[Any, Any], Any]) -> Callable[[Any, Any], Any]:
        """
        Response Conversion Factory.
        """

        def patched_convert_to_response_dict(
            http_response: botocore.awsrequest.AWSResponse,
            operation_model: botocore.model.OperationModel,
        ) -> Any:
            return original(MockAWSResponse(http_response), operation_model)

        return patched_convert_to_response_dict

    aiobotocore.endpoint.convert_to_response_dict = factory(
        aiobotocore.endpoint.convert_to_response_dict
    )

`test_example.py`:

import boto3
from moto import mock_s3


@mock_s3
def test_mocked_s3() -> None:
    """
    Test S3 Mocking with S3FS
    """
    # Create the bucket in our Mocked S3 Env First
    conn = boto3.resource("s3", region_name="us-east-1")
    conn.create_bucket(Bucket="example-bucket")

    # Create the files using whatever you're doing with S3FS
    upload_three_files_using_s3fs(bucket="example-bucket") # pseudo code

    # Make some assertions about what's supposed to be in S3
    bucket = conn.Bucket("example-bucket")
    matching_file_objects = list(bucket.objects.all())
    assert len(matching_file_objects) == 3

Thank you! So I actually am running a. moto server which I copied from aiobotocore tests itself. But I have seen some issues when I try to use mock for boto3 and aiobotocore using moto server.