https://pantsbuild.org/ logo
#general
Title
# general
r

refined-addition-53644

11/15/2022, 2:30 PM
Hi, my
generate-lockfiles
was taking forever (>=10 mins) and won’t even finish. I figured it out that it’s because of adding s3fs which depends on aiobotocore. It’s the aiobotocore which has some very restrictive dependencies. How long is pants/pex supposed to keep trying? I tried it with pip and pip throws an error about this.
1
r

ripe-cpu-85141

11/15/2022, 2:45 PM
Hello, I had an issue quite similar and it was related to pants calling pip with the new resolver. pip was taking an eternity to resolve the dependencies. You can try to run the same pip command pants is running to confirm.
I personally fixed that by making sure my dependencies have at least a >= to ensure pip doesn't try with versions from the last century 🙂
r

refined-addition-53644

11/15/2022, 3:01 PM
I have somewhat relaxed dependencies. This rigid pinned dependency is coming from a different package which I can't modify myself. I will have to abandon using this package because of that.
h

happy-kitchen-89482

11/15/2022, 4:01 PM
Some very long backtracking resolves are faster in newer versions of pip, which pex now supports, but we haven’t yet upgraded in Pants.
g

gentle-painting-24549

11/15/2022, 4:07 PM
Do you have boto3 listed as well? S3FS + Boto3 takes forever to resolve - When I want S3FS and boto3 on the same project I’ve started leaving out boto3 entirely and using the aiobotocore extra instead to get it which has been helpful:
Copy code
s3fs
aiobotocore[boto3]
r

refined-addition-53644

11/15/2022, 4:08 PM
yeah I do have both. I am actually going other way around. Just trying to use boto3 for now since there is also type stubs for mypy and moto(pytest mock for boto3) available.
Ah I see what you mean
@gentle-painting-24549 By the way how do mock (if is it possible)
s3fs
for testing?
g

gentle-painting-24549

11/15/2022, 4:15 PM
I’m not actually sure about mocking s3fs, but theoretically (🤞) moto should handle that by mocking the underlying call 🤷
Looks like s3fs is using moto in their own tests too https://github.com/fsspec/s3fs/blob/main/s3fs/tests/test_s3fs.py
🙌 1
Looks like they’re running a Moto Server to get that mocking to work. TIL Moto Server 😍
❤️ 1
h

happy-kitchen-89482

11/15/2022, 8:06 PM
@refined-addition-53644 which version of pip throws an error? Can you try with pip 22.3 vs 22.2.2 vs 20.3.4?
Also, this will allow pip upgrades in Pants 2.16: https://github.com/pantsbuild/pants/pull/17555
Newer pips may have perf improvements
🙏 1
r

refined-addition-53644

11/16/2022, 8:21 PM
Didn’t get time to try different pip but the error was thrown by 22.3.1
g

gentle-painting-24549

02/09/2023, 10:40 PM
@refined-addition-53644 I forgot to follow back up here, I got s3fs and moto working for mocking after a lot of help from this thread: https://github.com/aio-libs/aiobotocore/issues/755 I’m currently pinned on
aiohttp==3.8.3
and
moto[s3]==4.1.0
in my dev dependencies to get this to work - and using
s3fs==2023.1.0
`conftest.py`:
Copy code
from unittest.mock import MagicMock

import aiobotocore.awsrequest
import aiobotocore.endpoint
import aiohttp
import aiohttp.client_reqrep
import aiohttp.typedefs
import botocore.awsrequest
import botocore.model
import pytest

class MockAWSResponse(aiobotocore.awsrequest.AioAWSResponse):
    """
    Mocked AWS Response.

    <https://github.com/aio-libs/aiobotocore/issues/755>
    <https://gist.github.com/giles-betteromics/12e68b88e261402fbe31c2e918ea4168>
    """

    def __init__(self, response: botocore.awsrequest.AWSResponse):
        self._moto_response = response
        self.status_code = response.status_code
        self.raw = MockHttpClientResponse(response)

    # adapt async methods to use moto's response
    async def _content_prop(self) -> bytes:
        return self._moto_response.content

    async def _text_prop(self) -> str:
        return self._moto_response.text


class MockHttpClientResponse(aiohttp.client_reqrep.ClientResponse):
    """
    Mocked HTP Response.

    See <MockAWSResponse> Notes
    """

    def __init__(self, response: botocore.awsrequest.AWSResponse):
        """
        Mocked Response Init.
        """

        async def read(self: MockHttpClientResponse, n: int = -1) -> bytes:
            return response.content

        self.content = MagicMock(aiohttp.StreamReader)
        self.content.read = read
        self.response = response

    @property
    def raw_headers(self) -> Any:
        """
        Return the headers encoded the way that aiobotocore expects them.
        """
        return {
            k.encode("utf-8"): str(v).encode("utf-8")
            for k, v in self.response.headers.items()
        }.items()


@pytest.fixture(scope="session", autouse=True)
def patch_aiobotocore() -> None:
    """
    Pytest Fixture Supporting S3FS Mocks.

    See <MockAWSResponse> Notes
    """

    def factory(original: Callable[[Any, Any], Any]) -> Callable[[Any, Any], Any]:
        """
        Response Conversion Factory.
        """

        def patched_convert_to_response_dict(
            http_response: botocore.awsrequest.AWSResponse,
            operation_model: botocore.model.OperationModel,
        ) -> Any:
            return original(MockAWSResponse(http_response), operation_model)

        return patched_convert_to_response_dict

    aiobotocore.endpoint.convert_to_response_dict = factory(
        aiobotocore.endpoint.convert_to_response_dict
    )
`test_example.py`:
Copy code
import boto3
from moto import mock_s3


@mock_s3
def test_mocked_s3() -> None:
    """
    Test S3 Mocking with S3FS
    """
    # Create the bucket in our Mocked S3 Env First
    conn = boto3.resource("s3", region_name="us-east-1")
    conn.create_bucket(Bucket="example-bucket")

    # Create the files using whatever you're doing with S3FS
    upload_three_files_using_s3fs(bucket="example-bucket") # pseudo code

    # Make some assertions about what's supposed to be in S3
    bucket = conn.Bucket("example-bucket")
    matching_file_objects = list(bucket.objects.all())
    assert len(matching_file_objects) == 3
❤️ 1
r

refined-addition-53644

02/09/2023, 10:45 PM
Thank you! So I actually am running a. moto server which I copied from aiobotocore tests itself. But I have seen some issues when I try to use mock for boto3 and aiobotocore using moto server.
6 Views