straight-action-8031809/19/2022, 11:22 PM
that lets me bundle everything together, and then use EMR’s Docker support to run my code remotely? I’m not sure how to create an environment for the workers, although I assume there’s a way to install the Pants repo into the docker such that it’s available for the workers?
I think if I wasn’t using Pants, what I would do is setup a docker container such that my repo has been
’d, that way all of the imports work for the Python.
happy-kitchen-8948209/20/2022, 4:35 PM
can build a docker image that contains your python code and its requirements, all ready to run. See this blog post for a lot of detail on how to do this efficiently (you may not care about the performance issues at first, in which case you can experiment with just the simpler examples in that post)
crooked-country-193709/20/2022, 8:59 PM
happy-kitchen-8948209/20/2022, 10:37 PM
straight-action-8031809/21/2022, 12:15 AM