https://pantsbuild.org/ logo
Join Slack
Powered by
Hello there! We recently started using Pants as ou...
# general
o
Hello there! We recently started using Pants as our build system and are still getting used to some nuances. I have a few questions around protobuf and docker support - • When will protobuf support be re-enabled? It looks like it was removed in 2.0.0a2. • Are there any best practices for using pants to generate & run pex binaries inside Dockerfiles? Since we generate our pex files inside the docker container (see the Dockerfile snippet below), there is no caching of dependencies. Ideally, if none of the dependencies or source files change, pex shouldn't need to be regenerated in the container. • Is there a way to enable 'live reloading' while debugging/code changes? It will help improve the development flow a lot! Sample Dockerfile -
Copy code
FROM python:3.7.7-slim

RUN apt-get update \
    && apt-get install -y \
    curl \
    cmake \
    build-essential \
    && apt-get clean


EXPOSE 3000
WORKDIR /app

RUN curl -L -o ./pants <https://pantsbuild.github.io/setup/pants> \
    && chmod +x ./pants
# Copy src dir and pants config files
COPY ./src /app/src/
COPY BUILD /app/
COPY pants_docker.toml /app/pants.toml
COPY requirements.txt /app/

COPY ./dist /app/dist/

RUN ./pants --version

RUN ./pants binary src/nabu/services/api:server

CMD ./dist/server.pex nabu.services.api.wsgi:app -c src/nabu/services/api/gunicorn.conf.py
h
re: protobuf support, https://github.com/pantsbuild/pants/issues/10683 is the issue we're using to track it. this is a surprisingly tricky point, but we are actively working on it and will release a fix asap
as far as pants best practices with docker go, I haven't personally thought much about this (maybe @hundreds-father-404 has some thoughts?)
long-term, we would probably like to have pants itself create a docker image as one of many types of build artifacts that pants can produce
I can't think of a way offhand to get good caching if you want to retain a workflow that installs and runs pants as part of the docker image creation process
what sort of workflow are you imagining with "live reloading"? pants has the functionality to run a daemon (pantsd) that keeps a bunch of pants state in memory and allows subsequent pants commands to run more quickly
h
Hey Paritosh! Welcome. And pardon the delay - most of us live in the US and were afk during Labor Day
Is there a way to enable ‘live reloading’ while debugging/code changes?
I agree this would be useful. We have an experimental option 
--loop
, but it does not work well right. We’ve been working on the upcoming 2.0 release, and as part of that, we’re putting polish on lots of pieces relevant to getting 
--loop
working, such as improving the Pants daemon (pantsd) https://github.com/pantsbuild/pants/issues/9462 tracks this feature. Could you please comment that you’d appreciate this feature and how you’d want to use it (e.g. for 
repl
, for 
test
, for something else?)? https://github.com/pantsbuild/pants/issues/9462
Are there any best practices for using pants to generate & run pex binaries inside Dockerfiles?
Interesting. So Pants stores its caches (by default) in 
~/.cache/pants/lmdb_store
and 
~/.cache/pants/named_caches
. The problem is that you’d need that to be persisted across runs. In our CI, we use volumes, along with setting some image args so that we can call 
WORKDIR
. See https://github.com/pantsbuild/pants/blob/master/build-support/docker/travis_ci/Dockerfile, then https://github.com/pantsbuild/pants/blob/bc16088727e63739a648605bc996b81a65a513d1/.travis.yml#L78-L84 for where we build and run the image. https://dev.to/jibinliu/how-to-persist-data-in-docker-container-2m72 is maybe useful background too on Docker volumes Would something like this work?
h
Re protobuf, just to emphasize: restoring protobuf support is our #1 priority, and we will definitely do it before the 2.0 stable release. We temporarily deactivated it because it triggered a performance bug (even if you weren't actually using protobuf support) that we're actively working to fix.
👍 4
o
Hey guys, really appreciate the detailed responses! And I apologize for the slow follow up from my side!
❤️ 1
re: protobuf support - glad to know that it will be restored very soon! Thanks for confirming! A quick follow up question to protobuf - is there a target that exists for building grpc related protobuf files? I see this in the v1 documentation - https://v1.pantsbuild.org/grpcio_gen.html but I couldn't find something similar in v2. If a target doesn't already exist, do I just add grpcio tools as a runtime target for protoc as shown in this example? - https://github.com/danieljanes/pants-python-protobuf/blob/master/pants.toml
long-term, we would probably like to have pants itself create a docker image as one of many types of build artifacts that pants can produce
This makes a lot of sense! Is there an issue I can upvote for this 🙂.
h
The v2 engine doesn’t yet support gRPC, but this is on the roadmap for the 2.0 release. Tracked by https://github.com/pantsbuild/pants/issues/10497
o
Got it. Is there a suggested workaround for this right now?
what sort of workflow are you imagining with "live reloading"? pants has the functionality to run a daemon (pantsd) that keeps a bunch of pants state in memory and allows subsequent pants commands to run more quickly
I agree this would be useful. We have an experimental option 
--loop
, but it does not work well right. We’ve been working on the upcoming 2.0 release, and as part of that, we’re putting polish on lots of pieces relevant to getting 
--loop
 working, such as improving the Pants daemon (pantsd)
Great! Would love to see this make it to a stable release. There are a few workflows but the most common one for us is manual testing of APIs on a gRPC or a Flask server. Debugging/development is much faster if the locally running server reloads when a source file changes.
https://github.com/pantsbuild/pants/issues/9462 tracks this feature. Could you please comment that you’d appreciate this feature and how you’d want to use it (e.g. for 
repl
, for 
test
, for something else?)?
Great, will do!
💯 1
h
Is there a suggested workaround for this right now?
Unfortunately, not yet. You’d have to use 1.30.0rc0 and the v1 implementation of Python. You can mix and match the v1 and v2 implementations, like use v2 for the Python formatters and linters, but v1 for 
test
, 
run
, 
binary
, and 
repl
. The v2 linters and formatters wouldn’t have access to the code genned files, but generally that should be fine afaict
o
Got it. Let me give that a try and get back to you!
In our CI, we use volumes, along with setting some image args so that we can call 
WORKDIR
. See https://github.com/pantsbuild/pants/blob/master/build-support/docker/travis_ci/Dockerfile, then https://github.com/pantsbuild/pants/blob/bc16088727e63739a648605bc996b81a65a513d1/.travis.yml#L78-L84 for where we build and run the image. https://dev.to/jibinliu/how-to-persist-data-in-docker-container-2m72 is maybe useful background too on Docker volumes
I'll definitely give this suggestion a try. I've used volumes before for something similar (ex. hot reloading inside a docker container as a source file on the host changes) . My only concern here is that there are some libraries (ex. bcrypt) that need to be built on the same platform as the runtime platform.
👍 1
h
Cool! We’ll try to get gRPC going asap, as soon as we fix Protobuf in general. You’ll want pants.toml to look something like this:
Copy code
[GLOBAL]
v1 = true
v2 = true
backend_packages = ["pants.backend.python"]
backend_packages2 = ["pants.backend.python.lint.isort", ...]
Pants will use v2 for 
fmt
and 
lint
, but v1 for everything else
o
To clarify - this will enable the 
python_grpcio_library
target mentioned here - https://v1.pantsbuild.org/grpcio_gen.html right?
h
Oh, actually, now that I think about it, leave off 
backend_packages = ["pants.backend.python"]
. In v1, we activated all backends by default. (Which was a bad idea, but difficult to deprecate). The snippet I had given would override the default. You can run 
./pants target-types
to confirm that 
python_grpcio_library
shows up
o
Thanks for the tip @hundreds-father-404. This seems to be working!
❤️ 1
m
@hundreds-father-404 With Docker binary support for python, are you considering PEX to be the format for packaging python environment within docker image or allow the option for skipping the PEX format and install 3rd party python dependencies and first party python packages directly in the global python environment within docker image?
h
Honestly, I don’t think we have it very well thought out. It’s still a little amorphous to us what docker support tangibly looks like
@miniature-lamp-45970 and @orange-beach-75711, could you please describe what your ideal integration with Docker would look like in https://github.com/pantsbuild/pants/issues/2648? cc @wonderful-iron-54019 if you want to add what y’all would like too
m
@hundreds-father-404 Yes we can consolidate and add the requirements in the ticket
❤️ 1
h
Thanks! We’re thinking this might be a good post-2.0 priority. And we are also eager to support anyone who would want to contribute, which would help get the feature sooner. Either way, first step is scoping what the feature means.
👍 1
7 Views
Open in Slack
PreviousNext
Powered by