https://pantsbuild.org/ logo
Join Slack
Powered by
It seems like the error `IndexError: string index ...
# general
a
It seems like the error 
IndexError: string index out of range
is a rather generic one. See example below:
Copy code
(pants) [root@9a5752857756 pants-training]# ./pants dependencies src/server/greeter_server.py 
src/proto/helloworld.proto
src/server/greeter_server.py
(pants) [root@9a5752857756 pants-training]# ./pants dependees src/server/greeter_server.py 
14:18:47.17 [ERROR] 1 Exception encountered:

 IndexError: string index out of range

(pants) [root@9a5752857756 pants-training]#
The 
dependencies
goal worked but 
dependees
failed on the same target. If the target doesn’t have any dependees then I guess it shouldn’t fail.
h
@ancient-rose-27306 Sorry for these useless errors. Even if there is a problem in your repo, that is obviously a terrible error message and a bug.
The best way for us to debug that is if you can push a complete repo to github that reproduces it
h
Also running the same command with 
--print-stacktrace
would help, please
a
I had executed this command yesterday for a different repo but the same error and it was not of much help 😔
h
the stacktrace is likely to be useful to us Pants devs, if you're able to run again by chance
👍 1
a
So, here’s the stacktrace output:
Copy code
(pants) [root@9a5752857756 pants-training]# ./pants --print-stacktrace package src/server:grpc-server
17:20:05.42 [ERROR] 1 Exception encountered:

Engine traceback:
 in select
 in pants.core.goals.package.package_asset
 in pants.backend.python.goals.setup_py.package_python_dist (src/server:grpc-server)
 in pants.engine.internals.graph.transitive_targets
 in pants.engine.internals.graph.transitive_dependency_mapping
 in pants.engine.internals.graph.resolve_targets (src/proto/helloworld.proto)
 in pants.engine.internals.graph.resolve_unexpanded_targets (src/proto/helloworld.proto)
 in pants.engine.internals.graph.resolve_dependencies (src/proto/helloworld.proto)
 in pants.backend.codegen.protobuf.python.python_protobuf_subsystem.inject_dependencies (src/proto/helloworld.proto)
Traceback (most recent call last):
 File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37/lib/python3.7/site-packages/pants/engine/internals/selectors.py", line 705, in native_engine_generator_send
  res = func.send(arg)
 File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37/lib/python3.7/site-packages/pants/backend/codegen/protobuf/python/python_protobuf_subsystem.py", line 129, in inject_dependencies
  disable_inference_option=f"[{python_protobuf.options_scope}].infer_runtime_dependency",
 File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37/lib/python3.7/site-packages/pants/backend/codegen/utils.py", line 56, in find_python_runtime_library_or_raise_error
  else ""
 File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37/lib/python3.7/site-packages/pants/util/strutil.py", line 240, in softwrap
  if text[0] == "\n":
IndexError: string index out of range

(pants) [root@9a5752857756 pants-training]#
But looking at the trace, it is indeed going to generate proto stubs.
h
Oof, that's a stupid bug. Thank you so much for reporting. I'll get out a fix right now
in the meantime, this bug is triggered when you're missing 
python_requirement
targets for 
protobuf
and possibly 
grpcio
. This step: https://www.pantsbuild.org/docs/protobuf-python#step-2-set-up-the-protobuf-and-grpcio-runtime-libraries The bug is only preventing you from getting a good error message, but Pants would still be erroring either way
a
in the meantime, this bug is triggered when you’re missing 
python_requirement
targets for 
protobuf
and possibly 
grpcio
. This step: https://www.pantsbuild.org/docs/protobuf-python#step-2-set-up-the-protobuf-and-grpcio-runtime-libraries
The bug is only preventing you from getting a good error message, but Pants would still be erroring either way
Regarding this, is it mandatory to have grpcio requiements within the same directory (or source root) as proto files? If you looked at my repo, I have the requirements.txt in 
src/server
.
h
it can be defined anywhere 🙂 it looks like you're missing 
protobuf
though
a
The dependency?
h
yep, the runtime library. you have to define which version you want to use. for example, you could add something like 
protobuf>=3.12.1
to 
requirements.txt
(the error message would have made this clear, if it triggered)
a
It will be installed as part of grpcio-tools, wouldn’t that work?
h
Ah that's a good point; Pants's eager check right now is too naive to recognize that, so it requires the explicit protobuf entry. But would you want to open a feature request to clean that up? https://github.com/pantsbuild/pants/issues/new/choose
a
I think I will have to also set grpc=True here.
➕ 1
h
Correct
a
Ah that’s a good point; Pants’s eager check right now is too naive to recognize that, so it requires the explicit protobuf entry. But would you want to open a feature request to clean that up? https://github.com/pantsbuild/pants/issues/new/choose
Cool, I will do that but I suppose it will take time for the feature to be released and so, I anyways have to explicitly mention protobuf.
After the PR is complete, do I need to bootstrap pants again? Or it will automatically start looking at the new code?
h
Yeah, we would not cherry-pick that feature to 2.12 or 2.13 because it's a "new feature". Pants 2.14 is at least a few weeks away from stable release. So in the meantime, you'll want to add 
protobuf
to your requirements.txt
After the PR is complete
Sorry, which PR? To improve pantsbuild/pants, or to add protobuf to your requirements.txt?
a
The PR for that softwrap bugfix
h
Ah. So that PR will only give you a better error message: it won't stop the error from actually happening. To fix the error from happening, you have to add 
protobuf
to your 
requirements.txt
-- Generally, whenever a PR merges, you can set 
PANTS_SHA
to consume that commit. Or wait for a new Pants release https://www.pantsbuild.org/docs/manual-installation#running-pants-from-unreleased-builds
a
Ok, yes I understand that I need to be changing some stuff in requirements and proto BUILD but was just curious how do I point to the new code. Thanks! I will be asking more questions though as I progress 🙂
❤️ 1
For now, I have this question that if I remove a directory from source root, would BUILD file in it be automatically deleted? Can I run 
./pants tailor
multiple times and the BUILDs will be updated automatically (including deletion of BUILD wherever not required anymore)?
h
./pants tailor
is solely additive: it will not remove or modify existing targets
a
Ok, so that needs to be done manually. Alright. Also, if I add a Dockerfile to some source root, I believe tailor would automatically create a docker_image target? What about the 
instructions
though?
h
yeah, 
tailor
should automatically add 
docker_image
targets based on finding 
Dockerfile
. I'm not sure you need to mess with source roots for that to happen though, I'm pretty sure Docker doesn't really use source roots. That's more for JVM and Python For 
instructions
, there is nothing that 
tailor
can autodetect. 
instructions
is solely meant as a way for you to define a 
Dockerfile
in your BUILD file, rather than a concrete 
Dockerfile
on disk
a
Oh yeah, I meant if I added Dockerfile to an existing source root. Thanks. Regarding 
instructions
, I am thinking from CI perspective. So, does that mean we need not execute 
./pants tailor
in CI? I think I was mistaken in thinking that tailor needs to be executed everytime we run the build pipeline.
h
tailor doesn't "need" to be run, but we typically do recommend using 
tailor --check
in CI because it can help make sure your developers aren't forgetting to add Pants targets Whether you use 
instructions
or not does not really impact whether you should use 
tailor --check
in CI. There are other target types like 
python_sources
that 
tailor
adds
a
Yeah, so I meant if we run tailor in CI, there’s no way to add 
instructions
automatically in CI. And so tailor need not (or rather should not) be run in CI. Thanks for informing about 
tailor --check
though. Same goes for other target types e.g. 
python_distribution()
or 
protobuf_sources()
for that matter where we have to write setup and grpc related things manually.
h
there’s no way to add instructions automatically in CI.
Taking a step back, we don't recommend that CI ever automatically adds targets. Your BUILD files should be created locally and checked in to version control That is, you should not run 
./pants tailor
in CI, only 
./pants tailor --check
. Does that make sense?
a
Yup, completely 🙂
Ok, here’s another thing - I executed 
./pants tailor
after moving 
requirements.txt
from 
src/server
to 
3rdparty
(a new directory) and it gave following error:
Copy code
(pants) [root@9a5752857756 pants-training]# ./pants tailor
19:10:09.99 [INFO] Initialization options changed: reinitializing scheduler...
19:10:10.87 [INFO] Scheduler initialized.
19:10:10.99 [ERROR] 1 Exception encountered:

 Exception: Unmatched glob from python_requirements(address="src/server:reqs", description=None, module_mapping=FrozenDict({}), overrides=None, source=requirements.txt, tags=None, type_stubs_module_mapping=FrozenDict({}))'s field `source`: "src/server/requirements.txt"

Do the file(s) exist? If so, check if the file(s) are in your `.gitignore` or the global `pants_ignore` option, which may result in Pants not being able to see the file(s) even though they exist on disk. Refer to <https://www.pantsbuild.org/v2.12/docs/troubleshooting#pants-cannot-find-a-file-in-your-project>.

(pants) [root@9a5752857756 pants-training]#
Do you think its a good idea to always delete BUILD files and then re-tailor them?
I am asking these basic questions so that I have a clear set of instructions to be passed on to my team on how to work with Pants. We are just starting with Pants at work.
💜 2
h
In this case, I would recommend that you move the 
python_requirements()
target generator line from 
src/server/BUILD
to 
3rdparty/BUILD
, since the code it describes 
requirements.txt
has now moved
a
Yeah, I could have done that but I was just trying to see how powerful tailor is.
h
Ah, it is extremely conservative. Once you have checked in BUILD files, we can no longer tell what was hand edited vs auto-generated. So, for safety, we never modify existing BUILD files. We only add to them
a
What if I wanted to switch from 
python_distribution()
to 
pex_binary()
for a certain target? Just add 
pex_binary()
to the existing BUILD and not just ever 
package
the previous 
python_distribution()
target?
h
That would be a manual change. 
./pants tailor
can't figure out what your intent is
a
Oh yes, sorry if the question was not clear. This was not a tailor question but rather a follow up to following:
So, for safety, we never modify existing BUILD files. We only add to them
h
Just add pex_binary() to the existing BUILD and not just ever package the previous python_distribution() target?
Add the 
pex_binary()
target, and then delete the 
python_distribution()
if you don't want to package that way anymore
a
But then wouldn’t it mean that BUILD was modified and something new was not just added to it?
h
To be clear, you are encouraged to manually modify BUILD files. Only 
./pants tailor
won't do it for you
a
Cool, thanks! I’ll be back with more of those 🙂
❤️ 1
Is it mandatory to mention a version for 
protobuf>=3.12.1
and simply 
protobuf
wouldn’t work? The current version of protobuf is 3.20
I ask because the 
package
failed even after mentioning this dependency in 
requirements.txt
.
It failed even when I used the constraint for the version.
I have pushed new changes to repo in case you want to have a look.
h
@happy-kitchen-89482 you were encountering something like that this morning, right? Where the version of 
protobuf
matters?
a
I tried both ways and failed 😔
Good Morning @hundreds-father-404 and @happy-kitchen-89482. So I see the PR was merged and I set my 
PANTS_SHA
before running the 
package
goal again and received following error now:
Copy code
(pants) [root@9a5752857756 pants-training]# export PANTS_SHA=e43d8f655816351d11850b96d972034b2443b496
(pants) [root@9a5752857756 pants-training]# ./pants package src/server:grpc-server
Bootstrapping Pants using /root/miniconda3/envs/pants/bin/python3.7
Installing pantsbuild.pants==2.12.1rc3+gite43d8f65 into a virtual environment at /root/.cache/pants/setup/bootstrap-Linux-x86_64/2.12.1rc3+gite43d8f65_py37
ERROR: Could not find a version that satisfies the requirement pantsbuild.pants==2.12.1rc3+gite43d8f65 (from versions: 0.0.17, 0.0.18, 0.0.20, 0.0.21, 0.0.22, 0.0.23, 0.0.24, 0.0.25, 0.0.26, 0.0.27, 0.0.28, 0.0.29, 0.0.30, 0.0.31, 0.0.32, 0.0.33, 0.0.34, 0.0.35, 0.0.36, 0.0.37, 0.0.38, 0.0.39, 0.0.40, 0.0.41, 0.0.42, 0.0.43, 0.0.44, 0.0.45, 0.0.46, 0.0.47, 0.0.48, 0.0.49, 0.0.50, 0.0.51, 0.0.52, 0.0.53, 0.0.54, 0.0.55, 0.0.56, 0.0.57, 0.0.58, 0.0.59, 0.0.60, 0.0.61, 0.0.62, 0.0.63, 0.0.64, 0.0.65, 0.0.66, 0.0.67, 0.0.68, 0.0.69, 0.0.70, 0.0.71, 0.0.72, 0.0.73, 0.0.74, 0.0.75, 0.0.76, 0.0.77, 0.0.79, 0.0.80, 0.0.81, 0.0.82, 1.0.0, 1.0.1, 1.1.0, 1.2.0, 1.2.1, 1.3.0, 1.4.0, 1.5.0, 1.6.0, 1.7.0, 1.15.0, 1.16.0, 1.17.0, 1.18.0, 1.19.0, 1.20.0, 1.21.0, 1.22.0, 1.23.0, 1.24.0, 1.25.0, 1.26.0, 1.27.0, 1.28.0, 1.29.0, 1.30.0, 1.30.1, 1.30.2, 1.30.3, 1.30.4, 1.30.5rc1, 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2, 2.2.3, 2.2.4, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.4.0, 2.4.1, 2.4.2, 2.5.0, 2.5.1rc5, 2.5.1rc6, 2.5.1, 2.5.2rc0, 2.5.2rc1, 2.5.2rc2, 2.5.2rc3, 2.5.2, 2.6.0rc0, 2.6.0rc1, 2.6.0rc2, 2.6.0rc3, 2.6.0rc4, 2.6.0, 2.6.1rc0, 2.6.1rc1, 2.6.1rc2, 2.6.1rc3, 2.6.1, 2.7.0rc0, 2.7.0rc1, 2.7.0rc2, 2.7.0rc3, 2.7.0rc4, 2.7.0rc5, 2.7.0, 2.7.1rc0, 2.7.1rc1, 2.7.1, 2.7.2rc0, 2.7.2rc1, 2.7.2rc2, 2.7.2, 2.8.0.dev5, 2.8.0rc0, 2.8.0rc1, 2.8.0rc2, 2.8.0rc3, 2.8.0rc4, 2.8.0rc5, 2.8.0rc6, 2.8.0, 2.8.1rc0, 2.8.1rc1, 2.8.1rc2, 2.8.1, 2.9.0.dev0, 2.9.0.dev1, 2.9.0.dev2, 2.9.0.dev3, 2.9.0.dev4, 2.9.0rc0, 2.9.0rc1, 2.9.0rc2, 2.9.0rc3, 2.9.0rc4, 2.9.0rc5, 2.9.0rc6, 2.9.0, 2.9.1rc0, 2.9.1rc1, 2.9.1rc2, 2.9.1, 2.9.2rc0, 2.9.2, 2.10.0.dev0, 2.10.0.dev1, 2.10.0.dev2, 2.10.0.dev3, 2.10.0rc0, 2.10.0rc1, 2.10.0rc2, 2.10.0rc3, 2.10.0rc4, 2.10.0rc5, 2.10.0, 2.10.1rc0, 2.10.1rc1, 2.10.1, 2.11.0.dev0, 2.11.0.dev1, 2.11.0.dev2, 2.11.0.dev3, 2.11.0rc0, 2.11.0rc1, 2.11.0rc2, 2.11.0rc3, 2.11.0rc4, 2.11.0rc5, 2.11.0rc6, 2.11.0, 2.11.1rc0, 2.11.1rc1, 2.11.1rc2, 2.11.1rc3, 2.11.1, 2.12.0.dev0, 2.12.0.dev1, 2.12.0.dev2, 2.12.0.dev3, 2.12.0a0, 2.12.0rc0, 2.12.0rc1, 2.12.0rc2, 2.12.0rc3, 2.12.0, 2.12.1rc0, 2.12.1rc1, 2.12.1rc2, 2.12.1rc3, 2.13.0.dev0, 2.13.0.dev1, 2.13.0.dev2, 2.13.0.dev3, 2.13.0.dev4, 2.13.0.dev5, 2.13.0a0, 2.13.0a1, 2.13.0rc0, 2.13.0rc1, 2.14.0.dev0, 2.14.0.dev1, 2.14.0.dev2, 2.14.0.dev3, 2.14.0.dev4, 2.14.0.dev5, 2.14.0.dev6)
ERROR: No matching distribution found for pantsbuild.pants==2.12.1rc3+gite43d8f65
I set the env variable so that I am able to get a meaningful error message to be able to proceed with the 
package
. I am using RHEL distribution inside a docker container and 2.12.0 was working fine before setting the `PANTS_SHA`:
Copy code
(pants) [root@9a5752857756 pants-training]# cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.6 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.6"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="<https://www.redhat.com/>"
DOCUMENTATION_URL="<https://access.redhat.com/documentation/red_hat_enterprise_linux/8/>"
BUG_REPORT_URL="<https://bugzilla.redhat.com/>"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.6"
(pants) [root@9a5752857756 pants-training]#
Could you guys please help me here? My latest code is pushed in the repo.
👀 1
Here are the last few lines when I run 
package
with `-ldebug`:
Copy code
14:24:25.22 [DEBUG] Completed: Find interpreter for constraints: CPython==3.7.*
14:24:25.22 [DEBUG] Completed: Scheduling: Find interpreter for constraints: CPython==3.7.*
14:24:25.22 [DEBUG] Completed: Find Python interpreter for constraints - Selected /root/miniconda3/envs/pants/bin/python3.7 to run PEXes with.
14:24:25.22 [DEBUG] Completed: Scheduling: Determine Python dependencies for src/server/greeter_server.py
14:24:25.24 [DEBUG] Completed: Generate `python_requirement` targets from requirements.txt
14:24:25.25 [DEBUG] Completed: Find all targets in the project
14:24:25.25 [DEBUG] Completed: Find all Python targets in project
14:24:25.25 [DEBUG] Completed: Creating map of third party targets to Python modules
14:24:25.25 [DEBUG] Completed: Find all Protobuf targets in project
14:24:25.25 [DEBUG] Completed: Creating map of first party Python targets to Python modules
14:24:25.25 [DEBUG] Completed: Creating map of Protobuf targets to generated Python modules
14:24:25.25 [DEBUG] Completed: pants.backend.python.dependency_inference.module_mapper.merge_first_party_module_mappings
14:24:25.26 [DEBUG] Completed: Resolve transitive targets
14:24:25.26 [DEBUG] Completed: `package` goal
14:24:25.26 [DEBUG] computed 1 nodes in 1.616815 seconds. there are 219 total nodes.
14:24:25.26 [ERROR] 1 Exception encountered:

 IndexError: string index out of range

(pants) [root@9a5752857756 pants-training]#
h
@hundreds-father-404 Looks like that fix was merged directly into 2.12.x and is not present on main? Does it not apply to 2.13.x and beyond?
Not sure if we build wheels for SHAs on branches that aren’t 
main
a
I think that’s ok. That fix was for me to be able to get a better error message. In the absence of which I tried debugging the 
package
goal but could not find any relevant info.
h
What are you running in that repo that is failing?
At current HEAD on pants main I was able to run 
./pants package src/client/::
successfully in your repo
And the 
_pb2.py
files are in the resulting pex
a
I am running 
./pants package src/server:grpc-server
Let me try what you are running.
I can’t run 
./pants package src/client/::
Copy code
(pants) [root@9a5752857756 pants-training]# ./pants package src/client/::
19:25:56.77 [ERROR] 1 Exception encountered:

 IndexError: string index out of range

(pants) [root@9a5752857756 pants-training]#
Do you think deleting 
$HOME/.cache/pants/
would help? That would mean I will be bootstrapping pants again.
h
@ancient-rose-27306 can you run 
./pants list ::
please? @happy-kitchen-89482, indeed, the fix is only for the error message to be accurate rather than 
IndexError
. The underlying error is that 
protobuf
is not discoverable by Pants. The 
IndexError
was fixed in an unrelated PR in 2.13 and 2.14 already
h
Yeah, I ran that command against Pants at HEAD, not in 2.12.x
No, deleting .cache/pants will probably not help
👍 1
@ancient-rose-27306 The IndexError issue is fixed in the 2.12 branch but you’ll have to wait for a release from that branch in order to consume it
But I guess we know what the underlying error is and should fix that?
a
Yes, please. As I mentioned, the error doesn’t bother me much because @hundreds-father-404 helped me understand the root cause and I rectified it. What bothers me is even after rectification, I am unable to 
package
successfully.
@hundreds-father-404 here’s the output for `./pants list::`:
Copy code
(pants) [root@9a5752857756 pants-training]# ./pants list ::
20:56:37.47 [INFO] Initializing scheduler...
20:56:37.79 [INFO] Scheduler initialized.
3rdparty:reqs
3rdparty:reqs#grpcio
3rdparty:reqs#grpcio-tools
3rdparty:reqs#protobuf
3rdparty/requirements.txt:reqs
src/client:client
src/client:greeter_client
src/client/greeter_client.py
src/proto:protos
src/proto/helloworld.proto:protos
src/server:server
src/server:docker
src/server:greeter_server
src/server:grpc-server
src/server/greeter_server.py
(pants) [root@9a5752857756 pants-training]#
h
Oh wait, but you're still getting IndexError even with the fix of adding 
protobuf
? Hm. Can you please run one of the commands that fails with 
--print-stacktrace
? (Or set 
print_stacktrace = true
in 
[GLOBAL]
in 
pants.toml
a
The stacktrace output is the same as before:
Copy code
(pants) [root@9a5752857756 pants-training]# ./pants --print-stacktrace package src/server:grpc-server
20:59:01.84 [INFO] Initialization options changed: reinitializing scheduler...
20:59:02.48 [INFO] Scheduler initialized.
20:59:05.24 [ERROR] 1 Exception encountered:

Engine traceback:
 in select
 in pants.core.goals.package.package_asset
 in pants.backend.python.goals.setup_py.package_python_dist (src/server:grpc-server)
 in pants.engine.internals.graph.transitive_targets
 in pants.engine.internals.graph.transitive_dependency_mapping
 in pants.engine.internals.graph.resolve_targets (src/proto/helloworld.proto:protos)
 in pants.engine.internals.graph.resolve_unexpanded_targets (src/proto/helloworld.proto:protos)
 in pants.engine.internals.graph.resolve_dependencies (src/proto/helloworld.proto:protos)
 in pants.backend.codegen.protobuf.python.python_protobuf_subsystem.inject_dependencies (src/proto/helloworld.proto:protos)
Traceback (most recent call last):
 File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37/lib/python3.7/site-packages/pants/engine/internals/selectors.py", line 705, in native_engine_generator_send
  res = func.send(arg)
 File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37/lib/python3.7/site-packages/pants/backend/codegen/protobuf/python/python_protobuf_subsystem.py", line 144, in inject_dependencies
  disable_inference_option=f"[{python_protobuf.options_scope}].infer_runtime_dependency",
 File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37/lib/python3.7/site-packages/pants/backend/codegen/utils.py", line 56, in find_python_runtime_library_or_raise_error
  else ""
 File "/root/.cache/pants/setup/bootstrap-Linux-x86_64/2.12.0_py37/lib/python3.7/site-packages/pants/util/strutil.py", line 240, in softwrap
  if text[0] == "\n":
IndexError: string index out of range

(pants) [root@9a5752857756 pants-training]#
h
Hmmmm that should not still be triggering. I'll clone your repo right now https://github.com/dheerajramchandani/pants-training
Sin_titulo.diff
Sin_titulo.diff
Alright this diff gets the 
python_distribution
building via 
./pants package src:grpc-server
. It looks like there's a bug that for some reason we have to explicitly tell Pants that 
grpcio
provides 
grpc
, which it should recognize by default. I'm going to investigate this bug now Then I also moved where the 
python_distribution
is defined up one directory. Because Pants supports multiple `python_distribution`s in the same repo, there is an algorithm to ensure that there are not multiple ambiguous owners of the same file. So, the error message prompted me to move the target up a BUILD level to avoid that. In the process, Pants wanted me to tweak your source roots. Now, you would import 
server.greeter_server
, whereas before it was only 
greeter_server
. I suspect this is what you want. I kept it that Protobuf is 
helloworld_pb2
rather than 
proto.helloworld_pb2
though
a
Ok, so is there only one BUILD file for both client and server at src level? And one BUILD file each for proto and 3rdparty/requirements?
h
You still have 
src/{proto,client,server}/BUILD
, which are giving metadata for the individual 
.py
and 
.proto
files via the 
protobuf_sources
and 
python_sources
targets You also have 
src/BUILD
, which is where you declare to Pants that you want to build a 
python_distribution
made up of code inside the folder 
src/server
, which itself depends on code from 
src/proto
. Does that make sense?
a
Yeah, I was able to guess that after I wrote to you. But why is that required? And wouldn’t packaging (and later publishing) work without it?
h
why is what required?
a
Defining a BUILD in src?
h
a quirk of the 
python_distribution
target
Then I also moved where the python_distribution is defined up one directory. Because Pants supports multiple python_distributions in the same repo, there is an algorithm to ensure that there are not multiple ambiguous owners of the same file. So, the error message prompted me to move the target up a BUILD level to avoid that.
a
Was this the error message:
Copy code
NoOwnerError: No python_distribution target found to own src/proto/helloworld.proto:protos. Note that the owner must be in or above the owned target's directory, and must depend on it (directly or indirectly). See <https://www.pantsbuild.org/v2.12/docs/python-distributions> for how python_sources targets are mapped to distributions. See <https://www.pantsbuild.org/v2.12/docs/python-distributions>.
I got this after adding in 
grpc
to the requirements.
h
exactly
a
Ok, so I was getting the index error because there was one more requirement missing even after I added 
protobuf
. I got that. Can this 
NoOwnerError
be resolved by defining a 
python_distribution()
for protos separately and not defining a BUILD at src?
h
Ok, so I was getting the index error because there was one more requirement missing even after I added protobuf . I got that.
Which is a bug. You already had 
grpcio
in your 
requirements.txt
, and Pants by default is supposed to know that 
grpcio
provides the module 
grpc
. I'm trying to figure out now why this is not working
a
Yes, I got that from your initial message.
👍 1
h
Can this NoOwnerError be resolved by defining a python_distribution() for protos separately and not defining a BUILD at src?
Yep, that is 100% valid too! Then, Pants will detect that `src/server`'s distribution depends on the 
src/proto
distribution.
a
Cool, this is more intuitive to me because in a world without Pants, I’d first publish a library having the gRPC generated stubs and then have the server import that.
💯 1
h
Exactly, and I think your idea is better than mine. That way, if 
src/client
depends on 
src/proto
also, Pants will detect that it too depends on the distribution in 
src/proto
a
However, I wouldn’t have been able to make any meaning of NoOwnerFound error 🙂.
👍 1
h
Yeah, it took me a minute to figure out too. Good feedback. How would you suggest improving it?
a
Exactly, and I think your idea is better than mine. That way, if 
src/client
depends on 
src/proto
also, Pants will detect that it too depends on the distribution in 
src/proto
The client indeed depends on proto and that’s why I was concerned about a BUILD in src. However, I think that can be overcome by adding in another 
python_distribution()
in 
src/BUILD
for client and we should be fine…no?
h
Ah, then yeah I recommend reverting my change with 
src/BUILD
and defining a 
python_distribution
in 
src/proto
. Also revert my source root changes maybe It's not legal for two `python_distribution`s to "own" the same code, in this case 
src/proto/*
a
It’s not legal for two `python_distribution`s to “own” the same code, in this case 
src/proto/*
Does that mean I can’t have two `python_distribution()`s in 
src/BUILD
(one each for client and server)? This concept of ownership is new to me as I am learning it for the first time now.
h
I think you technically can define all 3 
python_distribution
targets in 
src/BUILD
-- the important thing is that because 
src/proto
is used by both distributions, that you have the dedicated 
python_distribution
for it. That way, 
src/client
and 
src/server
can unambiguously depend on the same thing. Does that make sense? (Btw, this is a fairly wonky part of Pants that is only relevant to the 
python_distribution
target, due to publishing semantics)
That way, src/client and src/server can unambiguously depend on the same thing.
Alternatively, I think you can have either 
src/client
or 
src/server
be the "owner" of the Protobuf file; then the other will have to depend on that one, like 
src/client
must depend on 
src/server
. Which I imagine isn't what you want
a
Yeah right.
Can’t I use a 
src/BUILD
like below:
Copy code
python_distribution(
    name="grpc-server",
    dependencies=["src/server:server"],
    sdist=False,
    provides=python_artifact(
        name="grpc-server",
        version="0.1.0",
    )
)

python_distribution(
    name="grpc-client",
    dependencies=["src/client:client"],
    sdist=False,
    provides=python_artifact(
        name="grpc-client",
        version="0.1.0",
    )
)

python_distribution(
    name="proto-lib",
    dependencies=["src/proto:protos"],
    sdist=False,
    provides=python_artifact(
        name="proto-lib",
        version="0.1.0",
    )
)
Where’s the ambiguity coming from? Each target has their own name. I am not going to use it like this as I prefer separate BUILD files but just for the purpose of understanding. And this change would have me modify the source_roots as only 
src/
, I believe I will have to change the import statements to: 
import proto.helloworld_pb2
import proto.helloworld_pb2_grpc
h
I believe that should work. Is it erroring when you do 
./pants package src:
? (In 2.13, you can simply do 
./pants package src
to say "build everything in this directory")
a
No, I haven’t tried it yet. I was just trying to understand it.
h
I believe I will have to change the import statements to:
Or, set source roots to 
/src
and 
/src/proto
. Pants uses the most precise source root possible for the file in question. Although that might cause more confusion than it's worth
No, I haven’t tried it yet. I was just trying to understand it.
I believe what you have above should work
a
Yes, it worked. Kept the only source root as 
"/src
and modified the import statements. I have three libraries now as opposed to just single wheel before for the server because earlier, I only built server and now, I built src. I believe I could build just one wheel even now.
👍 1
let me try
h
yeah, one wheel should also be valid 🙂
a
Yes, this worked: 
(pants) [root@9a5752857756 pants-training]# ./pants package src:grpc-client
22:22:58.82 [INFO] Wrote dist/grpc_client-0.1.0-py3-none-any.whl
👍 1
Today can be declared as the first day that I am liking Pants 😛
h
I'm really glad you reached out for help! That definitely was not a very good first impression with the 
IndexError
and everything, and I'm glad we can fix this for future folks trying it out
a
I am using 
name="grpc-client"
in my 
python_distribution()
but the whl is getting created with the name grpc_client (
-
replaced by 
_
). This is probably to adhere to python standards.
I’m really glad you reached out for help! That definitely was not a very good first impression with the 
IndexError
and everything, and I’m glad we can fix this for future folks trying it out
I can’t thank you and @happy-kitchen-89482 enough to help me move through this. I am asking very basic questions and you guys are so patient to answer each of them.
❤️ 2
h
We're happy to help! I first got involved with the Pants community because people were super nice to me asking questions when I was a summer intern 🙂 And your feedback and questions are really helpful for us making Pants better for others. It's like when teachers say "if you have a question, probably 10 others have the same question but are too afraid to ask."
🙌 1
oh well that's a huge oversight 😱we did not have 
grpcio
in the default module mapping until Pants 2.14.0.dev5 https://github.com/pantsbuild/pants/pull/16337. I will cherry-pick this fix to 2.12 and 2.13. Thank you again so much for the report!
a
Happy to contribute 🙂
h
s
Hello from 11 days in the future - just wanted to thank you guys for documenting this problem in the thread so thoroughly. I made a quick issue in the Github tracker (and then immediately closed it) just so this could be searched by someone in the future. https://github.com/pantsbuild/pants/issues/16759
h
Thank you Max! 2.12.1 is released with 
grpc
in the defaults, so you can now remove the 
module_mapping
if you want 🙂
Thanks again for the bug report! That really really helps the project
✅ 1
a
Thank you Max 😊
7 Views
Open in Slack
PreviousNext
Powered by