https://pantsbuild.org/ logo
Join Slack
Powered by
I have question about thrift BUILD file. This is m...
# general
f
I have question about thrift BUILD file. This is my BUILD file:
Copy code
thrift_sources(
    name='gen',
    sources=['*.thrift'],
)
When I run ./pants run …, I got this:
Copy code
Could not find include file xxx.thrift
It looks like it couldn’t find each other in the same folder. How do I fix this?
f
have you set the root of the directory tree with the thrift sources as a Pants “source root”?
(you probably need to do that if you haven’t)
f
ok! I will look into it.
My thrift folder is like this:
Copy code
mythrift
├── definitions
│   ├── data
│   │   ├── a.thrift
│   │   ├── b.thrift
│   │   ├── c.thrift
│   │   └── BUILD
│   ├── otherfolder
│   │   └── xxx.thrift
│   ├── anotherfolder
│   │   └── xxx.thrift
│   └── README
└── README.md
And I have specify roots in 
pants.toml
:
Copy code
[source]
root_patterns = [
  'mythrift',
]
But I still get 
Could not find include file a.thrift
error
f
and are your imports of the form 
include 'definitions/data/a.thrift
or 
include 'data/a.thrift'
?
could you also run 
./pants dependencies
on one of the thrift files that imports 
a.thrift
? It may be that we don’t infer dependencies on includes from the same directory.
seeing the dependencies will very quickly reveal whether that is the problem or not
f
oh! I got it. I fix the import path. But the I got this:
Copy code
09:16:22.25 [ERROR] 1 Exception encountered:

  Exception: String("Can only merge Directories with no duplicates, but found 2 duplicate entries in c2thrift/myproject_thrift/log/batch:\n\n`ttypes.py`: 1.) file digest=38cef792a9b7183fcebeb48861ade76a0e0ec58e806e1048806affbd04b43160 size=88071:\n\n#\n# Autogenerated by Thrift Compiler (0.9.3)\n#\n# DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING\n#\n#  options string: py\n#\n\nfrom thrift.Thrift import TType, TMessageType, TException, TApplicationException\n\nfrom thrift.transport import TTransport\nfrom thrift.protocol import TBinaryProtocol, TProtocol\ntry:\n  from thrift.protocol import fastbinary\nexcept:\n  fastbinary = None\n\n\nclass NotificationType:\n  UNKNOWN = 0\n  NEWSLETTER = 1\n\n  _VALUES_TO_NAMES = {\n    0: \"UNKNOWN\",\n    1: \"NEWSLETTER\",\n  }\n\n  _NAMES_TO_VALUES = {\n    \"UNKNOWN\": 0,\n    \"NEWSLETTER\": 1,\n  }\n\n\nclass AutoPricingEventTable:\n  \"\"\"\n  Attributes:\n   - type\n   - vendor_id\n   - listing_id\n   - house_id\n   - core_comp\n   - core_comp_price\n   - map_\n   - msrp\n   - cost_to_myproject\n   - hsr\n   - price\n   - final_price\n   - old_price\n   - est_ship\n   - campaign_id\n   - comp_match_status\n   - ab_test\n   - wc\n   - adj_wc\n   - altPrice\n   - isTradePrice\n   - ts\n   - ab_test_metadata\n  \"\"\"\n\n  thrift_spec = (\n    None, # 0\n    (1, TType.STRING\n... TRUNCATED contents from 88071B to 1024B (Pass -ldebug to see full contents).\n\n`ttypes.py`: 2.) file digest=664b9495c5a60663a0e544057585a75fcc8fb5d181e5ca7752398385472c1a02 size=92706:\n\n#\n# Autogenerated by Thrift Compiler (0.9.3)\n#\n# DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING\n#\n#  options string: py\n#\n\nfrom thrift.Thrift import TType, TMessageType, TException, TApplicationException\nimport myproject_thrift.log.session.ttypes\nimport myproject_thrift.log.batch.ttypes\n\n\nfrom thrift.transport import TTransport\nfrom thrift.protocol import TBinaryProtocol, TProtocol\ntry:\n  from thrift.protocol import fastbinary\nexcept:\n  fastbinary = None\n\n\n\nclass BatchReqLogThrift:\n  \"\"\"\n  Attributes:\n   - request_common\n   - events_auto_pricing\n   - event_email_real_sent\n   - event_email_send\n   - event_email_not_send\n   - event_newsletter\n   - event_email_content\n   - event_personalized_email_content\n   - event_personalized_email_context\n   - event_personalized_email_error\n   - event_personal_message_sents\n   - request_events_thrift\n   - test_selection_map_thrift\n   - event_pro_user_test_selection\n  \"\"\"\n\n  thrift_spec = (\n    None, # 0\n    (1, TType.STRUCT, 'request_common', (myproject_thrift.log.sess\n... TRUNCATED contents from 92706B to 1024B (Pass -ldebug to see full contents).")
f
so two problems potentially then: 1. Thrift supports same directory includes without relying on 
-I
options and Pants needs to infer dependencies in that case. The work-around for now is a source root relative include path. 2. This digest merging issue is probably from Thrift generating same filenames when processing different Thrift files targeting the same Python package. I’ll need to investigate.
f
source root relative include path? Is that mean I have to set the root as 
mythrift
and in each thrift file, I have to use 
include 'definitions/data/a.thrift'
, right?
f
for now, yes
opened https://github.com/pantsbuild/pants/issues/15152 for the same directory dependency inference
opened https://github.com/pantsbuild/pants/issues/15153 for the Digest merging issue
do any of the Thrift files share the same 
namespace py
directive?
also it would help if you could produce a minimal reproduction that demonstrates the issues
I’ve made assumptions about what the issues might be, but those assumptions may not actually be what the issues are.
f
Hmmm, but I used the source root relative include path workaround and it still failed. Same error.
And Yes, some of them share the same 
namespace py
@fast-nail-55400 do you have any suggestion?
Is there any workaround😭
f
I need to see an actual project with the bug. So either the sources of the existing project if that project is open source or else a small project that reproduces the issue that can be shared.
One of my guesses turned out wrong when I tried to investigate, so having an actual project with the bug that can be shared is essential to narrowing down the problem.
f
ok cool, I will work on that to reproduce it
I found out that my error also happened in the old pants. It would complain about it but won’t stop the whole build. In this version, it will just stop. And I found this: https://issues.apache.org/jira/browse/THRIFT-515
I will try the solution discussed in the link.
After I merge some thrift files into one, I ran again and got this:
Copy code
04:41:45.27 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.run.run
  in pants.backend.python.goals.run_pex_binary.create_pex_binary_run_request (bin:clean_config)
  in pants.backend.python.util_rules.python_sources.prepare_python_sources
  in pants.core.util_rules.source_files.determine_source_files
  in pants.engine.internals.graph.hydrate_sources (c2thrift/definitions/data/batch_logs.thrift:gen)
  in pants.backend.codegen.thrift.apache.python.rules.generate_python_from_thrift
  in pants.backend.codegen.thrift.apache.rules.generate_apache_thrift_sources
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/2.10.0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 290, in fallible_to_exec_result_or_raise
    process_cleanup=process_cleanup.val,
pants.engine.process.ProcessExecutionFailure: Process 'Generating Python sources from xxx.thrift:xxx.' failed with exit code -11.
stdout:

stderr:
What is exit code -11? I don’t know how to fix this, cuz there isn’t any error messages.
f
negative exit codes are from the process receiving a signal
-11 is probably SIGSEGV (segmentation fault)
f
ok I will retry it
hmm any idea why this seg fault happen?
f
does it occur if you manually run 
thift
against the merged file?
f
ok it also encounter seg fault lol
I find a way to bypass this
https://issues.apache.org/jira/browse/THRIFT-515 Also my previous problem is the same as this, so you don’t need to open that issue.
Open in Slack
PreviousNext
Powered by