https://pantsbuild.org/ logo
#general
Title
# general
f

freezing-area-97131

04/14/2022, 8:21 PM
I have question about thrift BUILD file. This is my BUILD file:
Copy code
thrift_sources(
    name='gen',
    sources=['*.thrift'],
)
When I run ./pants run …, I got this:
Copy code
Could not find include file xxx.thrift
It looks like it couldn’t find each other in the same folder. How do I fix this?
f

fast-nail-55400

04/14/2022, 8:22 PM
have you set the root of the directory tree with the thrift sources as a Pants “source root”?
(you probably need to do that if you haven’t)
f

freezing-area-97131

04/14/2022, 8:27 PM
ok! I will look into it.
My thrift folder is like this:
Copy code
mythrift
├── definitions
│   ├── data
│   │   ├── a.thrift
│   │   ├── b.thrift
│   │   ├── c.thrift
│   │   └── BUILD
│   ├── otherfolder
│   │   └── xxx.thrift
│   ├── anotherfolder
│   │   └── xxx.thrift
│   └── README
└── README.md
And I have specify roots in
pants.toml
:
Copy code
[source]
root_patterns = [
  'mythrift',
]
But I still get
Could not find include file a.thrift
error
f

fast-nail-55400

04/15/2022, 4:12 PM
and are your imports of the form
include 'definitions/data/a.thrift
or
include 'data/a.thrift'
?
could you also run
./pants dependencies
on one of the thrift files that imports
a.thrift
? It may be that we don’t infer dependencies on includes from the same directory.
seeing the dependencies will very quickly reveal whether that is the problem or not
f

freezing-area-97131

04/15/2022, 4:17 PM
oh! I got it. I fix the import path. But the I got this:
Copy code
09:16:22.25 [ERROR] 1 Exception encountered:

  Exception: String("Can only merge Directories with no duplicates, but found 2 duplicate entries in c2thrift/myproject_thrift/log/batch:\n\n`ttypes.py`: 1.) file digest=38cef792a9b7183fcebeb48861ade76a0e0ec58e806e1048806affbd04b43160 size=88071:\n\n#\n# Autogenerated by Thrift Compiler (0.9.3)\n#\n# DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING\n#\n#  options string: py\n#\n\nfrom thrift.Thrift import TType, TMessageType, TException, TApplicationException\n\nfrom thrift.transport import TTransport\nfrom thrift.protocol import TBinaryProtocol, TProtocol\ntry:\n  from thrift.protocol import fastbinary\nexcept:\n  fastbinary = None\n\n\nclass NotificationType:\n  UNKNOWN = 0\n  NEWSLETTER = 1\n\n  _VALUES_TO_NAMES = {\n    0: \"UNKNOWN\",\n    1: \"NEWSLETTER\",\n  }\n\n  _NAMES_TO_VALUES = {\n    \"UNKNOWN\": 0,\n    \"NEWSLETTER\": 1,\n  }\n\n\nclass AutoPricingEventTable:\n  \"\"\"\n  Attributes:\n   - type\n   - vendor_id\n   - listing_id\n   - house_id\n   - core_comp\n   - core_comp_price\n   - map_\n   - msrp\n   - cost_to_myproject\n   - hsr\n   - price\n   - final_price\n   - old_price\n   - est_ship\n   - campaign_id\n   - comp_match_status\n   - ab_test\n   - wc\n   - adj_wc\n   - altPrice\n   - isTradePrice\n   - ts\n   - ab_test_metadata\n  \"\"\"\n\n  thrift_spec = (\n    None, # 0\n    (1, TType.STRING\n... TRUNCATED contents from 88071B to 1024B (Pass -ldebug to see full contents).\n\n`ttypes.py`: 2.) file digest=664b9495c5a60663a0e544057585a75fcc8fb5d181e5ca7752398385472c1a02 size=92706:\n\n#\n# Autogenerated by Thrift Compiler (0.9.3)\n#\n# DO NOT EDIT UNLESS YOU ARE SURE THAT YOU KNOW WHAT YOU ARE DOING\n#\n#  options string: py\n#\n\nfrom thrift.Thrift import TType, TMessageType, TException, TApplicationException\nimport myproject_thrift.log.session.ttypes\nimport myproject_thrift.log.batch.ttypes\n\n\nfrom thrift.transport import TTransport\nfrom thrift.protocol import TBinaryProtocol, TProtocol\ntry:\n  from thrift.protocol import fastbinary\nexcept:\n  fastbinary = None\n\n\n\nclass BatchReqLogThrift:\n  \"\"\"\n  Attributes:\n   - request_common\n   - events_auto_pricing\n   - event_email_real_sent\n   - event_email_send\n   - event_email_not_send\n   - event_newsletter\n   - event_email_content\n   - event_personalized_email_content\n   - event_personalized_email_context\n   - event_personalized_email_error\n   - event_personal_message_sents\n   - request_events_thrift\n   - test_selection_map_thrift\n   - event_pro_user_test_selection\n  \"\"\"\n\n  thrift_spec = (\n    None, # 0\n    (1, TType.STRUCT, 'request_common', (myproject_thrift.log.sess\n... TRUNCATED contents from 92706B to 1024B (Pass -ldebug to see full contents).")
f

fast-nail-55400

04/15/2022, 4:23 PM
so two problems potentially then: 1. Thrift supports same directory includes without relying on
-I
options and Pants needs to infer dependencies in that case. The work-around for now is a source root relative include path. 2. This digest merging issue is probably from Thrift generating same filenames when processing different Thrift files targeting the same Python package. I’ll need to investigate.
f

freezing-area-97131

04/15/2022, 4:54 PM
source root relative include path? Is that mean I have to set the root as
mythrift
and in each thrift file, I have to use
include 'definitions/data/a.thrift'
, right?
f

fast-nail-55400

04/15/2022, 8:43 PM
for now, yes
opened https://github.com/pantsbuild/pants/issues/15152 for the same directory dependency inference
opened https://github.com/pantsbuild/pants/issues/15153 for the Digest merging issue
do any of the Thrift files share the same
namespace py
directive?
also it would help if you could produce a minimal reproduction that demonstrates the issues
I’ve made assumptions about what the issues might be, but those assumptions may not actually be what the issues are.
f

freezing-area-97131

04/16/2022, 3:20 PM
Hmmm, but I used the source root relative include path workaround and it still failed. Same error.
And Yes, some of them share the same
namespace py
@fast-nail-55400 do you have any suggestion?
Is there any workaround😭
f

fast-nail-55400

04/19/2022, 12:49 AM
I need to see an actual project with the bug. So either the sources of the existing project if that project is open source or else a small project that reproduces the issue that can be shared.
One of my guesses turned out wrong when I tried to investigate, so having an actual project with the bug that can be shared is essential to narrowing down the problem.
f

freezing-area-97131

04/19/2022, 12:51 AM
ok cool, I will work on that to reproduce it
I found out that my error also happened in the old pants. It would complain about it but won’t stop the whole build. In this version, it will just stop. And I found this: https://issues.apache.org/jira/browse/THRIFT-515
I will try the solution discussed in the link.
After I merge some thrift files into one, I ran again and got this:
Copy code
04:41:45.27 [ERROR] 1 Exception encountered:

Engine traceback:
  in select
  in pants.core.goals.run.run
  in pants.backend.python.goals.run_pex_binary.create_pex_binary_run_request (bin:clean_config)
  in pants.backend.python.util_rules.python_sources.prepare_python_sources
  in pants.core.util_rules.source_files.determine_source_files
  in pants.engine.internals.graph.hydrate_sources (c2thrift/definitions/data/batch_logs.thrift:gen)
  in pants.backend.codegen.thrift.apache.python.rules.generate_python_from_thrift
  in pants.backend.codegen.thrift.apache.rules.generate_apache_thrift_sources
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/home/cedriczheng/.cache/pants/setup/bootstrap-Linux-x86_64/2.10.0_py37/lib/python3.7/site-packages/pants/engine/process.py", line 290, in fallible_to_exec_result_or_raise
    process_cleanup=process_cleanup.val,
pants.engine.process.ProcessExecutionFailure: Process 'Generating Python sources from xxx.thrift:xxx.' failed with exit code -11.
stdout:

stderr:
What is exit code -11? I don’t know how to fix this, cuz there isn’t any error messages.
f

fast-nail-55400

04/20/2022, 2:38 PM
negative exit codes are from the process receiving a signal
-11 is probably SIGSEGV (segmentation fault)
f

freezing-area-97131

04/20/2022, 3:07 PM
ok I will retry it
hmm any idea why this seg fault happen?
f

fast-nail-55400

04/20/2022, 5:45 PM
does it occur if you manually run
thift
against the merged file?
f

freezing-area-97131

04/21/2022, 11:51 AM
ok it also encounter seg fault lol
I find a way to bypass this
https://issues.apache.org/jira/browse/THRIFT-515 Also my previous problem is the same as this, so you don’t need to open that issue.