Bug 1001483

Summary: riak crash when listing keys
Product: [Fedora] Fedora Reporter: Maciek Borzecki <maciek.borzecki>
Component: riakAssignee: Peter Lemenkov <lemenkov>
Status: CLOSED EOL QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 19CC: lemenkov, patrice
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-02-18 11:25:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Maciek Borzecki 2013-08-27 07:04:40 UTC
Description of problem:
Riak crashes whenever client attempts to list keys in a bucket

Version-Release number of selected component (if applicable):
erlang-riak_api-1.3.2-2.fc19.x86_64
erlang-riak_client-1.3.3-1.fc19.x86_64
erlang-riak_control-1.3.2-3.fc19.x86_64
erlang-riak_core-1.3.2-1.fc19.x86_64
erlang-riak_err-1.0.1-4.fc19.x86_64
erlang-riak_kv-1.3.2-1.fc19.x86_64
erlang-riak_pb-1.3.3-1.fc19.x86_64
erlang-riak_pipe-1.3.2-1.fc19.x86_64
erlang-riak_search-1.3.2-1.fc19.x86_64
erlang-riak_sysmon-1.1.3-2.fc19.x86_64
erlang-riaknostic-1.1.0-1.fc19.x86_64
riak-1.3.2-1.fc19.x86_64

How reproducible:
always

Steps to Reproduce:
1. start riak
2. execute:
{ok, Pid} = riakc_pb_socket:start_link("localhost", 8087).
% store something in DB
Obj1 = riakc_obj:new(<<"test">>, <<"test-key">>, "test value").
riakc_pb_socket:put(Pid, Obj1).
% verify that it's there
{ok, Data} = riakc_pb_socket:get(Pid, <<"test">>, <<"test-key">>).
binary_to_term(riakc_obj:get_value(Data)).
% list buckets - works
riakc_pb_socket:list_buckets(Pid).
% list keys - fails, error report is shown
riakc_pb_socket:list_keys(Pid, <<"test">>).

=ERROR REPORT==== 27-Aug-2013::08:57:46 ===
** Generic server <0.34.0> terminating 
** Last message in was {tcp_closed,#Port<0.2589>}
** When Server state == {state,"localhost",8087,false,false,undefined,
                               undefined,
                               {[],[]},
                               1,[],infinity,100}
** Reason for termination == 
** disconnected
** exception exit: disconnected

3. verify that crash is found in /var/log/riak/error.log
Example log from error.log:
2013-08-27 08:49:34.113 [error] <0.13366.1> gen_server <0.13366.1> terminated with reason: {error,function_clause,[{riak_client,stream_list_keys,[<<"test">>,60000,<0.13366.1>],[{file,"src/riak_client.erl"},{line,348}]},{riak_kv_pb_bucket,process,2,[{file,"src/riak_kv_pb_bucket.erl"},{line,86}]},{riak_api_pb_server,process_message,4,[{file,"src/riak_api_pb_server.erl"},{line,223}]},{riak_api_pb_server,handle_message,3,[{file,"src/riak_api_pb_server.erl"},{line,200}]},{riak_api_pb_server,decode_buffer,1,[{file,"src/riak_api_pb_server.erl"},{line,172}]},{gen_server,handle_msg,5,[{file,...},...]},...]}
2013-08-27 08:49:34.187 [error] <0.13366.1> CRASH REPORT Process <0.13366.1> with 0 neighbours exited with reason: {error,function_clause,[{riak_client,stream_list_keys,[<<"test">>,60000,<0.13366.1>],[{file,"src/riak_client.erl"},{line,348}]},{riak_kv_pb_bucket,process,2,[{file,"src/riak_kv_pb_bucket.erl"},{line,86}]},{riak_api_pb_server,process_message,4,[{file,"src/riak_api_pb_server.erl"},{line,223}]},{riak_api_pb_server,handle_message,3,[{file,"src/riak_api_pb_server.erl"},{line,200}]},{riak_api_pb_server,decode_buffer,1,[{file,"src/riak_api_pb_server.erl"},{line,172}]},{gen_server,handle_msg,5,[{file,...},...]},...]} in gen_server:terminate/6 line 744
2013-08-27 08:49:34.238 [error] <0.277.0> Supervisor riak_api_pb_sup had child undefined started with {riak_api_pb_server,start_link,undefined} at <0.13366.1> exit with reason {error,function_clause,[{riak_client,stream_list_keys,[<<"test">>,60000,<0.13366.1>],[{file,"src/riak_client.erl"},{line,348}]},{riak_kv_pb_bucket,process,2,[{file,"src/riak_kv_pb_bucket.erl"},{line,86}]},{riak_api_pb_server,process_message,4,[{file,"src/riak_api_pb_server.erl"},{line,223}]},{riak_api_pb_server,handle_message,3,[{file,"src/riak_api_pb_server.erl"},{line,200}]},{riak_api_pb_server,decode_buffer,1,[{file,"src/riak_api_pb_server.erl"},{line,172}]},{gen_server,handle_msg,5,[{file,...},...]},...]} in context child_terminated

backtrace goes through riak_api -> riak_kv

I tried changing backends from bitcast to eleveldb, but the effect is the same.

Comment 1 patrice 2013-10-09 09:18:19 UTC
I encountered exactly the same issue.
My scripts work fine before upgrade to Riak coming from Fedora packages.

--- ERROR ---
1> {ok, Pid} = riakc_pb_socket:start_link("localhost", 8081).
{ok,<0.34.0>}
2> riakc_pb_socket:list_keys(Pid, <<"TestBucket">>).

=ERROR REPORT==== 9-Oct-2013::10:37:30 ===
** Generic server <0.34.0> terminating
** Last message in was {tcp_closed,#Port<0.1593>}
** When Server state == {state,"computer1",8081,false,false,undefined,
                               undefined,
                               {[],[]},
                               1,[],infinity,100}
** Reason for termination ==
** disconnected
** exception exit: disconnected


--- PACKAGES ---

erlang-riak_search-1.3.2-1.fc19.x86_64
erlang-riak_core-1.3.2-1.fc19.x86_64
erlang-riak_client-1.3.3-1.fc19.x86_64
erlang-riak_pb-1.3.3-1.fc19.x86_64
riak-1.3.2-1.fc19.x86_64
erlang-riak_err-1.0.1-4.fc19.x86_64
erlang-riak_api-1.3.2-2.fc19.x86_64
erlang-riak_control-1.3.2-3.fc19.x86_64
erlang-riak_sysmon-1.1.3-2.fc19.x86_64
erlang-riak_pipe-1.3.2-1.fc19.x86_64
erlang-riaknostic-1.1.0-1.fc19.x86_64
erlang-riak_kv-1.3.2-1.fc19.x86_64

Tested with
versions from installed packages :
  protobuffs-0.8.0 riak_pb-1.3.3 riakc-1.3.3
but also with:
  protobuffs-0.8.0 riak_pb-1.3.0 riakc-1.3.0
and from fresh github.

And the issue is always present.

--- LOG /var/log/riak/error.log ---

(riak-vm.xx.xx)1> 10:36:16.187 [error] CRASH REPORT Process <0.12870.38> with 0 neighbours exited with reason: {error,function_clause,[{riak_client,stream_list_keys,[<<"TestBucket">>,60000,<0.12870.38>],[{file,"src/riak_client.erl"},{line,348}]},{riak_kv_pb_bucket,process,2,[{file,"src/riak_kv_pb_bucket.erl"},{line,86}]},{riak_api_pb_server,process_message,4,[{file,"src/riak_api_pb_server.erl"},{line,223}]},{riak_api_pb_server,handle_message,3,[{file,"src/riak_api_pb_server.erl"},{line,200}]},{riak_api_pb_server,decode_buffer,1,[{file,"src/riak_api_pb_server.erl"},{line,172}]},{gen_server,handle_msg,5,[{file,...},...]},...]} in gen_server:terminate/6 line 744
(riak-vm.xx.xx)1> 10:36:16.190 [error] Supervisor riak_api_pb_sup had child undefined started with {riak_api_pb_server,start_link,undefined} at <0.12870.38> exit with reason {error,function_clause,[{riak_client,stream_list_keys,[<<"TestBucket">>,60000,<0.12870.38>],[{file,"src/riak_client.erl"},{line,348}]},{riak_kv_pb_bucket,process,2,[{file,"src/riak_kv_pb_bucket.erl"},{line,86}]},{riak_api_pb_server,process_message,4,[{file,"src/riak_api_pb_server.erl"},{line,223}]},{riak_api_pb_server,handle_message,3,[{file,"src/riak_api_pb_server.erl"},{line,200}]},{riak_api_pb_server,decode_buffer,1,[{file,"src/riak_api_pb_server.erl"},{line,172}]},{gen_server,handle_msg,5,[{file,...},...]},...]} in context child_terminated

Comment 2 Fedora End Of Life 2015-01-09 22:16:28 UTC
This message is a notice that Fedora 19 is now at end of life. Fedora 
has stopped maintaining and issuing updates for Fedora 19. It is 
Fedora's policy to close all bug reports from releases that are no 
longer maintained. Approximately 4 (four) weeks from now this bug will
be closed as EOL if it remains open with a Fedora 'version' of '19'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 19 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 3 Fedora End Of Life 2015-02-18 11:25:16 UTC
Fedora 19 changed to end-of-life (EOL) status on 2015-01-06. Fedora 19 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.