1726653 – [RGW]: bucket list terminates with a 'ceph::buffer::end_of_buffer'

Bug 1726653 - [RGW]: bucket list terminates with a 'ceph::buffer::end_of_buffer'

Summary: [RGW]: bucket list terminates with a 'ceph::buffer::end_of_buffer'

Keywords:
Status:	CLOSED NOTABUG
Alias:	None
Product:	Red Hat Ceph Storage
Classification:	Red Hat Storage
Component:	RGW
Sub Component:
Version:	3.2
Hardware:	x86_64
OS:	Linux
Priority:	unspecified
Severity:	medium
Target Milestone:	z1
Target Release:	3.3
Assignee:	J. Eric Ivancich
QA Contact:	ceph-qe-bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1727980
TreeView+	depends on / blocked

Reported:	2019-07-03 11:17 UTC by Tejas
Modified:	2019-07-24 12:28 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-07-24 12:28:27 UTC
Embargoed:
Dependent Products:

Attachments	(Terms of Use)

Description Tejas 2019-07-03 11:17:50 UTC

Description of problem:
   On ceph version 12.2.8-128.el7cp, tried a bucket list command of a specific bucket with the --max-entries greater than the max num of objects in the bucket.


~]# radosgw-admin bucket list --bucket mybucket1 --max-entries=10000000  >mb1

terminate called after throwing an instance of 'ceph::buffer::end_of_buffer'
  what():  buffer::end_of_buffer
*** Caught signal (Aborted) **
 in thread 7f3e603d8700 thread_name:msgr-worker-2
 ceph version 12.2.8-128.el7cp (030358773c5213a14c1444a5147258672b2dc15f) luminous (stable)
 1: (()+0x4b9b21) [0x560aa3f2fb21]
 2: (()+0xf5d0) [0x7f3e709665d0]
 3: (gsignal()+0x37) [0x7f3e656f3207]
 4: (abort()+0x148) [0x7f3e656f48f8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f3e660027d5]
 6: (()+0x5e746) [0x7f3e66000746]
 7: (()+0x5e773) [0x7f3e66000773]
 8: (()+0xb5105) [0x7f3e66057105]
 9: (()+0x7dd5) [0x7f3e7095edd5]
 10: (clone()+0x6d) [0x7f3e657baead]
2019-07-03 10:55:50.121836 7f3e603d8700 -1 *** Caught signal (Aborted) **
 in thread 7f3e603d8700 thread_name:msgr-worker-2

 ceph version 12.2.8-128.el7cp (030358773c5213a14c1444a5147258672b2dc15f) luminous (stable)
 1: (()+0x4b9b21) [0x560aa3f2fb21]
 2: (()+0xf5d0) [0x7f3e709665d0]
 3: (gsignal()+0x37) [0x7f3e656f3207]
 4: (abort()+0x148) [0x7f3e656f48f8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f3e660027d5]
 6: (()+0x5e746) [0x7f3e66000746]
 7: (()+0x5e773) [0x7f3e66000773]
 8: (()+0xb5105) [0x7f3e66057105]
 9: (()+0x7dd5) [0x7f3e7095edd5]
 10: (clone()+0x6d) [0x7f3e657baead]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
  -500> 2019-07-03 10:15:31.786848 7f3e72395e00 10 monclient(hunting): picked mon.noname-c con 0x560aa5b0b0f0 addr 10.8.130.212:6789/0
  -499> 2019-07-03 10:15:31.786870 7f3e72395e00 10 monclient(hunting): picked mon.noname-a con 0x560aa5b0e700 addr 10.8.130.210:6789/0
  -498> 2019-07-03 10:15:31.786915 7f3e72395e00 10 monclient(hunting): _renew_subs
  -497> 2019-07-03 10:15:31.786924 7f3e72395e00 10 monclient(hunting): authenticate will time out at 2019-07-03 10:20:31.786922
  -496> 2019-07-03 10:15:31.787682 7f3e5f3d6700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  -495> 2019-07-03 10:15:31.787720 7f3e5f3d6700 10 monclient(hunting):  got monmap 1, mon.noname-a is now rank -1
  -494> 2019-07-03 10:15:31.787724 7f3e5f3d6700 10 monclient(hunting): dump:
epoch 1
fsid 3f57e404-2a88-46e9-b9c4-1fb9b9ff171e
last_changed 2019-05-21 14:21:38.022975
created 2019-05-21 14:21:38.022975
0: 10.8.130.210:6789/0 mon.extensa010
1: 10.8.130.211:6789/0 mon.extensa011
2: 10.8.130.212:6789/0 mon.extensa012

  -493> 2019-07-03 10:15:31.787773 7f3e5f3d6700 10 monclient(hunting): my global_id is 828441
  -492> 2019-07-03 10:15:31.787877 7f3e5f3d6700 10 monclient(hunting): handle_monmap mon_map magic: 0 v1
  -491> 2019-07-03 10:15:31.787890 7f3e5f3d6700 10 monclient(hunting):  got monmap 1, mon.extensa012 is now rank 2
  -490> 2019-07-03 10:15:31.787894 7f3e5f3d6700 10 monclient(hunting): dump:
epoch 1
fsid 3f57e404-2a88-46e9-b9c4-1fb9b9ff171e
last_changed 2019-05-21 14:21:38.022975
created 2019-05-21 14:21:38.022975
0: 10.8.130.210:6789/0 mon.extensa010
1: 10.8.130.211:6789/0 mon.extensa011
2: 10.8.130.212:6789/0 mon.extensa012

  -489> 2019-07-03 10:15:31.787908 7f3e5f3d6700 10 monclient(hunting): my global_id is 455393
  -488> 2019-07-03 10:15:31.788531 7f3e5f3d6700  1 monclient: found mon.extensa010
  -487> 2019-07-03 10:15:31.788563 7f3e5f3d6700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -486> 2019-07-03 10:15:31.788586 7f3e72395e00  5 monclient: authenticate success, global_id 828441
  -485> 2019-07-03 10:15:31.788601 7f3e72395e00 10 monclient: _renew_subs
  -484> 2019-07-03 10:15:31.788605 7f3e72395e00 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -483> 2019-07-03 10:15:31.788675 7f3e72395e00 10 monclient: _renew_subs
  -482> 2019-07-03 10:15:31.788680 7f3e72395e00 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -481> 2019-07-03 10:15:31.788735 7f3e72395e00  1 librados: init done
  -480> 2019-07-03 10:15:31.788750 7f3e5f3d6700 10 monclient: handle_monmap mon_map magic: 0 v1
  -479> 2019-07-03 10:15:31.788775 7f3e72395e00  5 asok(0x560aa59b7280) register_command cr dump hook 0x560aa5b13318
  -478> 2019-07-03 10:15:31.788784 7f3e5f3d6700 10 monclient:  got monmap 1, mon.extensa010 is now rank 0
  -477> 2019-07-03 10:15:31.788789 7f3e5f3d6700 10 monclient: dump:
epoch 1
fsid 3f57e404-2a88-46e9-b9c4-1fb9b9ff171e
last_changed 2019-05-21 14:21:38.022975
created 2019-05-21 14:21:38.022975
0: 10.8.130.210:6789/0 mon.extensa010
1: 10.8.130.211:6789/0 mon.extensa011
2: 10.8.130.212:6789/0 mon.extensa012

  -476> 2019-07-03 10:15:31.788892 7f3e5f3d6700  4 mgrc handle_mgr_map Got map version 11
  -475> 2019-07-03 10:15:31.788896 7f3e5f3d6700  4 mgrc handle_mgr_map Active mgr is now 10.8.130.211:6800/27532
  -474> 2019-07-03 10:15:31.788903 7f3e5f3d6700  4 mgrc reconnect Starting new session with 10.8.130.211:6800/27532
  -473> 2019-07-03 10:15:31.788917 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -472> 2019-07-03 10:15:41.786876 7f3e5e3d4700 10 monclient: tick
  -471> 2019-07-03 10:15:46.789049 7f3e5fbd7700 10 monclient: _renew_subs
  -470> 2019-07-03 10:15:46.789068 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -469> 2019-07-03 10:15:51.787055 7f3e5e3d4700 10 monclient: tick
  -468> 2019-07-03 10:15:53.789012 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -467> 2019-07-03 10:16:01.787177 7f3e5e3d4700 10 monclient: tick
  -466> 2019-07-03 10:16:11.787333 7f3e5e3d4700 10 monclient: tick
  -465> 2019-07-03 10:16:15.789123 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -464> 2019-07-03 10:16:21.787469 7f3e5e3d4700 10 monclient: tick
  -463> 2019-07-03 10:16:31.787589 7f3e5e3d4700 10 monclient: tick
  -462> 2019-07-03 10:16:37.789161 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -461> 2019-07-03 10:16:41.787739 7f3e5e3d4700 10 monclient: tick
  -460> 2019-07-03 10:16:51.787886 7f3e5e3d4700 10 monclient: tick
  -459> 2019-07-03 10:16:59.789244 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -458> 2019-07-03 10:17:01.788028 7f3e5e3d4700 10 monclient: tick
  -457> 2019-07-03 10:17:01.790986 7f3e5fbd7700 10 monclient: _renew_subs
  -456> 2019-07-03 10:17:01.790995 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -455> 2019-07-03 10:17:06.791140 7f3e5fbd7700 10 monclient: _renew_subs
  -454> 2019-07-03 10:17:06.791152 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -453> 2019-07-03 10:17:11.788178 7f3e5e3d4700 10 monclient: tick
  -452> 2019-07-03 10:17:21.788327 7f3e5e3d4700 10 monclient: tick
  -451> 2019-07-03 10:17:21.789350 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -450> 2019-07-03 10:17:31.788470 7f3e5e3d4700 10 monclient: tick
  -449> 2019-07-03 10:17:41.788618 7f3e5e3d4700 10 monclient: tick
  -448> 2019-07-03 10:17:43.789479 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -447> 2019-07-03 10:17:51.788750 7f3e5e3d4700 10 monclient: tick
  -446> 2019-07-03 10:18:01.788901 7f3e5e3d4700 10 monclient: tick
  -445> 2019-07-03 10:18:05.789585 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -444> 2019-07-03 10:18:11.789015 7f3e5e3d4700 10 monclient: tick
  -443> 2019-07-03 10:18:16.792862 7f3e5fbd7700 10 monclient: _renew_subs
  -442> 2019-07-03 10:18:16.792877 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -441> 2019-07-03 10:18:21.789145 7f3e5e3d4700 10 monclient: tick
  -440> 2019-07-03 10:18:21.793034 7f3e5fbd7700 10 monclient: _renew_subs
  -439> 2019-07-03 10:18:21.793048 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -438> 2019-07-03 10:18:27.789657 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -437> 2019-07-03 10:18:31.789291 7f3e5e3d4700 10 monclient: tick
  -436> 2019-07-03 10:18:41.789444 7f3e5e3d4700 10 monclient: tick
  -435> 2019-07-03 10:18:46.793662 7f3e5fbd7700 10 monclient: _renew_subs
  -434> 2019-07-03 10:18:46.793670 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -433> 2019-07-03 10:18:49.789726 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -432> 2019-07-03 10:18:51.789607 7f3e5e3d4700 10 monclient: tick
  -431> 2019-07-03 10:18:51.793801 7f3e5fbd7700 10 monclient: _renew_subs
  -430> 2019-07-03 10:18:51.793815 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -429> 2019-07-03 10:19:01.789736 7f3e5e3d4700 10 monclient: tick
  -428> 2019-07-03 10:19:11.789837 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -427> 2019-07-03 10:19:11.789866 7f3e5e3d4700 10 monclient: tick
  -426> 2019-07-03 10:19:21.790001 7f3e5e3d4700 10 monclient: tick
  -425> 2019-07-03 10:19:31.790137 7f3e5e3d4700 10 monclient: tick
  -424> 2019-07-03 10:19:33.789956 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -423> 2019-07-03 10:19:41.790282 7f3e5e3d4700 10 monclient: tick
  -422> 2019-07-03 10:19:51.790426 7f3e5e3d4700 10 monclient: tick
  -421> 2019-07-03 10:19:55.790025 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -420> 2019-07-03 10:19:56.795401 7f3e5fbd7700 10 monclient: _renew_subs
  -419> 2019-07-03 10:19:56.795407 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -418> 2019-07-03 10:20:01.790556 7f3e5e3d4700 10 monclient: tick
  -417> 2019-07-03 10:20:01.795559 7f3e5fbd7700 10 monclient: _renew_subs
  -416> 2019-07-03 10:20:01.795573 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -415> 2019-07-03 10:20:11.790691 7f3e5e3d4700 10 monclient: tick
  -414> 2019-07-03 10:20:17.790139 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -413> 2019-07-03 10:20:21.790844 7f3e5e3d4700 10 monclient: tick
  -412> 2019-07-03 10:20:31.791005 7f3e5e3d4700 10 monclient: tick
  -411> 2019-07-03 10:20:31.796375 7f3e5fbd7700 10 monclient: _renew_subs
  -410> 2019-07-03 10:20:31.796383 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -409> 2019-07-03 10:20:36.796563 7f3e5fbd7700 10 monclient: _renew_subs
  -408> 2019-07-03 10:20:36.796579 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -407> 2019-07-03 10:20:39.790246 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -406> 2019-07-03 10:20:41.791175 7f3e5e3d4700 10 monclient: tick
  -405> 2019-07-03 10:20:51.791286 7f3e5e3d4700 10 monclient: tick
  -404> 2019-07-03 10:21:01.790346 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -403> 2019-07-03 10:21:01.791432 7f3e5e3d4700 10 monclient: tick
  -402> 2019-07-03 10:21:11.791563 7f3e5e3d4700 10 monclient: tick
  -401> 2019-07-03 10:21:21.791748 7f3e5e3d4700 10 monclient: tick
  -400> 2019-07-03 10:21:23.790465 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -399> 2019-07-03 10:21:31.791889 7f3e5e3d4700 10 monclient: tick
  -398> 2019-07-03 10:21:41.792024 7f3e5e3d4700 10 monclient: tick
  -397> 2019-07-03 10:21:45.790566 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -396> 2019-07-03 10:21:51.792158 7f3e5e3d4700 10 monclient: tick
  -395> 2019-07-03 10:21:56.799032 7f3e5fbd7700 10 monclient: _renew_subs
  -394> 2019-07-03 10:21:56.799045 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -393> 2019-07-03 10:22:01.792321 7f3e5e3d4700 10 monclient: tick
  -392> 2019-07-03 10:22:01.799194 7f3e5fbd7700 10 monclient: _renew_subs
  -391> 2019-07-03 10:22:01.799208 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -390> 2019-07-03 10:22:06.799333 7f3e5fbd7700 10 monclient: _renew_subs
  -389> 2019-07-03 10:22:06.799341 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -388> 2019-07-03 10:22:07.790661 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -387> 2019-07-03 10:22:11.792469 7f3e5e3d4700 10 monclient: tick
  -386> 2019-07-03 10:22:21.792594 7f3e5e3d4700 10 monclient: tick
  -385> 2019-07-03 10:22:29.790768 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -384> 2019-07-03 10:22:31.792770 7f3e5e3d4700 10 monclient: tick
  -383> 2019-07-03 10:22:41.792917 7f3e5e3d4700 10 monclient: tick
  -382> 2019-07-03 10:22:51.790864 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -381> 2019-07-03 10:22:51.793056 7f3e5e3d4700 10 monclient: tick
  -380> 2019-07-03 10:23:01.793170 7f3e5e3d4700 10 monclient: tick
  -379> 2019-07-03 10:23:11.793300 7f3e5e3d4700 10 monclient: tick
  -378> 2019-07-03 10:23:13.790984 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -377> 2019-07-03 10:23:21.793452 7f3e5e3d4700 10 monclient: tick
  -376> 2019-07-03 10:23:31.793599 7f3e5e3d4700 10 monclient: tick
  -375> 2019-07-03 10:23:35.791056 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -374> 2019-07-03 10:23:41.793734 7f3e5e3d4700 10 monclient: tick
  -373> 2019-07-03 10:23:51.793875 7f3e5e3d4700 10 monclient: tick
  -372> 2019-07-03 10:23:57.791096 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -371> 2019-07-03 10:24:01.793998 7f3e5e3d4700 10 monclient: tick
  -370> 2019-07-03 10:24:06.803040 7f3e5fbd7700 10 monclient: _renew_subs
  -369> 2019-07-03 10:24:06.803055 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -368> 2019-07-03 10:24:11.794140 7f3e5e3d4700 10 monclient: tick
  -367> 2019-07-03 10:24:11.803214 7f3e5fbd7700 10 monclient: _renew_subs
  -366> 2019-07-03 10:24:11.803228 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -365> 2019-07-03 10:24:19.791159 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -364> 2019-07-03 10:24:21.794271 7f3e5e3d4700 10 monclient: tick
  -363> 2019-07-03 10:24:31.794396 7f3e5e3d4700 10 monclient: tick
  -362> 2019-07-03 10:24:41.791249 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -361> 2019-07-03 10:24:41.794563 7f3e5e3d4700 10 monclient: tick
  -360> 2019-07-03 10:24:51.794707 7f3e5e3d4700 10 monclient: tick
  -359> 2019-07-03 10:25:01.794854 7f3e5e3d4700 10 monclient: tick
  -358> 2019-07-03 10:25:01.804406 7f3e5fbd7700 10 monclient: _renew_subs
  -357> 2019-07-03 10:25:01.804422 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -356> 2019-07-03 10:25:03.791335 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -355> 2019-07-03 10:25:06.804604 7f3e5fbd7700 10 monclient: _renew_subs
  -354> 2019-07-03 10:25:06.804621 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -353> 2019-07-03 10:25:11.795033 7f3e5e3d4700 10 monclient: tick
  -352> 2019-07-03 10:25:21.795165 7f3e5e3d4700 10 monclient: tick
  -351> 2019-07-03 10:25:25.791406 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -350> 2019-07-03 10:25:31.795292 7f3e5e3d4700 10 monclient: tick
  -349> 2019-07-03 10:25:41.795456 7f3e5e3d4700 10 monclient: tick
  -348> 2019-07-03 10:25:47.791482 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -347> 2019-07-03 10:25:51.795586 7f3e5e3d4700 10 monclient: tick
  -346> 2019-07-03 10:26:01.795729 7f3e5e3d4700 10 monclient: tick
  -345> 2019-07-03 10:26:09.791593 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -344> 2019-07-03 10:26:11.795859 7f3e5e3d4700 10 monclient: tick
  -343> 2019-07-03 10:26:21.795998 7f3e5e3d4700 10 monclient: tick
  -342> 2019-07-03 10:26:31.791712 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -341> 2019-07-03 10:26:31.796147 7f3e5e3d4700 10 monclient: tick
  -340> 2019-07-03 10:26:41.796310 7f3e5e3d4700 10 monclient: tick
  -339> 2019-07-03 10:26:51.796451 7f3e5e3d4700 10 monclient: tick
  -338> 2019-07-03 10:26:53.791830 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -337> 2019-07-03 10:26:56.807401 7f3e5fbd7700 10 monclient: _renew_subs
  -336> 2019-07-03 10:26:56.807415 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -335> 2019-07-03 10:27:01.796599 7f3e5e3d4700 10 monclient: tick
  -334> 2019-07-03 10:27:01.807543 7f3e5fbd7700 10 monclient: _renew_subs
  -333> 2019-07-03 10:27:01.807550 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -332> 2019-07-03 10:27:06.807683 7f3e5fbd7700 10 monclient: _renew_subs
  -331> 2019-07-03 10:27:06.807691 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -330> 2019-07-03 10:27:11.796737 7f3e5e3d4700 10 monclient: tick
  -329> 2019-07-03 10:27:15.791944 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -328> 2019-07-03 10:27:21.796866 7f3e5e3d4700 10 monclient: tick
  -327> 2019-07-03 10:27:31.796986 7f3e5e3d4700 10 monclient: tick
  -326> 2019-07-03 10:27:37.792048 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -325> 2019-07-03 10:27:41.797123 7f3e5e3d4700 10 monclient: tick
  -324> 2019-07-03 10:27:51.797274 7f3e5e3d4700 10 monclient: tick
  -323> 2019-07-03 10:27:59.792152 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -322> 2019-07-03 10:28:01.797420 7f3e5e3d4700 10 monclient: tick
  -321> 2019-07-03 10:28:11.797565 7f3e5e3d4700 10 monclient: tick
  -320> 2019-07-03 10:28:21.792260 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -319> 2019-07-03 10:28:21.797722 7f3e5e3d4700 10 monclient: tick
  -318> 2019-07-03 10:28:31.797870 7f3e5e3d4700 10 monclient: tick
  -317> 2019-07-03 10:28:41.797997 7f3e5e3d4700 10 monclient: tick
  -316> 2019-07-03 10:28:43.792357 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -315> 2019-07-03 10:28:51.798168 7f3e5e3d4700 10 monclient: tick
  -314> 2019-07-03 10:29:01.798322 7f3e5e3d4700 10 monclient: tick
  -313> 2019-07-03 10:29:05.792454 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -312> 2019-07-03 10:29:11.798506 7f3e5e3d4700 10 monclient: tick
  -311> 2019-07-03 10:29:21.798672 7f3e5e3d4700 10 monclient: tick
  -310> 2019-07-03 10:29:27.792551 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -309> 2019-07-03 10:29:31.798835 7f3e5e3d4700 10 monclient: tick
  -308> 2019-07-03 10:29:41.798969 7f3e5e3d4700 10 monclient: tick
  -307> 2019-07-03 10:29:46.811645 7f3e5fbd7700 10 monclient: _renew_subs
  -306> 2019-07-03 10:29:46.811652 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -305> 2019-07-03 10:29:49.792665 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -304> 2019-07-03 10:29:51.799121 7f3e5e3d4700 10 monclient: tick
  -303> 2019-07-03 10:29:51.811812 7f3e5fbd7700 10 monclient: _renew_subs
  -302> 2019-07-03 10:29:51.811828 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -301> 2019-07-03 10:30:01.799252 7f3e5e3d4700 10 monclient: tick
  -300> 2019-07-03 10:30:11.792742 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -299> 2019-07-03 10:30:11.799378 7f3e5e3d4700 10 monclient: tick
  -298> 2019-07-03 10:30:21.799511 7f3e5e3d4700 10 monclient: tick
  -297> 2019-07-03 10:30:31.789958 7f3e5f3d6700  4 mgrc ms_handle_reset ms_handle_reset con 0x7f3e4c00afb0
  -296> 2019-07-03 10:30:31.789969 7f3e5f3d6700  4 mgrc reconnect Terminating session with 10.8.130.211:6800/27532
  -295> 2019-07-03 10:30:31.790004 7f3e5f3d6700  4 mgrc reconnect Starting new session with 10.8.130.211:6800/27532
  -294> 2019-07-03 10:30:31.799641 7f3e5e3d4700 10 monclient: tick
  -293> 2019-07-03 10:30:31.806460 7f3e5f3d6700  1 client.828441.objecter ms_handle_reset 0x7f3e48006b30 session 0x7f3e480063b0 osd.46
  -292> 2019-07-03 10:30:31.806576 7f3e5f3d6700  1 client.828441.objecter ms_handle_reset 0x560aa5b27640 session 0x560aa5b26ec0 osd.66
  -291> 2019-07-03 10:30:33.792857 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -290> 2019-07-03 10:30:41.799774 7f3e5e3d4700 10 monclient: tick
  -289> 2019-07-03 10:30:51.799922 7f3e5e3d4700 10 monclient: tick
  -288> 2019-07-03 10:30:55.792982 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -287> 2019-07-03 10:31:01.800072 7f3e5e3d4700 10 monclient: tick
  -286> 2019-07-03 10:31:11.800206 7f3e5e3d4700 10 monclient: tick
  -285> 2019-07-03 10:31:17.793079 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -284> 2019-07-03 10:31:21.800349 7f3e5e3d4700 10 monclient: tick
  -283> 2019-07-03 10:31:31.800497 7f3e5e3d4700 10 monclient: tick
  -282> 2019-07-03 10:31:39.793175 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -281> 2019-07-03 10:31:41.800641 7f3e5e3d4700 10 monclient: tick
  -280> 2019-07-03 10:31:51.800789 7f3e5e3d4700 10 monclient: tick
  -279> 2019-07-03 10:32:01.793296 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -278> 2019-07-03 10:32:01.800958 7f3e5e3d4700 10 monclient: tick
  -277> 2019-07-03 10:32:11.801042 7f3e5e3d4700 10 monclient: tick
  -276> 2019-07-03 10:32:16.815393 7f3e5fbd7700 10 monclient: _renew_subs
  -275> 2019-07-03 10:32:16.815407 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -274> 2019-07-03 10:32:21.801170 7f3e5e3d4700 10 monclient: tick
  -273> 2019-07-03 10:32:21.815562 7f3e5fbd7700 10 monclient: _renew_subs
  -272> 2019-07-03 10:32:21.815576 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -271> 2019-07-03 10:32:23.793395 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -270> 2019-07-03 10:32:31.801301 7f3e5e3d4700 10 monclient: tick
  -269> 2019-07-03 10:32:41.801413 7f3e5e3d4700 10 monclient: tick
  -268> 2019-07-03 10:32:45.793459 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -267> 2019-07-03 10:32:51.801558 7f3e5e3d4700 10 monclient: tick
  -266> 2019-07-03 10:33:01.802448 7f3e5e3d4700 10 monclient: tick
  -265> 2019-07-03 10:33:07.793554 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -264> 2019-07-03 10:33:11.802597 7f3e5e3d4700 10 monclient: tick
  -263> 2019-07-03 10:33:21.802696 7f3e5e3d4700 10 monclient: tick
  -262> 2019-07-03 10:33:29.793592 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -261> 2019-07-03 10:33:31.802810 7f3e5e3d4700 10 monclient: tick
  -260> 2019-07-03 10:33:41.802921 7f3e5e3d4700 10 monclient: tick
  -259> 2019-07-03 10:33:51.793653 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -258> 2019-07-03 10:33:51.803036 7f3e5e3d4700 10 monclient: tick
  -257> 2019-07-03 10:34:01.803194 7f3e5e3d4700 10 monclient: tick
  -256> 2019-07-03 10:34:11.803348 7f3e5e3d4700 10 monclient: tick
  -255> 2019-07-03 10:34:13.793726 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -254> 2019-07-03 10:34:21.804334 7f3e5e3d4700 10 monclient: tick
  -253> 2019-07-03 10:34:31.804460 7f3e5e3d4700 10 monclient: tick
  -252> 2019-07-03 10:34:35.793844 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -251> 2019-07-03 10:34:41.804583 7f3e5e3d4700 10 monclient: tick
  -250> 2019-07-03 10:34:46.818693 7f3e5fbd7700 10 monclient: _renew_subs
  -249> 2019-07-03 10:34:46.818707 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -248> 2019-07-03 10:34:51.804728 7f3e5e3d4700 10 monclient: tick
  -247> 2019-07-03 10:34:51.818843 7f3e5fbd7700 10 monclient: _renew_subs
  -246> 2019-07-03 10:34:51.818857 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -245> 2019-07-03 10:34:57.793888 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -244> 2019-07-03 10:35:01.804877 7f3e5e3d4700 10 monclient: tick
  -243> 2019-07-03 10:35:06.819202 7f3e5fbd7700 10 monclient: _renew_subs
  -242> 2019-07-03 10:35:06.819217 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -241> 2019-07-03 10:35:11.805009 7f3e5e3d4700 10 monclient: tick
  -240> 2019-07-03 10:35:11.819362 7f3e5fbd7700 10 monclient: _renew_subs
  -239> 2019-07-03 10:35:11.819370 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -238> 2019-07-03 10:35:19.793956 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -237> 2019-07-03 10:35:21.805138 7f3e5e3d4700 10 monclient: tick
  -236> 2019-07-03 10:35:31.805304 7f3e5e3d4700 10 monclient: tick
  -235> 2019-07-03 10:35:41.794051 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -234> 2019-07-03 10:35:41.805435 7f3e5e3d4700 10 monclient: tick
  -233> 2019-07-03 10:35:51.805585 7f3e5e3d4700 10 monclient: tick
  -232> 2019-07-03 10:36:01.805749 7f3e5e3d4700 10 monclient: tick
  -231> 2019-07-03 10:36:03.794154 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -230> 2019-07-03 10:36:11.805893 7f3e5e3d4700 10 monclient: tick
  -229> 2019-07-03 10:36:21.806055 7f3e5e3d4700 10 monclient: tick
  -228> 2019-07-03 10:36:25.794224 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -227> 2019-07-03 10:36:31.806182 7f3e5e3d4700 10 monclient: tick
  -226> 2019-07-03 10:36:41.806286 7f3e5e3d4700 10 monclient: tick
  -225> 2019-07-03 10:36:46.821511 7f3e5fbd7700 10 monclient: _renew_subs
  -224> 2019-07-03 10:36:46.821520 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -223> 2019-07-03 10:36:47.794330 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -222> 2019-07-03 10:36:51.806395 7f3e5e3d4700 10 monclient: tick
  -221> 2019-07-03 10:36:51.821669 7f3e5fbd7700 10 monclient: _renew_subs
  -220> 2019-07-03 10:36:51.821677 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -219> 2019-07-03 10:37:01.806507 7f3e5e3d4700 10 monclient: tick
  -218> 2019-07-03 10:37:09.794428 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -217> 2019-07-03 10:37:11.806639 7f3e5e3d4700 10 monclient: tick
  -216> 2019-07-03 10:37:21.806800 7f3e5e3d4700 10 monclient: tick
  -215> 2019-07-03 10:37:31.794496 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -214> 2019-07-03 10:37:31.806961 7f3e5e3d4700 10 monclient: tick
  -213> 2019-07-03 10:37:41.807106 7f3e5e3d4700 10 monclient: tick
  -212> 2019-07-03 10:37:51.807239 7f3e5e3d4700 10 monclient: tick
  -211> 2019-07-03 10:37:51.823154 7f3e5fbd7700 10 monclient: _renew_subs
  -210> 2019-07-03 10:37:51.823169 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -209> 2019-07-03 10:37:53.794590 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -208> 2019-07-03 10:37:56.823325 7f3e5fbd7700 10 monclient: _renew_subs
  -207> 2019-07-03 10:37:56.823340 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -206> 2019-07-03 10:38:01.807368 7f3e5e3d4700 10 monclient: tick
  -205> 2019-07-03 10:38:01.823492 7f3e5fbd7700 10 monclient: _renew_subs
  -204> 2019-07-03 10:38:01.823506 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -203> 2019-07-03 10:38:06.823677 7f3e5fbd7700 10 monclient: _renew_subs
  -202> 2019-07-03 10:38:06.823696 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -201> 2019-07-03 10:38:11.807481 7f3e5e3d4700 10 monclient: tick
  -200> 2019-07-03 10:38:15.794703 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -199> 2019-07-03 10:38:21.807647 7f3e5e3d4700 10 monclient: tick
  -198> 2019-07-03 10:38:31.807795 7f3e5e3d4700 10 monclient: tick
  -197> 2019-07-03 10:38:37.794820 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -196> 2019-07-03 10:38:41.807936 7f3e5e3d4700 10 monclient: tick
  -195> 2019-07-03 10:38:51.808095 7f3e5e3d4700 10 monclient: tick
  -194> 2019-07-03 10:38:59.794924 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -193> 2019-07-03 10:39:01.808238 7f3e5e3d4700 10 monclient: tick
  -192> 2019-07-03 10:39:11.808362 7f3e5e3d4700 10 monclient: tick
  -191> 2019-07-03 10:39:21.795007 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -190> 2019-07-03 10:39:21.808521 7f3e5e3d4700 10 monclient: tick
  -189> 2019-07-03 10:39:31.808668 7f3e5e3d4700 10 monclient: tick
  -188> 2019-07-03 10:39:41.808799 7f3e5e3d4700 10 monclient: tick
  -187> 2019-07-03 10:39:43.795090 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -186> 2019-07-03 10:39:51.808922 7f3e5e3d4700 10 monclient: tick
  -185> 2019-07-03 10:40:01.809041 7f3e5e3d4700 10 monclient: tick
  -184> 2019-07-03 10:40:05.795197 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -183> 2019-07-03 10:40:11.809173 7f3e5e3d4700 10 monclient: tick
  -182> 2019-07-03 10:40:21.809325 7f3e5e3d4700 10 monclient: tick
  -181> 2019-07-03 10:40:27.795275 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -180> 2019-07-03 10:40:31.809487 7f3e5e3d4700 10 monclient: tick
  -179> 2019-07-03 10:40:41.809573 7f3e5e3d4700 10 monclient: tick
  -178> 2019-07-03 10:40:49.795389 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -177> 2019-07-03 10:40:51.809706 7f3e5e3d4700 10 monclient: tick
  -176> 2019-07-03 10:40:51.827473 7f3e5fbd7700 10 monclient: _renew_subs
  -175> 2019-07-03 10:40:51.827488 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -174> 2019-07-03 10:40:56.827643 7f3e5fbd7700 10 monclient: _renew_subs
  -173> 2019-07-03 10:40:56.827658 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -172> 2019-07-03 10:41:01.809820 7f3e5e3d4700 10 monclient: tick
  -171> 2019-07-03 10:41:11.795503 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -170> 2019-07-03 10:41:11.809943 7f3e5e3d4700 10 monclient: tick
  -169> 2019-07-03 10:41:11.828033 7f3e5fbd7700 10 monclient: _renew_subs
  -168> 2019-07-03 10:41:11.828041 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -167> 2019-07-03 10:41:21.810798 7f3e5e3d4700 10 monclient: tick
  -166> 2019-07-03 10:41:31.810943 7f3e5e3d4700 10 monclient: tick
  -165> 2019-07-03 10:41:33.795626 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -164> 2019-07-03 10:41:41.811066 7f3e5e3d4700 10 monclient: tick
  -163> 2019-07-03 10:41:51.811186 7f3e5e3d4700 10 monclient: tick
  -162> 2019-07-03 10:41:55.795722 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -161> 2019-07-03 10:42:01.811306 7f3e5e3d4700 10 monclient: tick
  -160> 2019-07-03 10:42:11.811415 7f3e5e3d4700 10 monclient: tick
  -159> 2019-07-03 10:42:17.795817 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -158> 2019-07-03 10:42:21.811522 7f3e5e3d4700 10 monclient: tick
  -157> 2019-07-03 10:42:31.811671 7f3e5e3d4700 10 monclient: tick
  -156> 2019-07-03 10:42:39.795933 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -155> 2019-07-03 10:42:41.811820 7f3e5e3d4700 10 monclient: tick
  -154> 2019-07-03 10:42:51.811971 7f3e5e3d4700 10 monclient: tick
  -153> 2019-07-03 10:43:01.796018 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -152> 2019-07-03 10:43:01.812116 7f3e5e3d4700 10 monclient: tick
  -151> 2019-07-03 10:43:11.812263 7f3e5e3d4700 10 monclient: tick
  -150> 2019-07-03 10:43:16.831657 7f3e5fbd7700 10 monclient: _renew_subs
  -149> 2019-07-03 10:43:16.831667 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -148> 2019-07-03 10:43:21.812399 7f3e5e3d4700 10 monclient: tick
  -147> 2019-07-03 10:43:21.831800 7f3e5fbd7700 10 monclient: _renew_subs
  -146> 2019-07-03 10:43:21.831808 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -145> 2019-07-03 10:43:23.796119 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -144> 2019-07-03 10:43:31.812542 7f3e5e3d4700 10 monclient: tick
  -143> 2019-07-03 10:43:41.812695 7f3e5e3d4700 10 monclient: tick
  -142> 2019-07-03 10:43:41.832304 7f3e5fbd7700 10 monclient: _renew_subs
  -141> 2019-07-03 10:43:41.832313 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -140> 2019-07-03 10:43:45.796218 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -139> 2019-07-03 10:43:46.832435 7f3e5fbd7700 10 monclient: _renew_subs
  -138> 2019-07-03 10:43:46.832449 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -137> 2019-07-03 10:43:51.812850 7f3e5e3d4700 10 monclient: tick
  -136> 2019-07-03 10:44:01.813012 7f3e5e3d4700 10 monclient: tick
  -135> 2019-07-03 10:44:07.796321 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -134> 2019-07-03 10:44:11.813122 7f3e5e3d4700 10 monclient: tick
  -133> 2019-07-03 10:44:21.813274 7f3e5e3d4700 10 monclient: tick
  -132> 2019-07-03 10:44:29.796425 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -131> 2019-07-03 10:44:31.813420 7f3e5e3d4700 10 monclient: tick
  -130> 2019-07-03 10:44:41.813583 7f3e5e3d4700 10 monclient: tick
  -129> 2019-07-03 10:44:51.796549 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -128> 2019-07-03 10:44:51.813753 7f3e5e3d4700 10 monclient: tick
  -127> 2019-07-03 10:45:01.813917 7f3e5e3d4700 10 monclient: tick
  -126> 2019-07-03 10:45:11.814049 7f3e5e3d4700 10 monclient: tick
  -125> 2019-07-03 10:45:13.796659 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -124> 2019-07-03 10:45:21.814190 7f3e5e3d4700 10 monclient: tick
  -123> 2019-07-03 10:45:31.790913 7f3e5f3d6700  4 mgrc ms_handle_reset ms_handle_reset con 0x7f3e4c035380
  -122> 2019-07-03 10:45:31.790940 7f3e5f3d6700  4 mgrc reconnect Terminating session with 10.8.130.211:6800/27532
  -121> 2019-07-03 10:45:31.790969 7f3e5f3d6700  4 mgrc reconnect Starting new session with 10.8.130.211:6800/27532
  -120> 2019-07-03 10:45:31.792606 7f3e5f3d6700  1 client.828441.objecter ms_handle_reset 0x560aa5b1b8d0 session 0x560aa5b1b150 osd.0
  -119> 2019-07-03 10:45:31.800897 7f3e5f3d6700  1 client.828441.objecter ms_handle_reset 0x560aa5b20cd0 session 0x560aa5b20550 osd.32
  -118> 2019-07-03 10:45:31.804592 7f3e5f3d6700  1 client.828441.objecter ms_handle_reset 0x7f3e58005ed0 session 0x7f3e58005750 osd.40
  -117> 2019-07-03 10:45:31.805596 7f3e5f3d6700  1 client.828441.objecter ms_handle_reset 0x7f3e50006f30 session 0x7f3e500067b0 osd.15
  -116> 2019-07-03 10:45:31.807611 7f3e5f3d6700  1 client.828441.objecter ms_handle_reset 0x7f3e4c00afb0 session 0x7f3e480063b0 osd.46
  -115> 2019-07-03 10:45:31.807672 7f3e5f3d6700  1 client.828441.objecter ms_handle_reset 0x560aa5b16a20 session 0x560aa5b162a0 osd.85
  -114> 2019-07-03 10:45:31.807686 7f3e5f3d6700  1 client.828441.objecter ms_handle_reset 0x7f3e4c0399a0 session 0x560aa5b26ec0 osd.66
  -113> 2019-07-03 10:45:31.814342 7f3e5e3d4700 10 monclient: tick
  -112> 2019-07-03 10:45:31.823573 7f3e5f3d6700  1 client.828441.objecter ms_handle_reset 0x560aa5b755d0 session 0x560aa5b74e50 osd.88
  -111> 2019-07-03 10:45:31.834624 7f3e5f3d6700  1 client.828441.objecter ms_handle_reset 0x560aa5b7ac80 session 0x560aa5b7a500 osd.62
  -110> 2019-07-03 10:45:35.796730 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -109> 2019-07-03 10:45:41.814463 7f3e5e3d4700 10 monclient: tick
  -108> 2019-07-03 10:45:51.814592 7f3e5e3d4700 10 monclient: tick
  -107> 2019-07-03 10:45:56.835501 7f3e5fbd7700 10 monclient: _renew_subs
  -106> 2019-07-03 10:45:56.835512 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -105> 2019-07-03 10:45:57.796847 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
  -104> 2019-07-03 10:46:01.814765 7f3e5e3d4700 10 monclient: tick
  -103> 2019-07-03 10:46:01.835669 7f3e5fbd7700 10 monclient: _renew_subs
  -102> 2019-07-03 10:46:01.835684 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
  -101> 2019-07-03 10:46:11.814874 7f3e5e3d4700 10 monclient: tick
  -100> 2019-07-03 10:46:19.796934 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -99> 2019-07-03 10:46:21.814990 7f3e5e3d4700 10 monclient: tick
   -98> 2019-07-03 10:46:31.815106 7f3e5e3d4700 10 monclient: tick
   -97> 2019-07-03 10:46:41.797026 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -96> 2019-07-03 10:46:41.815256 7f3e5e3d4700 10 monclient: tick
   -95> 2019-07-03 10:46:51.815410 7f3e5e3d4700 10 monclient: tick
   -94> 2019-07-03 10:47:01.815508 7f3e5e3d4700 10 monclient: tick
   -93> 2019-07-03 10:47:03.797159 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -92> 2019-07-03 10:47:11.815668 7f3e5e3d4700 10 monclient: tick
   -91> 2019-07-03 10:47:21.815817 7f3e5e3d4700 10 monclient: tick
   -90> 2019-07-03 10:47:25.797253 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -89> 2019-07-03 10:47:31.815967 7f3e5e3d4700 10 monclient: tick
   -88> 2019-07-03 10:47:41.816122 7f3e5e3d4700 10 monclient: tick
   -87> 2019-07-03 10:47:41.837862 7f3e5fbd7700 10 monclient: _renew_subs
   -86> 2019-07-03 10:47:41.837877 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
   -85> 2019-07-03 10:47:46.838035 7f3e5fbd7700 10 monclient: _renew_subs
   -84> 2019-07-03 10:47:46.838043 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
   -83> 2019-07-03 10:47:47.797357 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -82> 2019-07-03 10:47:51.816272 7f3e5e3d4700 10 monclient: tick
   -81> 2019-07-03 10:48:01.816416 7f3e5e3d4700 10 monclient: tick
   -80> 2019-07-03 10:48:03.812595 7f3e5f3d6700  1 client.828441.objecter ms_handle_reset 0x7f3e4c02a660 session 0x7f3e4c0243c0 osd.27
   -79> 2019-07-03 10:48:09.797459 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -78> 2019-07-03 10:48:11.816574 7f3e5e3d4700 10 monclient: tick
   -77> 2019-07-03 10:48:11.838659 7f3e5fbd7700 10 monclient: _renew_subs
   -76> 2019-07-03 10:48:11.838670 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
   -75> 2019-07-03 10:48:16.838823 7f3e5fbd7700 10 monclient: _renew_subs
   -74> 2019-07-03 10:48:16.838837 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
   -73> 2019-07-03 10:48:21.816728 7f3e5e3d4700 10 monclient: tick
   -72> 2019-07-03 10:48:31.797573 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -71> 2019-07-03 10:48:31.816901 7f3e5e3d4700 10 monclient: tick
   -70> 2019-07-03 10:48:41.817727 7f3e5e3d4700 10 monclient: tick
   -69> 2019-07-03 10:48:51.817869 7f3e5e3d4700 10 monclient: tick
   -68> 2019-07-03 10:48:53.797683 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -67> 2019-07-03 10:49:01.818004 7f3e5e3d4700 10 monclient: tick
   -66> 2019-07-03 10:49:11.818136 7f3e5e3d4700 10 monclient: tick
   -65> 2019-07-03 10:49:15.797756 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -64> 2019-07-03 10:49:21.818283 7f3e5e3d4700 10 monclient: tick
   -63> 2019-07-03 10:49:31.818409 7f3e5e3d4700 10 monclient: tick
   -62> 2019-07-03 10:49:37.797876 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -61> 2019-07-03 10:49:41.818554 7f3e5e3d4700 10 monclient: tick
   -60> 2019-07-03 10:49:51.818709 7f3e5e3d4700 10 monclient: tick
   -59> 2019-07-03 10:49:59.797988 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -58> 2019-07-03 10:50:01.818840 7f3e5e3d4700 10 monclient: tick
   -57> 2019-07-03 10:50:11.818965 7f3e5e3d4700 10 monclient: tick
   -56> 2019-07-03 10:50:21.798071 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -55> 2019-07-03 10:50:21.819109 7f3e5e3d4700 10 monclient: tick
   -54> 2019-07-03 10:50:31.819254 7f3e5e3d4700 10 monclient: tick
   -53> 2019-07-03 10:50:41.819404 7f3e5e3d4700 10 monclient: tick
   -52> 2019-07-03 10:50:43.798144 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -51> 2019-07-03 10:50:51.819584 7f3e5e3d4700 10 monclient: tick
   -50> 2019-07-03 10:51:01.819751 7f3e5e3d4700 10 monclient: tick
   -49> 2019-07-03 10:51:05.798237 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -48> 2019-07-03 10:51:11.819888 7f3e5e3d4700 10 monclient: tick
   -47> 2019-07-03 10:51:11.843739 7f3e5fbd7700 10 monclient: _renew_subs
   -46> 2019-07-03 10:51:11.843747 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
   -45> 2019-07-03 10:51:16.843820 7f3e5fbd7700 10 monclient: _renew_subs
   -44> 2019-07-03 10:51:16.843826 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
   -43> 2019-07-03 10:51:21.820017 7f3e5e3d4700 10 monclient: tick
   -42> 2019-07-03 10:51:27.798345 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -41> 2019-07-03 10:51:31.820161 7f3e5e3d4700 10 monclient: tick
   -40> 2019-07-03 10:51:41.820332 7f3e5e3d4700 10 monclient: tick
   -39> 2019-07-03 10:51:49.798464 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -38> 2019-07-03 10:51:51.821311 7f3e5e3d4700 10 monclient: tick
   -37> 2019-07-03 10:52:01.821458 7f3e5e3d4700 10 monclient: tick
   -36> 2019-07-03 10:52:11.798579 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -35> 2019-07-03 10:52:11.821597 7f3e5e3d4700 10 monclient: tick
   -34> 2019-07-03 10:52:21.821727 7f3e5e3d4700 10 monclient: tick
   -33> 2019-07-03 10:52:31.821869 7f3e5e3d4700 10 monclient: tick
   -32> 2019-07-03 10:52:33.798697 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -31> 2019-07-03 10:52:41.822004 7f3e5e3d4700 10 monclient: tick
   -30> 2019-07-03 10:52:51.822156 7f3e5e3d4700 10 monclient: tick
   -29> 2019-07-03 10:52:55.798801 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -28> 2019-07-03 10:53:01.822292 7f3e5e3d4700 10 monclient: tick
   -27> 2019-07-03 10:53:11.823604 7f3e5e3d4700 10 monclient: tick
   -26> 2019-07-03 10:53:17.798862 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -25> 2019-07-03 10:53:21.823717 7f3e5e3d4700 10 monclient: tick
   -24> 2019-07-03 10:53:31.823863 7f3e5e3d4700 10 monclient: tick
   -23> 2019-07-03 10:53:36.847116 7f3e5fbd7700 10 monclient: _renew_subs
   -22> 2019-07-03 10:53:36.847131 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
   -21> 2019-07-03 10:53:39.798937 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -20> 2019-07-03 10:53:41.824033 7f3e5e3d4700 10 monclient: tick
   -19> 2019-07-03 10:53:41.847280 7f3e5fbd7700 10 monclient: _renew_subs
   -18> 2019-07-03 10:53:41.847293 7f3e5fbd7700 10 monclient: _send_mon_message to mon.extensa010 at 10.8.130.210:6789/0
   -17> 2019-07-03 10:53:51.824159 7f3e5e3d4700 10 monclient: tick
   -16> 2019-07-03 10:54:01.798994 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -15> 2019-07-03 10:54:01.824294 7f3e5e3d4700 10 monclient: tick
   -14> 2019-07-03 10:54:11.824418 7f3e5e3d4700 10 monclient: tick
   -13> 2019-07-03 10:54:21.824591 7f3e5e3d4700 10 monclient: tick
   -12> 2019-07-03 10:54:23.799106 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
   -11> 2019-07-03 10:54:31.824728 7f3e5e3d4700 10 monclient: tick
   -10> 2019-07-03 10:54:41.824897 7f3e5e3d4700 10 monclient: tick
    -9> 2019-07-03 10:54:45.799219 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
    -8> 2019-07-03 10:54:51.825034 7f3e5e3d4700 10 monclient: tick
    -7> 2019-07-03 10:55:01.825165 7f3e5e3d4700 10 monclient: tick
    -6> 2019-07-03 10:55:07.799314 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
    -5> 2019-07-03 10:55:11.825307 7f3e5e3d4700 10 monclient: tick
    -4> 2019-07-03 10:55:21.825429 7f3e5e3d4700 10 monclient: tick
    -3> 2019-07-03 10:55:29.799397 7f3e577fe700  2 RGWDataChangesLog::ChangesRenewThread: start
    -2> 2019-07-03 10:55:31.825581 7f3e5e3d4700 10 monclient: tick
    -1> 2019-07-03 10:55:41.825738 7f3e5e3d4700 10 monclient: tick
     0> 2019-07-03 10:55:50.121836 7f3e603d8700 -1 *** Caught signal (Aborted) **
 in thread 7f3e603d8700 thread_name:msgr-worker-2

 ceph version 12.2.8-128.el7cp (030358773c5213a14c1444a5147258672b2dc15f) luminous (stable)
 1: (()+0x4b9b21) [0x560aa3f2fb21]
 2: (()+0xf5d0) [0x7f3e709665d0]
 3: (gsignal()+0x37) [0x7f3e656f3207]
 4: (abort()+0x148) [0x7f3e656f48f8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f3e660027d5]
 6: (()+0x5e746) [0x7f3e66000746]
 7: (()+0x5e773) [0x7f3e66000773]
 8: (()+0xb5105) [0x7f3e66057105]
 9: (()+0x7dd5) [0x7f3e7095edd5]
 10: (clone()+0x6d) [0x7f3e657baead]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 1 lockdep
   0/ 1 context
   1/ 1 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 1 buffer
   0/ 1 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 journaler
   0/ 5 objectcacher
   0/ 5 client
   1/ 5 osd
   0/ 5 optracker
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 0 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
  -2/-2 (syslog threshold)
  99/99 (stderr threshold)
  max_recent       500
  max_new         1000
  log_file 
--- end dump of recent events ---
Aborted (core dumped)




~]#  radosgw-admin bucket stats --bucket mybucket1
{
    "bucket": "mybucket1",
    "zonegroup": "e4460222-1aae-4c86-9c70-721cd9c13d86",
    "placement_rule": "default-placement",
    "explicit_placement": {
        "data_pool": "",
        "data_extra_pool": "",
        "index_pool": ""
    },
    "id": "55c9e095-9d20-4a0f-91ef-35e786611b6f.258756.1",
    "marker": "55c9e095-9d20-4a0f-91ef-35e786611b6f.258756.1",
    "index_type": "Normal",
    "owner": "usr",
    "ver": "0#37628478",
    "master_ver": "0#0",
    "mtime": "2019-06-23 09:35:34.776343",
    "max_marker": "0#00037628477.50172474.10",
    "usage": {
        "rgw.main": {
            "size": 26896651673600,
            "size_actual": 26915083829248,
            "size_utilized": 26896651673600,
            "size_kb": 26266261400,
            "size_kb_actual": 26284261552,
            "size_kb_utilized": 26266261400,
            "num_objects": 12542738
        }
    },
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    }
}

Note You need to log in before you can comment on or make changes to this bug.