Hide Forgot
Created attachment 1132882 [details] skynet.log.bz2 Description of problem ====================== During *Create Cluster* task, I see lot of ERROR messages in skynet.log file complaining about dbus handlers. Version-Release number of selected component ============================================ rhscon-core-0.0.8-10.el7.x86_64 rhscon-ui-0.0.19-1.el7.noarch rhscon-ceph-0.0.6-10.el7.x86_64 rhscon-agent-0.0.3-3.el7.noarch How reproducible ================ 5% (not sure how to reproduce) Steps to Reproduce ================== Here are the steps I did: 1. Install usm following the documentation. 2. Create cluster 1st cluster, but the task failed 3. Removed the 1st cluster 4. Created 2nd cluster, the task finished this time 5. Look into skynet.log on node machines Actual results ============== There are lot of error messages in skynet.log files on all node machines: ~~~ 2016-03-02 12:00:52,162 - INFO - skynetd: Initializing the daemon 2016-03-02 12:48:34,173 - INFO - skynetd: Initializing the daemon 2016-03-02 12:48:35,076 - INFO - skynetd: Started the daemon 2016-03-02 13:40:48,228 - ERROR - skynetd: Error in handler: block_add | Reason: 'NoneType' object is not iterable | Args: (<skynetd.callback.Callback object at 0x21b7150>, dbus.ObjectPath('/org/storaged/Storaged/block_devices/vdc'), dbus.Dictionary({dbus.String(u'org.storaged.Storaged.PartitionTable'): dbus.Dictionary({dbus.String(u'Type'): dbus.String(u'gpt', variant_level=1)}, signature=dbus.Signature('sv'))}, signature=dbus.Signature('sa{sv}'))) | Kwargs: {'path': dbus.ObjectPath('/org/storaged/Storaged')} 2016-03-02 13:40:48,230 - ERROR - skynetd: Error in handler: drive_add | Reason: 'NoneType' object is not iterable | Args: (<skynetd.callback.Callback object at 0x21b7150>, dbus.ObjectPath('/org/storaged/Storaged/block_devices/vdc'), dbus.Dictionary({dbus.String(u'org.storaged.Storaged.PartitionTable'): dbus.Dictionary({dbus.String(u'Type'): dbus.String(u'gpt', variant_level=1)}, signature=dbus.Signature('sv'))}, signature=dbus.Signature('sa{sv}'))) | Kwargs: {'path': dbus.ObjectPath('/org/storaged/Storaged')} 2016-03-02 13:40:50,886 - ERROR - skynetd: Error in handler: drive_add | Reason: 'NoneType' object is not iterable | Args: (<skynetd.callback.Callback object at 0x21b7150>, dbus.ObjectPath('/org/storaged/Storaged/block_devices/vdc2'), dbus.Dictionary({dbus.String(u'org.storaged.Storaged.Block'): dbus.Dictionary({dbus.String(u'IdType'): dbus.String(u'', variant_level=1), dbus.String(u'PreferredDevice'): dbus.Array([dbus.Byte(47), dbus.Byte(100), dbus.Byte(101), dbus.Byte(118), dbus.Byte(47), dbus.Byte(118), dbus.Byte(100), dbus.Byte(99), dbus.Byte(50), dbus.Byte(0)], signature=dbus.Signature('y'), variant_level=1), dbus.String(u'IdLabel'): dbus.String(u'', variant_level=1), dbus.String(u'HintSymbolicIconName'): dbus.String(u'', variant_level=1), dbus.String(u'HintPartitionable'): dbus.Boolean(True, variant_level=1), dbus.String(u'IdUUID'): dbus.String(u'', variant_level=1), dbus.String(u'MDRaid'): dbus.ObjectPath('/', variant_level=1), dbus.String(u'IdVersion'): dbus.String(u'', variant_level=1), dbus.String(u'ReadOnly'): dbus.Boolean(False, variant_level=1), dbus.String(u'CryptoBackingDevice'): dbus.ObjectPath('/', variant_level=1), dbus.String(u'Device'): dbus.Array([dbus.Byte(47), dbus.Byte(100), dbus.Byte(101), dbus.Byte(118), dbus.Byte(47), dbus.Byte(118), dbus.Byte(100), dbus.Byte(99), dbus.Byte(50), dbus.Byte(0)], signature=dbus.Signature('y'), variant_level=1), dbus.String(u'Configuration'): dbus.Array([], signature=dbus.Signature('(sa{sv})'), variant_level=1), dbus.String(u'Symlinks'): dbus.Array([dbus.Array([dbus.Byte(47), dbus.Byte(100), dbus.Byte(101), dbus.Byte(118), dbus.Byte(47), dbus.Byte(100), dbus.Byte(105), dbus.Byte(115), dbus.Byte(107), dbus.Byte(47), dbus.Byte(98), dbus.Byte(121), dbus.Byte(45), dbus.Byte(105), dbus.Byte(100), dbus.Byte(47), dbus.Byte(118), dbus.Byte(105), dbus.Byte(114), dbus.Byte(116), dbus.Byte(105), dbus.Byte(111), dbus.Byte(45), dbus.Byte(56), dbus.Byte(100), dbus.Byte(48), dbus.Byte(101), dbus.Byte(97), dbus.Byte(101), dbus.Byte(52), dbus.Byte(97), dbus.Byte(45), dbus.Byte(100), dbus.Byte(55), dbus.Byte(49), dbus.Byte(50), dbus.Byte(45), dbus.Byte(52), dbus.Byte(99), dbus.Byte(100), dbus.Byte(102), dbus.Byte(45), dbus.Byte(56), dbus.Byte(45), dbus.Byte(112), dbus.Byte(97), dbus.Byte(114), dbus.Byte(116), dbus.Byte(50), dbus.Byte(0)], signature=dbus.Signature('y')), dbus.Array([dbus.Byte(47), dbus.Byte(100), dbus.Byte(101), dbus.Byte(118), dbus.Byte(47), dbus.Byte(100), dbus.Byte(105), dbus.Byte(115), dbus.Byte(107), dbus.Byte(47), dbus.Byte(98), dbus.Byte(121), dbus.Byte(45), dbus.Byte(112), dbus.Byte(97), dbus.Byte(114), dbus.Byte(116), dbus.Byte(108), dbus.Byte(97), dbus.Byte(98), dbus.Byte(101), dbus.Byte(108), dbus.Byte(47), dbus.Byte(99), dbus.Byte(101), dbus.Byte(112), dbus.Byte(104), dbus.Byte(92), dbus.Byte(120), dbus.Byte(50), dbus.Byte(48), dbus.Byte(106), dbus.Byte(111), dbus.Byte(117), dbus.Byte(114), dbus.Byte(110), dbus.Byte(97), dbus.Byte(108), dbus.Byte(0)], signature=dbus.Signature('y')), dbus.Array([dbus.Byte(47), dbus.Byte(100), dbus.Byte(101), dbus.Byte(118), dbus.Byte(47), dbus.Byte(100), dbus.Byte(105), dbus.Byte(115), dbus.Byte(107), dbus.Byte(47), dbus.Byte(98), dbus.Byte(121), dbus.Byte(45), dbus.Byte(112), dbus.Byte(97), dbus.Byte(114), dbus.Byte(116), dbus.Byte(116), dbus.Byte(121), dbus.Byte(112), dbus.Byte(101), dbus.Byte(117), dbus.Byte(117), dbus.Byte(105), dbus.Byte(100), dbus.Byte(47), dbus.Byte(52), dbus.Byte(53), dbus.Byte(98), dbus.Byte(48), dbus.Byte(57), dbus.Byte(54), dbus.Byte(57), dbus.Byte(101), dbus.Byte(45), dbus.Byte(57), dbus.Byte(98), dbus.Byte(48), dbus.Byte(51), dbus.Byte(45), dbus.Byte(52), dbus.Byte(102), dbus.Byte(51), dbus.Byte(48), dbus.Byte(45), dbus.Byte(98), dbus.Byte(52), dbus.Byte(99), dbus.Byte(54), dbus.Byte(45), dbus.Byte(98), dbus.Byte(52), dbus.Byte(98), dbus.Byte(56), dbus.Byte(48), dbus.Byte(99), dbus.Byte(101), dbus.Byte(102), dbus.Byte(102), dbus.Byte(49), dbus.Byte(48), dbus.Byte(54), dbus.Byte(46), dbus.Byte(53), dbus.Byte(52), dbus.Byte(102), dbus.Byte(48), dbus.Byte(97), dbus.Byte(53), dbus.Byte(102), dbus.Byte(50), dbus.Byte(45), dbus.Byte(98), dbus.Byte(51), dbus.Byte(98), dbus.Byte(97), dbus.Byte(45), dbus.Byte(52), dbus.Byte(101), dbus.Byte(52), dbus.Byte(100), dbus.Byte(45), dbus.Byte(98), dbus.Byte(54), dbus.Byte(51), dbus.Byte(56), dbus.Byte(45), dbus.Byte(52), dbus.Byte(49), dbus.Byte(55), dbus.Byte(56), dbus.Byte(98), dbus.Byte(56), dbus.Byte(54), dbus.Byte(100), dbus.Byte(98), dbus.Byte(57), dbus.Byte(101), dbus.Byte(52), dbus.Byte(0)], signature=dbus.Signature('y')), dbus.Array([dbus.Byte(47), dbus.Byte(100), dbus.Byte(101), dbus.Byte(118), dbus.Byte(47), dbus.Byte(100), dbus.Byte(105), dbus.Byte(115), dbus.Byte(107), dbus.Byte(47), dbus.Byte(98), dbus.Byte(121), dbus.Byte(45), dbus.Byte(112), dbus.Byte(97), dbus.Byte(114), dbus.Byte(116), dbus.Byte(117), dbus.Byte(117), dbus.Byte(105), dbus.Byte(100), dbus.Byte(47), dbus.Byte(53), dbus.Byte(52), dbus.Byte(102), dbus.Byte(48), dbus.Byte(97), dbus.Byte(53), dbus.Byte(102), dbus.Byte(50), dbus.Byte(45), dbus.Byte(98), dbus.Byte(51), dbus.Byte(98), dbus.Byte(97), dbus.Byte(45), dbus.Byte(52), dbus.Byte(101), dbus.Byte(52), dbus.Byte(100), dbus.Byte(45), dbus.Byte(98), dbus.Byte(54), dbus.Byte(51), dbus.Byte(56), dbus.Byte(45), dbus.Byte(52), dbus.Byte(49), dbus.Byte(55), dbus.Byte(56), dbus.Byte(98), dbus.Byte(56), dbus.Byte(54), dbus.Byte(100), dbus.Byte(98), dbus.Byte(57), dbus.Byte(101), dbus.Byte(52), dbus.Byte(0)], signature=dbus.Signature('y'))], signature=dbus.Signature('ay'), variant_level=1), dbus.String(u'HintIconName'): dbus.String(u'', variant_level=1), dbus.String(u'Drive'): dbus.ObjectPath('/org/storaged/Storaged/drives/VirtIO_Disk_8d0eae4a_d712_4cdf_8', variant_level=1), dbus.String(u'IdUsage'): dbus.String(u'', variant_level=1), dbus.String(u'MDRaidMember'): dbus.ObjectPath('/', variant_level=1), dbus.String(u'DeviceNumber'): dbus.UInt64(64802L, variant_level=1), dbus.String(u'HintSystem'): dbus.Boolean(True, variant_level=1), dbus.String(u'HintName'): dbus.String(u'', variant_level=1), dbus.String(u'HintIgnore'): dbus.Boolean(False, variant_level=1), dbus.String(u'Id'): dbus.String(u'by-id-virtio-8d0eae4a-d712-4cdf-8-part2', variant_level=1), dbus.String(u'HintAuto'): dbus.Boolean(False, variant_level=1), dbus.String(u'Size'): dbus.UInt64(1072693760L, variant_level=1)}, signature=dbus.Signature('sv')), dbus.String(u'org.storaged.Storaged.Partition'): dbus.Dictionary({dbus.String(u'Name'): dbus.String(u'ceph journal', variant_level=1), dbus.String(u'Number'): dbus.UInt32(2L, variant_level=1), dbus.String(u'IsContainer'): dbus.Boolean(False, variant_level=1), dbus.String(u'UUID'): dbus.String(u'54f0a5f2-b3ba-4e4d-b638-4178b86db9e4', variant_level=1), dbus.String(u'Flags'): dbus.UInt64(0L, variant_level=1), dbus.String(u'Offset'): dbus.UInt64(1048576L, variant_level=1), dbus.String(u'IsContained'): dbus.Boolean(False, variant_level=1), dbus.String(u'Table'): dbus.ObjectPath('/org/storaged/Storaged/block_devices/vdc', variant_level=1), dbus.String(u'Type'): dbus.String(u'45b0969e-9b03-4f30-b4c6-b4b80ceff106', variant_level=1), dbus.String(u'Size'): dbus.UInt64(1072693760L, variant_level=1)}, signature=dbus.Signature('sv'))}, signature=dbus.Signature('sa{sv}'))) | Kwargs: {'path': dbus.ObjectPath('/org/storaged/Storaged')} ~~~ Full log file attached. Expected results ================ There are no dbus errors. Additional info =============== On March 2 I discussed this with ndarshan who confirmed it as a valid issue.
Patch sent to upstream for review: https://review.gerrithub.io/#/c/281518/
I still see this issue with server: ceph-ansible-1.0.5-31.el7scon.noarch ceph-installer-1.0.14-1.el7scon.noarch rhscon-ceph-0.0.39-1.el7scon.x86_64 rhscon-core-0.0.39-1.el7scon.x86_64 rhscon-core-selinux-0.0.39-1.el7scon.noarch rhscon-ui-0.0.51-1.el7scon.noarch node: ceph-base-10.2.2-33.el7cp.x86_64 ceph-common-10.2.2-33.el7cp.x86_64 ceph-osd-10.2.2-33.el7cp.x86_64 ceph-selinux-10.2.2-33.el7cp.x86_64 libcephfs1-10.2.2-33.el7cp.x86_64 python-cephfs-10.2.2-33.el7cp.x86_64 rhscon-agent-0.0.16-1.el7scon.noarch rhscon-core-selinux-0.0.39-1.el7scon.noarch
Analysed this as per the discussion in the bug scrub. This won't create any issues as such. Hence moving this to next release.