Bug 1387297 - [ceph-iscsi-ansible]: "rbd: sysfs write failed" error seen when trying to create a big number of images in one playbook run.
Summary: [ceph-iscsi-ansible]: "rbd: sysfs write failed" error seen when trying to cre...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Storage Console
Classification: Red Hat Storage
Component: ceph-ansible
Version: 2
Hardware: Unspecified
OS: Linux
urgent
urgent
Target Milestone: ---
: 2
Assignee: Mike Christie
QA Contact: ceph-qe-bugs
URL:
Whiteboard:
: 1388079 (view as bug list)
Depends On:
Blocks: 1379890
TreeView+ depends on / blocked
 
Reported: 2016-10-20 14:51 UTC by Tejas
Modified: 2016-11-22 23:41 UTC (History)
12 users (show)

Fixed In Version: ceph-iscsi-config-1.5-1.el7cp
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-22 23:41:39 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:2817 0 normal SHIPPED_LIVE ceph-iscsi-ansible and ceph-ansible bug fix update 2017-04-18 19:50:43 UTC

Comment 11 Mike Christie 2016-10-24 15:11:56 UTC
*** Bug 1388079 has been marked as a duplicate of this bug. ***

Comment 13 Mike Christie 2016-10-27 18:14:36 UTC
Sorry. Forgot to update bz. It made the 1.4-1.el7 rpm.

Comment 15 Hemanth Kumar 2016-10-28 13:25:13 UTC
The issue is still seen with the latest fix : ceph-iscsi-config-1.4-1.el7cp

Created 10 LUNs during the initial run of ansible-playbook
Mounted the Luns on Initiators
added 20 more entries to the ceph gw yml and re-ran the playbook and hit the issue again..

ok: [magna005] => (item={'pool': 'rbd', 'host': 'magna012', 'state': 'present', 'image': 'ansible9', 'size': '90G'})
ok: [magna020] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible10', 'size': '100G'})
ok: [magna015] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible10', 'size': '100G'})
ok: [magna012] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible10', 'size': '100G'})
ok: [magna005] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible10', 'size': '100G'})
changed: [magna005] => (item={'pool': 'rbd', 'host': 'magna005', 'state': 'present', 'image': 'ansible11', 'size': '10G'})
changed: [magna020] => (item={'pool': 'rbd', 'host': 'magna005', 'state': 'present', 'image': 'ansible11', 'size': '10G'})
changed: [magna015] => (item={'pool': 'rbd', 'host': 'magna005', 'state': 'present', 'image': 'ansible11', 'size': '10G'})
changed: [magna012] => (item={'pool': 'rbd', 'host': 'magna005', 'state': 'present', 'image': 'ansible11', 'size': '10G'})
changed: [magna012] => (item={'pool': 'rbd', 'host': 'magna012', 'state': 'present', 'image': 'ansible12', 'size': '20G'})
changed: [magna005] => (item={'pool': 'rbd', 'host': 'magna012', 'state': 'present', 'image': 'ansible12', 'size': '20G'})
changed: [magna020] => (item={'pool': 'rbd', 'host': 'magna012', 'state': 'present', 'image': 'ansible12', 'size': '20G'})
changed: [magna015] => (item={'pool': 'rbd', 'host': 'magna012', 'state': 'present', 'image': 'ansible12', 'size': '20G'})
failed: [magna015] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible13', 'size': '30G'}) => {"failed": true, "item": {"host": "magna015", "image": "ansible13", "pool": "rbd", "size": "30G", "state": "present"}, "parsed": false}
rbd: sysfs write failed
rbd: map failed: (22) Invalid argument
Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1477656004.63-11136253425478/igw_lun", line 1770, in <module>
    ansible_main()
  File "/root/.ansible/tmp/ansible-tmp-1477656004.63-11136253425478/igw_lun", line 1742, in ansible_main
    lun.manage(desired_state)
  File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line 376, in manage
    self.allocate()
  File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line 573, in allocate
    self.config.commit()
  File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/common.py", line 304, in commit
    self.commit_config(post_action)
  File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/common.py", line 266, in _commit_rbd
    current_config = json.loads(ioctx.read(self.config_name))
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Unterminated string starting at: line 252 column 17 (char 8187)
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 8: Applying options for *
debug1: auto-mux: Trying existing master
debug1: mux_client_request_session: master session id: 2
Shared connection to magna015 closed.

failed: [magna012] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible13', 'size': '30G'}) => {"failed": true, "item": {"host": "magna015", "image": "ansible13", "pool": "rbd", "size": "30G", "state": "present"}}
msg: (LUN.allocate) waited too long for the wwn information on image ansible13 to arrive
failed: [magna005] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible13', 'size': '30G'}) => {"failed": true, "item": {"host": "magna015", "image": "ansible13", "pool": "rbd", "size": "30G", "state": "present"}}
msg: (LUN.allocate) waited too long for the wwn information on image ansible13 to arrive
failed: [magna015] => (item={'pool': 'rbd', 'host': 'magna020', 'state': 'present', 'image': 'ansible14', 'size': '40G'}) => {"failed": true, "item": {"host": "magna020", "image": "ansible14", "pool": "rbd", "size": "40G", "state": "present"}}
msg: (LUN.allocate) timed out waiting for rbd to show up
failed: [magna020] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible13', 'size': '30G'}) => {"failed": true, "item": {"host": "magna015", "image": "ansible13", "pool": "rbd", "size": "30G", "state": "present"}}




======================================================================

Comment 21 Paul Cuzner 2016-11-02 04:39:21 UTC
please test against the 1.5 test rpms

Comment 22 John Poelstra 2016-11-02 04:48:22 UTC
Typically we don't move to ON_QA until new downstream packages are actually attached the errata.  Are they? If not, please move this back to MODIFIED.

Also it sounds like there needs to be a second bug opened. Has that happened and if it hasn't, could it? :)

Comment 23 Mike Christie 2016-11-02 05:08:45 UTC
(In reply to John Poelstra from comment #22)
> Also it sounds like there needs to be a second bug opened. Has that happened
> and if it hasn't, could it? :)

Is this about my comment #19? When I wrote it was a QE config issue, I meant they configured the system incorrectly. We didn't make a bz for it, because the fix was to just not configure the system that way.

Comment 24 Tejas 2016-11-02 10:11:31 UTC
Verified in build:
ceph-iscsi-ansible-1.5-1.el7test.noarch
ceph-iscsi-config-1.5-1.el7test.noarch

Created around 35 luns in one playbook run.

Comment 30 Tejas 2016-11-04 10:27:41 UTC
Verified in ceph-iscsi-config-1.5-1.el7cp
created 30+ luns in one run.

Comment 32 errata-xmlrpc 2016-11-22 23:41:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:2817


Note You need to log in before you can comment on or make changes to this bug.