Bug 1387297
| Summary: | [ceph-iscsi-ansible]: "rbd: sysfs write failed" error seen when trying to create a big number of images in one playbook run. | ||
|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Storage Console | Reporter: | Tejas <tchandra> |
| Component: | ceph-ansible | Assignee: | Mike Christie <mchristi> |
| Status: | CLOSED ERRATA | QA Contact: | ceph-qe-bugs <ceph-qe-bugs> |
| Severity: | urgent | Docs Contact: | |
| Priority: | urgent | ||
| Version: | 2 | CC: | adeza, aschoen, ceph-eng-bugs, gmeno, hnallurv, hyelloji, jdillama, kdreyer, mchristi, nthomas, pcuzner, sankarshan |
| Target Milestone: | --- | ||
| Target Release: | 2 | ||
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | ceph-iscsi-config-1.5-1.el7cp | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-11-22 23:41:39 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1379890 | ||
|
Comment 11
Mike Christie
2016-10-24 15:11:56 UTC
Sorry. Forgot to update bz. It made the 1.4-1.el7 rpm. The issue is still seen with the latest fix : ceph-iscsi-config-1.4-1.el7cp
Created 10 LUNs during the initial run of ansible-playbook
Mounted the Luns on Initiators
added 20 more entries to the ceph gw yml and re-ran the playbook and hit the issue again..
ok: [magna005] => (item={'pool': 'rbd', 'host': 'magna012', 'state': 'present', 'image': 'ansible9', 'size': '90G'})
ok: [magna020] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible10', 'size': '100G'})
ok: [magna015] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible10', 'size': '100G'})
ok: [magna012] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible10', 'size': '100G'})
ok: [magna005] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible10', 'size': '100G'})
changed: [magna005] => (item={'pool': 'rbd', 'host': 'magna005', 'state': 'present', 'image': 'ansible11', 'size': '10G'})
changed: [magna020] => (item={'pool': 'rbd', 'host': 'magna005', 'state': 'present', 'image': 'ansible11', 'size': '10G'})
changed: [magna015] => (item={'pool': 'rbd', 'host': 'magna005', 'state': 'present', 'image': 'ansible11', 'size': '10G'})
changed: [magna012] => (item={'pool': 'rbd', 'host': 'magna005', 'state': 'present', 'image': 'ansible11', 'size': '10G'})
changed: [magna012] => (item={'pool': 'rbd', 'host': 'magna012', 'state': 'present', 'image': 'ansible12', 'size': '20G'})
changed: [magna005] => (item={'pool': 'rbd', 'host': 'magna012', 'state': 'present', 'image': 'ansible12', 'size': '20G'})
changed: [magna020] => (item={'pool': 'rbd', 'host': 'magna012', 'state': 'present', 'image': 'ansible12', 'size': '20G'})
changed: [magna015] => (item={'pool': 'rbd', 'host': 'magna012', 'state': 'present', 'image': 'ansible12', 'size': '20G'})
failed: [magna015] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible13', 'size': '30G'}) => {"failed": true, "item": {"host": "magna015", "image": "ansible13", "pool": "rbd", "size": "30G", "state": "present"}, "parsed": false}
rbd: sysfs write failed
rbd: map failed: (22) Invalid argument
Traceback (most recent call last):
File "/root/.ansible/tmp/ansible-tmp-1477656004.63-11136253425478/igw_lun", line 1770, in <module>
ansible_main()
File "/root/.ansible/tmp/ansible-tmp-1477656004.63-11136253425478/igw_lun", line 1742, in ansible_main
lun.manage(desired_state)
File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line 376, in manage
self.allocate()
File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line 573, in allocate
self.config.commit()
File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/common.py", line 304, in commit
self.commit_config(post_action)
File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/common.py", line 266, in _commit_rbd
current_config = json.loads(ioctx.read(self.config_name))
File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Unterminated string starting at: line 252 column 17 (char 8187)
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 8: Applying options for *
debug1: auto-mux: Trying existing master
debug1: mux_client_request_session: master session id: 2
Shared connection to magna015 closed.
failed: [magna012] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible13', 'size': '30G'}) => {"failed": true, "item": {"host": "magna015", "image": "ansible13", "pool": "rbd", "size": "30G", "state": "present"}}
msg: (LUN.allocate) waited too long for the wwn information on image ansible13 to arrive
failed: [magna005] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible13', 'size': '30G'}) => {"failed": true, "item": {"host": "magna015", "image": "ansible13", "pool": "rbd", "size": "30G", "state": "present"}}
msg: (LUN.allocate) waited too long for the wwn information on image ansible13 to arrive
failed: [magna015] => (item={'pool': 'rbd', 'host': 'magna020', 'state': 'present', 'image': 'ansible14', 'size': '40G'}) => {"failed": true, "item": {"host": "magna020", "image": "ansible14", "pool": "rbd", "size": "40G", "state": "present"}}
msg: (LUN.allocate) timed out waiting for rbd to show up
failed: [magna020] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible13', 'size': '30G'}) => {"failed": true, "item": {"host": "magna015", "image": "ansible13", "pool": "rbd", "size": "30G", "state": "present"}}
======================================================================
please test against the 1.5 test rpms Typically we don't move to ON_QA until new downstream packages are actually attached the errata. Are they? If not, please move this back to MODIFIED. Also it sounds like there needs to be a second bug opened. Has that happened and if it hasn't, could it? :) (In reply to John Poelstra from comment #22) > Also it sounds like there needs to be a second bug opened. Has that happened > and if it hasn't, could it? :) Is this about my comment #19? When I wrote it was a QE config issue, I meant they configured the system incorrectly. We didn't make a bz for it, because the fix was to just not configure the system that way. Verified in build: ceph-iscsi-ansible-1.5-1.el7test.noarch ceph-iscsi-config-1.5-1.el7test.noarch Created around 35 luns in one playbook run. Verified in ceph-iscsi-config-1.5-1.el7cp created 30+ luns in one run. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:2817 |