Bug 1387297
Summary: | [ceph-iscsi-ansible]: "rbd: sysfs write failed" error seen when trying to create a big number of images in one playbook run. | ||
---|---|---|---|
Product: | [Red Hat Storage] Red Hat Storage Console | Reporter: | Tejas <tchandra> |
Component: | ceph-ansible | Assignee: | Mike Christie <mchristi> |
Status: | CLOSED ERRATA | QA Contact: | ceph-qe-bugs <ceph-qe-bugs> |
Severity: | urgent | Docs Contact: | |
Priority: | urgent | ||
Version: | 2 | CC: | adeza, aschoen, ceph-eng-bugs, gmeno, hnallurv, hyelloji, jdillama, kdreyer, mchristi, nthomas, pcuzner, sankarshan |
Target Milestone: | --- | ||
Target Release: | 2 | ||
Hardware: | Unspecified | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | ceph-iscsi-config-1.5-1.el7cp | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2016-11-22 23:41:39 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | |||
Bug Blocks: | 1379890 |
Comment 11
Mike Christie
2016-10-24 15:11:56 UTC
Sorry. Forgot to update bz. It made the 1.4-1.el7 rpm. The issue is still seen with the latest fix : ceph-iscsi-config-1.4-1.el7cp Created 10 LUNs during the initial run of ansible-playbook Mounted the Luns on Initiators added 20 more entries to the ceph gw yml and re-ran the playbook and hit the issue again.. ok: [magna005] => (item={'pool': 'rbd', 'host': 'magna012', 'state': 'present', 'image': 'ansible9', 'size': '90G'}) ok: [magna020] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible10', 'size': '100G'}) ok: [magna015] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible10', 'size': '100G'}) ok: [magna012] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible10', 'size': '100G'}) ok: [magna005] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible10', 'size': '100G'}) changed: [magna005] => (item={'pool': 'rbd', 'host': 'magna005', 'state': 'present', 'image': 'ansible11', 'size': '10G'}) changed: [magna020] => (item={'pool': 'rbd', 'host': 'magna005', 'state': 'present', 'image': 'ansible11', 'size': '10G'}) changed: [magna015] => (item={'pool': 'rbd', 'host': 'magna005', 'state': 'present', 'image': 'ansible11', 'size': '10G'}) changed: [magna012] => (item={'pool': 'rbd', 'host': 'magna005', 'state': 'present', 'image': 'ansible11', 'size': '10G'}) changed: [magna012] => (item={'pool': 'rbd', 'host': 'magna012', 'state': 'present', 'image': 'ansible12', 'size': '20G'}) changed: [magna005] => (item={'pool': 'rbd', 'host': 'magna012', 'state': 'present', 'image': 'ansible12', 'size': '20G'}) changed: [magna020] => (item={'pool': 'rbd', 'host': 'magna012', 'state': 'present', 'image': 'ansible12', 'size': '20G'}) changed: [magna015] => (item={'pool': 'rbd', 'host': 'magna012', 'state': 'present', 'image': 'ansible12', 'size': '20G'}) failed: [magna015] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible13', 'size': '30G'}) => {"failed": true, "item": {"host": "magna015", "image": "ansible13", "pool": "rbd", "size": "30G", "state": "present"}, "parsed": false} rbd: sysfs write failed rbd: map failed: (22) Invalid argument Traceback (most recent call last): File "/root/.ansible/tmp/ansible-tmp-1477656004.63-11136253425478/igw_lun", line 1770, in <module> ansible_main() File "/root/.ansible/tmp/ansible-tmp-1477656004.63-11136253425478/igw_lun", line 1742, in ansible_main lun.manage(desired_state) File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line 376, in manage self.allocate() File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/lun.py", line 573, in allocate self.config.commit() File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/common.py", line 304, in commit self.commit_config(post_action) File "/usr/lib/python2.7/site-packages/ceph_iscsi_config/common.py", line 266, in _commit_rbd current_config = json.loads(ioctx.read(self.config_name)) File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode obj, end = self.scan_once(s, idx) ValueError: Unterminated string starting at: line 252 column 17 (char 8187) OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 8: Applying options for * debug1: auto-mux: Trying existing master debug1: mux_client_request_session: master session id: 2 Shared connection to magna015 closed. failed: [magna012] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible13', 'size': '30G'}) => {"failed": true, "item": {"host": "magna015", "image": "ansible13", "pool": "rbd", "size": "30G", "state": "present"}} msg: (LUN.allocate) waited too long for the wwn information on image ansible13 to arrive failed: [magna005] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible13', 'size': '30G'}) => {"failed": true, "item": {"host": "magna015", "image": "ansible13", "pool": "rbd", "size": "30G", "state": "present"}} msg: (LUN.allocate) waited too long for the wwn information on image ansible13 to arrive failed: [magna015] => (item={'pool': 'rbd', 'host': 'magna020', 'state': 'present', 'image': 'ansible14', 'size': '40G'}) => {"failed": true, "item": {"host": "magna020", "image": "ansible14", "pool": "rbd", "size": "40G", "state": "present"}} msg: (LUN.allocate) timed out waiting for rbd to show up failed: [magna020] => (item={'pool': 'rbd', 'host': 'magna015', 'state': 'present', 'image': 'ansible13', 'size': '30G'}) => {"failed": true, "item": {"host": "magna015", "image": "ansible13", "pool": "rbd", "size": "30G", "state": "present"}} ====================================================================== please test against the 1.5 test rpms Typically we don't move to ON_QA until new downstream packages are actually attached the errata. Are they? If not, please move this back to MODIFIED. Also it sounds like there needs to be a second bug opened. Has that happened and if it hasn't, could it? :) (In reply to John Poelstra from comment #22) > Also it sounds like there needs to be a second bug opened. Has that happened > and if it hasn't, could it? :) Is this about my comment #19? When I wrote it was a QE config issue, I meant they configured the system incorrectly. We didn't make a bz for it, because the fix was to just not configure the system that way. Verified in build: ceph-iscsi-ansible-1.5-1.el7test.noarch ceph-iscsi-config-1.5-1.el7test.noarch Created around 35 luns in one playbook run. Verified in ceph-iscsi-config-1.5-1.el7cp created 30+ luns in one run. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2016:2817 |