Bug 1342117
| Summary: | when ceph-disk fails with a Traceback it should not be ignored | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat Storage Console | Reporter: | Daniel Horák <dahorak> | ||||
| Component: | ceph-installer | Assignee: | Alfredo Deza <adeza> | ||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Daniel Horák <dahorak> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 2 | CC: | adeza, amaredia, aschoen, ceph-eng-bugs, gmeno, kdreyer, nthomas, sankarshan | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 3 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | RHEL: ceph-ansible-1.0.5-20.el7scon | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2016-09-23 17:00:20 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1343229 | ||||||
| Attachments: |
|
||||||
Tested and verified on:
USM/ceph-installer Server (RHEL 7.2):
ceph-ansible-1.0.5-26.el7scon.noarch
ceph-installer-1.0.14-1.el7scon.noarch
rhscon-ceph-0.0.33-1.el7scon.x86_64
rhscon-core-0.0.34-1.el7scon.x86_64
rhscon-core-selinux-0.0.34-1.el7scon.noarch
rhscon-ui-0.0.47-1.el7scon.noarch
Ceph OSD node (RHEL 7.2):
ceph-base-10.2.2-5.el7cp.x86_64
ceph-common-10.2.2-5.el7cp.x86_64
ceph-osd-10.2.2-5.el7cp.x86_64
ceph-selinux-10.2.2-5.el7cp.x86_64
libcephfs1-10.2.2-5.el7cp.x86_64
python-cephfs-10.2.2-5.el7cp.x86_64
rhscon-agent-0.0.15-1.el7scon.noarch
rhscon-core-selinux-0.0.34-1.el7scon.noarch
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# ceph-installer task 634ab5ea-5bb5-4dc7-a360-e0824ab1a757
--> endpoint: /api/osd/configure
--> succeeded: False
--> stdout:
<<TRUNCATED>>
TASK: [ceph-osd | prepare osd disk(s)] ****************************************
failed: [HOST1] => (item=[{'changed': False, 'end': '2016-07-14 15:07:16.122170', 'failed': False, 'stdout': u'', 'cmd': "parted --script /dev/vdh print | egrep -sq '^ 1.*ceph'", 'rc': 1, 'start': '2016-07-14 15:07:16.099744', 'item': u'/dev/vdh', 'warnings': [], 'delta': '0:00:00.022426', 'invocation': {'module_name': u'shell', 'module_complex_args': {}, 'module_args': u"parted --script /dev/vdh print | egrep -sq '^ 1.*ceph'"}, 'stdout_lines': [], 'failed_when_result': False, 'stderr': u''}, {'changed': False, 'end': '2016-07-14 15:07:06.097457', 'failed': False, 'stdout': u'', 'cmd': "echo '/dev/vdh' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", 'rc': 1, 'start': '2016-07-14 15:07:06.089188', 'item': u'/dev/vdh', 'warnings': [], 'delta': '0:00:00.008269', 'invocation': {'module_name': u'shell', 'module_complex_args': {}, 'module_args': u"echo '/dev/vdh' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'"}, 'stdout_lines': [], 'failed_when_result': False, 'stderr': u''}, u'/dev/vdh', u'/dev/vdc']) => {"changed": false, "cmd": ["ceph-disk", "prepare", "--cluster", "TestClusterA", "/dev/vdh", "/dev/vdc"], "delta": "0:00:05.504853", "end": "2016-07-14 15:07:22.855749", "item": [{"changed": false, "cmd": "parted --script /dev/vdh print | egrep -sq '^ 1.*ceph'", "delta": "0:00:00.022426", "end": "2016-07-14 15:07:16.122170", "failed": false, "failed_when_result": false, "invocation": {"module_args": "parted --script /dev/vdh print | egrep -sq '^ 1.*ceph'", "module_complex_args": {}, "module_name": "shell"}, "item": "/dev/vdh", "rc": 1, "start": "2016-07-14 15:07:16.099744", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}, {"changed": false, "cmd": "echo '/dev/vdh' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", "delta": "0:00:00.008269", "end": "2016-07-14 15:07:06.097457", "failed": false, "failed_when_result": false, "invocation": {"module_args": "echo '/dev/vdh' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", "module_complex_args": {}, "module_name": "shell"}, "item": "/dev/vdh", "rc": 1, "start": "2016-07-14 15:07:06.089188", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}, "/dev/vdh", "/dev/vdc"], "rc": 1, "start": "2016-07-14 15:07:17.350896", "warnings": []}
stderr: prepare_device: OSD will not be hot-swappable if journal is not the same device as the osd data
Could not create partition 1 from 2048 to 10487807
Unable to set partition 1's name to 'ceph journal'!
Could not change partition 1's type code to 45b0969e-9b03-4f30-b4c6-b4b80ceff106!
Error encountered; not saving changes.
Traceback (most recent call last):
File "/sbin/ceph-disk", line 9, in <module>
load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4994, in run
main(sys.argv[1:])
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4947, in main
main_catch(args.func, args)
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4972, in main_catch
func(args)
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1774, in main
Prepare.factory(args).prepare()
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1762, in prepare
self.prepare_locked()
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1794, in prepare_locked
self.data.prepare(self.journal)
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2446, in prepare
self.prepare_device(*to_prepare_list)
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2622, in prepare_device
to_prepare.prepare()
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1964, in prepare
self.prepare_device()
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2054, in prepare_device
num=num)
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1522, in create_partition
self.path,
File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 439, in command_check_call
return subprocess.check_call(arguments)
File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/sbin/sgdisk', '--new=1:0:+5120M', '--change-name=1:ceph journal', '--partition-guid=1:ffb82410-8e2e-41ba-aefc-0a4d3be2b4d1', '--typecode=1:45b0969e-9b03-4f30-b4c6-b4b80ceff106', '--mbrtogpt', '--', '/dev/vdc']' returned non-zero exit status 4
FATAL: all hosts have already failed -- aborting
PLAY RECAP ********************************************************************
to retry, use: --limit @/var/lib/ceph-installer/osd-configure.retry
HOST1 : ok=89 changed=0 unreachable=0 failed=1
HOST2 : ok=4 changed=0 unreachable=0 failed=0
HOST3 : ok=4 changed=0 unreachable=0 failed=0
HOST4 : ok=4 changed=0 unreachable=0 failed=0
--> started: 2016-07-14 15:06:22.282679
--> request: {"cluster_name":"TestClusterA","cluster_network":"IP/20","devices":{"/dev/vdh":"/dev/vdc"},"fsid":"00e4da1e-472c-4119-a2e4-bd1afb0c32b8","host":"HOST1","journal_size":5120,"monitors":[{"address":"IP2","host":"HOST2"},{"address":"IP3","host":"HOST3"},{"address":"IP4","host":"HOST4"}],"public_network":"IP/20","redhat_storage":true}
--> exit_code: 2
--> ended: 2016-07-14 15:07:22.916202
--> http_method: POST
--> command: /bin/ansible-playbook -v -u ceph-installer /usr/share/ceph-ansible/osd-configure.yml -i /tmp/634ab5ea-5bb5-4dc7-a360-e0824ab1a757_DYGqPD --extra-vars {"raw_journal_devices": ["/dev/vdc"], "devices": ["/dev/vdh"], "cluster": "TestClusterA", "ceph_stable_rh_storage_cdn_install": true, "public_network": "IP/20", "fetch_directory": "/var/lib/ceph-installer/fetch", "cluster_network": "IP/20", "journal_size": 5120, "raw_multi_journal": true, "fsid": "00e4da1e-472c-4119-a2e4-bd1afb0c32b8", "ceph_stable_rh_storage": true} --skip-tags package-install
--> user_agent: Go 1.1 package http
--> stderr:
--> identifier: 634ab5ea-5bb5-4dc7-a360-e0824ab1a757
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Failed ceph-disk command is not ignored.
>> VERIFIED
For posterity, this went upstream as https://github.com/ceph/ceph-ansible/pull/837 It shipped downstream in the RH Storage Console 2 GA, https://access.redhat.com/errata/RHEA-2016:1754 (We accidentally failed to attach this particular BZ to that advisory.) |
Created attachment 1164086 [details] ceph-installer task e1e52f53-3d4b-489e-84c4-fdaa88ad06a9 output Description of problem: When ceph-disk fails with a Traceback it should not be ignored and ceph-installer task should not exit with return code 0. Version-Release number of selected component (if applicable): ceph-ansible-1.0.5-18.el7scon.noarch ceph-installer-1.0.11-1.el7scon.noarch How reproducible: probably 100% Steps to Reproduce: 1. For example try to create bigger journal for OSD on disk without enough space. It was initially reproduced via creation of cluster in USM, but the root issue was, that it tries to create journal on disk without enough space. Actual results: There is following traceback in the ceph-installer task, but the overall result is "PASS". TASK: [ceph-osd | prepare osd disk(s)] **************************************** failed: [jenkins-usm1-node1.localdomain] => (item=[{'changed': False, 'end': '2016-06-02 10:47:41.209680', 'failed': False, 'stdout': u'', 'cmd': "parted --script /dev/vdd print | egrep -sq '^ 1.*ceph'", 'rc': 1, 'start': '2016-06-02 10:47:41.193943', 'item': u'/dev/vdd', 'warnings': [], 'delta': '0:00:00.015737', 'invocation': {'module_name': u'shell', 'module_complex_args': {}, 'module_args': u"parted --script /dev/vdd print | egrep -sq '^ 1.*ceph'"}, 'stdout_lines': [], 'failed_when_result': False, 'stderr': u''}, {'changed': False, 'end': '2016-06-02 10:47:36.069406', 'failed': False, 'stdout': u'', 'cmd': "echo '/dev/vdd' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", 'rc': 1, 'start': '2016-06-02 10:47:36.064903', 'item': u'/dev/vdd', 'warnings': [], 'delta': '0:00:00.004503', 'invocation': {'module_name': u'shell', 'module_complex_args': {}, 'module_args': u"echo '/dev/vdd' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'"}, 'stdout_lines': [], 'failed_when_result': False, 'stderr': u''}, u'/dev/vdd', u'/dev/vdb']) => {"changed": false, "cmd": ["ceph-disk", "prepare", "--cluster", "TestClusterA", "/dev/vdd", "/dev/vdb"], "delta": "0:00:00.433498", "end": "2016-06-02 10:47:42.377542", "item": [{"changed": false, "cmd": "parted --script /dev/vdd print | egrep -sq '^ 1.*ceph'", "delta": "0:00:00.015737", "end": "2016-06-02 10:47:41.209680", "failed": false, "failed_when_result": false, "invocation": {"module_args": "parted --script /dev/vdd print | egrep -sq '^ 1.*ceph'", "module_complex_args": {}, "module_name": "shell"}, "item": "/dev/vdd", "rc": 1, "start": "2016-06-02 10:47:41.193943", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}, {"changed": false, "cmd": "echo '/dev/vdd' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", "delta": "0:00:00.004503", "end": "2016-06-02 10:47:36.069406", "failed": false, "failed_when_result": false, "invocation": {"module_args": "echo '/dev/vdd' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", "module_complex_args": {}, "module_name": "shell"}, "item": "/dev/vdd", "rc": 1, "start": "2016-06-02 10:47:36.064903", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}, "/dev/vdd", "/dev/vdb"], "rc": 1, "start": "2016-06-02 10:47:41.944044", "stdout_lines": [], "warnings": []} stderr: prepare_device: OSD will not be hot-swappable if journal is not the same device as the osd data Could not create partition 3 from 20973568 to 31459327 Unable to set partition 3's name to 'ceph journal'! Could not change partition 3's type code to 45b0969e-9b03-4f30-b4c6-b4b80ceff106! Error encountered; not saving changes. Traceback (most recent call last): File "/sbin/ceph-disk", line 9, in <module> load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')() File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4993, in run main(sys.argv[1:]) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4946, in main main_catch(args.func, args) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4971, in main_catch func(args) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1774, in main Prepare.factory(args).prepare() File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1762, in prepare self.prepare_locked() File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1794, in prepare_locked self.data.prepare(self.journal) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2446, in prepare self.prepare_device(*to_prepare_list) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2622, in prepare_device to_prepare.prepare() File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1964, in prepare self.prepare_device() File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2054, in prepare_device num=num) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1522, in create_partition self.path, File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 439, in command_check_call return subprocess.check_call(arguments) File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/sbin/sgdisk', '--new=3:0:+5120M', '--change-name=3:ceph journal', '--partition-guid=3:2de7e567-6f62-4bd6-95db-fe864051fabc', '--typecode=3:45b0969e-9b03-4f30-b4c6-b4b80ceff106', '--mbrtogpt', '--', '/dev/vdb']' returned non-zero exit status 4 ...ignoring Expected results: ceph-installer task sould not return 0 exit code, if there is some issue during the OSD or journal creation. Additional info: See the full ceph-installer task output in attachment.