Fedora Account System
Red Hat Associate
Red Hat Customer
Created attachment 1164086 [details] ceph-installer task e1e52f53-3d4b-489e-84c4-fdaa88ad06a9 output Description of problem: When ceph-disk fails with a Traceback it should not be ignored and ceph-installer task should not exit with return code 0. Version-Release number of selected component (if applicable): ceph-ansible-1.0.5-18.el7scon.noarch ceph-installer-1.0.11-1.el7scon.noarch How reproducible: probably 100% Steps to Reproduce: 1. For example try to create bigger journal for OSD on disk without enough space. It was initially reproduced via creation of cluster in USM, but the root issue was, that it tries to create journal on disk without enough space. Actual results: There is following traceback in the ceph-installer task, but the overall result is "PASS". TASK: [ceph-osd | prepare osd disk(s)] **************************************** failed: [jenkins-usm1-node1.localdomain] => (item=[{'changed': False, 'end': '2016-06-02 10:47:41.209680', 'failed': False, 'stdout': u'', 'cmd': "parted --script /dev/vdd print | egrep -sq '^ 1.*ceph'", 'rc': 1, 'start': '2016-06-02 10:47:41.193943', 'item': u'/dev/vdd', 'warnings': [], 'delta': '0:00:00.015737', 'invocation': {'module_name': u'shell', 'module_complex_args': {}, 'module_args': u"parted --script /dev/vdd print | egrep -sq '^ 1.*ceph'"}, 'stdout_lines': [], 'failed_when_result': False, 'stderr': u''}, {'changed': False, 'end': '2016-06-02 10:47:36.069406', 'failed': False, 'stdout': u'', 'cmd': "echo '/dev/vdd' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", 'rc': 1, 'start': '2016-06-02 10:47:36.064903', 'item': u'/dev/vdd', 'warnings': [], 'delta': '0:00:00.004503', 'invocation': {'module_name': u'shell', 'module_complex_args': {}, 'module_args': u"echo '/dev/vdd' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'"}, 'stdout_lines': [], 'failed_when_result': False, 'stderr': u''}, u'/dev/vdd', u'/dev/vdb']) => {"changed": false, "cmd": ["ceph-disk", "prepare", "--cluster", "TestClusterA", "/dev/vdd", "/dev/vdb"], "delta": "0:00:00.433498", "end": "2016-06-02 10:47:42.377542", "item": [{"changed": false, "cmd": "parted --script /dev/vdd print | egrep -sq '^ 1.*ceph'", "delta": "0:00:00.015737", "end": "2016-06-02 10:47:41.209680", "failed": false, "failed_when_result": false, "invocation": {"module_args": "parted --script /dev/vdd print | egrep -sq '^ 1.*ceph'", "module_complex_args": {}, "module_name": "shell"}, "item": "/dev/vdd", "rc": 1, "start": "2016-06-02 10:47:41.193943", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}, {"changed": false, "cmd": "echo '/dev/vdd' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", "delta": "0:00:00.004503", "end": "2016-06-02 10:47:36.069406", "failed": false, "failed_when_result": false, "invocation": {"module_args": "echo '/dev/vdd' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", "module_complex_args": {}, "module_name": "shell"}, "item": "/dev/vdd", "rc": 1, "start": "2016-06-02 10:47:36.064903", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}, "/dev/vdd", "/dev/vdb"], "rc": 1, "start": "2016-06-02 10:47:41.944044", "stdout_lines": [], "warnings": []} stderr: prepare_device: OSD will not be hot-swappable if journal is not the same device as the osd data Could not create partition 3 from 20973568 to 31459327 Unable to set partition 3's name to 'ceph journal'! Could not change partition 3's type code to 45b0969e-9b03-4f30-b4c6-b4b80ceff106! Error encountered; not saving changes. Traceback (most recent call last): File "/sbin/ceph-disk", line 9, in <module> load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')() File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4993, in run main(sys.argv[1:]) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4946, in main main_catch(args.func, args) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4971, in main_catch func(args) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1774, in main Prepare.factory(args).prepare() File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1762, in prepare self.prepare_locked() File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1794, in prepare_locked self.data.prepare(self.journal) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2446, in prepare self.prepare_device(*to_prepare_list) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2622, in prepare_device to_prepare.prepare() File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1964, in prepare self.prepare_device() File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2054, in prepare_device num=num) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1522, in create_partition self.path, File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 439, in command_check_call return subprocess.check_call(arguments) File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/sbin/sgdisk', '--new=3:0:+5120M', '--change-name=3:ceph journal', '--partition-guid=3:2de7e567-6f62-4bd6-95db-fe864051fabc', '--typecode=3:45b0969e-9b03-4f30-b4c6-b4b80ceff106', '--mbrtogpt', '--', '/dev/vdb']' returned non-zero exit status 4 ...ignoring Expected results: ceph-installer task sould not return 0 exit code, if there is some issue during the OSD or journal creation. Additional info: See the full ceph-installer task output in attachment.
Tested and verified on: USM/ceph-installer Server (RHEL 7.2): ceph-ansible-1.0.5-26.el7scon.noarch ceph-installer-1.0.14-1.el7scon.noarch rhscon-ceph-0.0.33-1.el7scon.x86_64 rhscon-core-0.0.34-1.el7scon.x86_64 rhscon-core-selinux-0.0.34-1.el7scon.noarch rhscon-ui-0.0.47-1.el7scon.noarch Ceph OSD node (RHEL 7.2): ceph-base-10.2.2-5.el7cp.x86_64 ceph-common-10.2.2-5.el7cp.x86_64 ceph-osd-10.2.2-5.el7cp.x86_64 ceph-selinux-10.2.2-5.el7cp.x86_64 libcephfs1-10.2.2-5.el7cp.x86_64 python-cephfs-10.2.2-5.el7cp.x86_64 rhscon-agent-0.0.15-1.el7scon.noarch rhscon-core-selinux-0.0.34-1.el7scon.noarch ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ # ceph-installer task 634ab5ea-5bb5-4dc7-a360-e0824ab1a757 --> endpoint: /api/osd/configure --> succeeded: False --> stdout: <<TRUNCATED>> TASK: [ceph-osd | prepare osd disk(s)] **************************************** failed: [HOST1] => (item=[{'changed': False, 'end': '2016-07-14 15:07:16.122170', 'failed': False, 'stdout': u'', 'cmd': "parted --script /dev/vdh print | egrep -sq '^ 1.*ceph'", 'rc': 1, 'start': '2016-07-14 15:07:16.099744', 'item': u'/dev/vdh', 'warnings': [], 'delta': '0:00:00.022426', 'invocation': {'module_name': u'shell', 'module_complex_args': {}, 'module_args': u"parted --script /dev/vdh print | egrep -sq '^ 1.*ceph'"}, 'stdout_lines': [], 'failed_when_result': False, 'stderr': u''}, {'changed': False, 'end': '2016-07-14 15:07:06.097457', 'failed': False, 'stdout': u'', 'cmd': "echo '/dev/vdh' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", 'rc': 1, 'start': '2016-07-14 15:07:06.089188', 'item': u'/dev/vdh', 'warnings': [], 'delta': '0:00:00.008269', 'invocation': {'module_name': u'shell', 'module_complex_args': {}, 'module_args': u"echo '/dev/vdh' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'"}, 'stdout_lines': [], 'failed_when_result': False, 'stderr': u''}, u'/dev/vdh', u'/dev/vdc']) => {"changed": false, "cmd": ["ceph-disk", "prepare", "--cluster", "TestClusterA", "/dev/vdh", "/dev/vdc"], "delta": "0:00:05.504853", "end": "2016-07-14 15:07:22.855749", "item": [{"changed": false, "cmd": "parted --script /dev/vdh print | egrep -sq '^ 1.*ceph'", "delta": "0:00:00.022426", "end": "2016-07-14 15:07:16.122170", "failed": false, "failed_when_result": false, "invocation": {"module_args": "parted --script /dev/vdh print | egrep -sq '^ 1.*ceph'", "module_complex_args": {}, "module_name": "shell"}, "item": "/dev/vdh", "rc": 1, "start": "2016-07-14 15:07:16.099744", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}, {"changed": false, "cmd": "echo '/dev/vdh' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", "delta": "0:00:00.008269", "end": "2016-07-14 15:07:06.097457", "failed": false, "failed_when_result": false, "invocation": {"module_args": "echo '/dev/vdh' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", "module_complex_args": {}, "module_name": "shell"}, "item": "/dev/vdh", "rc": 1, "start": "2016-07-14 15:07:06.089188", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}, "/dev/vdh", "/dev/vdc"], "rc": 1, "start": "2016-07-14 15:07:17.350896", "warnings": []} stderr: prepare_device: OSD will not be hot-swappable if journal is not the same device as the osd data Could not create partition 1 from 2048 to 10487807 Unable to set partition 1's name to 'ceph journal'! Could not change partition 1's type code to 45b0969e-9b03-4f30-b4c6-b4b80ceff106! Error encountered; not saving changes. Traceback (most recent call last): File "/sbin/ceph-disk", line 9, in <module> load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')() File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4994, in run main(sys.argv[1:]) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4947, in main main_catch(args.func, args) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4972, in main_catch func(args) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1774, in main Prepare.factory(args).prepare() File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1762, in prepare self.prepare_locked() File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1794, in prepare_locked self.data.prepare(self.journal) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2446, in prepare self.prepare_device(*to_prepare_list) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2622, in prepare_device to_prepare.prepare() File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1964, in prepare self.prepare_device() File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2054, in prepare_device num=num) File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1522, in create_partition self.path, File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 439, in command_check_call return subprocess.check_call(arguments) File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/sbin/sgdisk', '--new=1:0:+5120M', '--change-name=1:ceph journal', '--partition-guid=1:ffb82410-8e2e-41ba-aefc-0a4d3be2b4d1', '--typecode=1:45b0969e-9b03-4f30-b4c6-b4b80ceff106', '--mbrtogpt', '--', '/dev/vdc']' returned non-zero exit status 4 FATAL: all hosts have already failed -- aborting PLAY RECAP ******************************************************************** to retry, use: --limit @/var/lib/ceph-installer/osd-configure.retry HOST1 : ok=89 changed=0 unreachable=0 failed=1 HOST2 : ok=4 changed=0 unreachable=0 failed=0 HOST3 : ok=4 changed=0 unreachable=0 failed=0 HOST4 : ok=4 changed=0 unreachable=0 failed=0 --> started: 2016-07-14 15:06:22.282679 --> request: {"cluster_name":"TestClusterA","cluster_network":"IP/20","devices":{"/dev/vdh":"/dev/vdc"},"fsid":"00e4da1e-472c-4119-a2e4-bd1afb0c32b8","host":"HOST1","journal_size":5120,"monitors":[{"address":"IP2","host":"HOST2"},{"address":"IP3","host":"HOST3"},{"address":"IP4","host":"HOST4"}],"public_network":"IP/20","redhat_storage":true} --> exit_code: 2 --> ended: 2016-07-14 15:07:22.916202 --> http_method: POST --> command: /bin/ansible-playbook -v -u ceph-installer /usr/share/ceph-ansible/osd-configure.yml -i /tmp/634ab5ea-5bb5-4dc7-a360-e0824ab1a757_DYGqPD --extra-vars {"raw_journal_devices": ["/dev/vdc"], "devices": ["/dev/vdh"], "cluster": "TestClusterA", "ceph_stable_rh_storage_cdn_install": true, "public_network": "IP/20", "fetch_directory": "/var/lib/ceph-installer/fetch", "cluster_network": "IP/20", "journal_size": 5120, "raw_multi_journal": true, "fsid": "00e4da1e-472c-4119-a2e4-bd1afb0c32b8", "ceph_stable_rh_storage": true} --skip-tags package-install --> user_agent: Go 1.1 package http --> stderr: --> identifier: 634ab5ea-5bb5-4dc7-a360-e0824ab1a757 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Failed ceph-disk command is not ignored. >> VERIFIED
For posterity, this went upstream as https://github.com/ceph/ceph-ansible/pull/837 It shipped downstream in the RH Storage Console 2 GA, https://access.redhat.com/errata/RHEA-2016:1754 (We accidentally failed to attach this particular BZ to that advisory.)