Bug 1342117 - when ceph-disk fails with a Traceback it should not be ignored
Summary: when ceph-disk fails with a Traceback it should not be ignored
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Storage Console
Classification: Red Hat Storage
Component: ceph-installer
Version: 2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3
Assignee: Alfredo Deza
QA Contact: Daniel Horák
URL:
Whiteboard:
Depends On:
Blocks: 1343229
TreeView+ depends on / blocked
 
Reported: 2016-06-02 13:17 UTC by Daniel Horák
Modified: 2016-09-23 17:00 UTC (History)
8 users (show)

Fixed In Version: RHEL: ceph-ansible-1.0.5-20.el7scon
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-09-23 17:00:20 UTC
Embargoed:


Attachments (Terms of Use)
ceph-installer task e1e52f53-3d4b-489e-84c4-fdaa88ad06a9 output (107.45 KB, text/plain)
2016-06-02 13:17 UTC, Daniel Horák
no flags Details

Description Daniel Horák 2016-06-02 13:17:38 UTC
Created attachment 1164086 [details]
ceph-installer task e1e52f53-3d4b-489e-84c4-fdaa88ad06a9 output

Description of problem:
  When ceph-disk fails with a Traceback it should not be ignored and ceph-installer task should not exit with return code 0.

Version-Release number of selected component (if applicable):
  ceph-ansible-1.0.5-18.el7scon.noarch
  ceph-installer-1.0.11-1.el7scon.noarch

How reproducible:
  probably 100%

Steps to Reproduce:
1. For example try to create bigger journal for OSD on disk without enough space. 

It was initially reproduced via creation of cluster in USM, but the root issue was, that it tries to create journal on disk without enough space.

Actual results:
  There is following traceback in the ceph-installer task, but the overall result is "PASS".

    TASK: [ceph-osd | prepare osd disk(s)] **************************************** 
  failed: [jenkins-usm1-node1.localdomain] => (item=[{'changed': False, 'end': '2016-06-02 10:47:41.209680', 'failed': False, 'stdout': u'', 'cmd': "parted --script /dev/vdd print | egrep -sq '^ 1.*ceph'", 'rc': 1, 'start': '2016-06-02 10:47:41.193943', 'item': u'/dev/vdd', 'warnings': [], 'delta': '0:00:00.015737', 'invocation': {'module_name': u'shell', 'module_complex_args': {}, 'module_args': u"parted --script /dev/vdd print | egrep -sq '^ 1.*ceph'"}, 'stdout_lines': [], 'failed_when_result': False, 'stderr': u''}, {'changed': False, 'end': '2016-06-02 10:47:36.069406', 'failed': False, 'stdout': u'', 'cmd': "echo '/dev/vdd' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", 'rc': 1, 'start': '2016-06-02 10:47:36.064903', 'item': u'/dev/vdd', 'warnings': [], 'delta': '0:00:00.004503', 'invocation': {'module_name': u'shell', 'module_complex_args': {}, 'module_args': u"echo '/dev/vdd' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'"}, 'stdout_lines': [], 'failed_when_result': False, 'stderr': u''}, u'/dev/vdd', u'/dev/vdb']) => {"changed": false, "cmd": ["ceph-disk", "prepare", "--cluster", "TestClusterA", "/dev/vdd", "/dev/vdb"], "delta": "0:00:00.433498", "end": "2016-06-02 10:47:42.377542", "item": [{"changed": false, "cmd": "parted --script /dev/vdd print | egrep -sq '^ 1.*ceph'", "delta": "0:00:00.015737", "end": "2016-06-02 10:47:41.209680", "failed": false, "failed_when_result": false, "invocation": {"module_args": "parted --script /dev/vdd print | egrep -sq '^ 1.*ceph'", "module_complex_args": {}, "module_name": "shell"}, "item": "/dev/vdd", "rc": 1, "start": "2016-06-02 10:47:41.193943", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}, {"changed": false, "cmd": "echo '/dev/vdd' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", "delta": "0:00:00.004503", "end": "2016-06-02 10:47:36.069406", "failed": false, "failed_when_result": false, "invocation": {"module_args": "echo '/dev/vdd' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", "module_complex_args": {}, "module_name": "shell"}, "item": "/dev/vdd", "rc": 1, "start": "2016-06-02 10:47:36.064903", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}, "/dev/vdd", "/dev/vdb"], "rc": 1, "start": "2016-06-02 10:47:41.944044", "stdout_lines": [], "warnings": []}
  stderr: prepare_device: OSD will not be hot-swappable if journal is not the same device as the osd data
  Could not create partition 3 from 20973568 to 31459327
  Unable to set partition 3's name to 'ceph journal'!
  Could not change partition 3's type code to 45b0969e-9b03-4f30-b4c6-b4b80ceff106!
  Error encountered; not saving changes.
  Traceback (most recent call last):
    File "/sbin/ceph-disk", line 9, in <module>
      load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4993, in run
      main(sys.argv[1:])
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4946, in main
      main_catch(args.func, args)
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4971, in main_catch
      func(args)
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1774, in main
      Prepare.factory(args).prepare()
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1762, in prepare
      self.prepare_locked()
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1794, in prepare_locked
      self.data.prepare(self.journal)
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2446, in prepare
      self.prepare_device(*to_prepare_list)
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2622, in prepare_device
      to_prepare.prepare()
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1964, in prepare
      self.prepare_device()
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2054, in prepare_device
      num=num)
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1522, in create_partition
      self.path,
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 439, in command_check_call
      return subprocess.check_call(arguments)
    File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/sbin/sgdisk', '--new=3:0:+5120M', '--change-name=3:ceph journal', '--partition-guid=3:2de7e567-6f62-4bd6-95db-fe864051fabc', '--typecode=3:45b0969e-9b03-4f30-b4c6-b4b80ceff106', '--mbrtogpt', '--', '/dev/vdb']' returned non-zero exit status 4
  ...ignoring

Expected results:
  ceph-installer task sould not return 0 exit code, if there is some issue during the OSD or journal creation.

Additional info:
  See the full ceph-installer task output in attachment.

Comment 3 Daniel Horák 2016-07-15 08:31:33 UTC
Tested and verified on:
  USM/ceph-installer Server (RHEL 7.2):
  ceph-ansible-1.0.5-26.el7scon.noarch
  ceph-installer-1.0.14-1.el7scon.noarch
  rhscon-ceph-0.0.33-1.el7scon.x86_64
  rhscon-core-0.0.34-1.el7scon.x86_64
  rhscon-core-selinux-0.0.34-1.el7scon.noarch
  rhscon-ui-0.0.47-1.el7scon.noarch
  
  Ceph OSD node (RHEL 7.2):
  ceph-base-10.2.2-5.el7cp.x86_64
  ceph-common-10.2.2-5.el7cp.x86_64
  ceph-osd-10.2.2-5.el7cp.x86_64
  ceph-selinux-10.2.2-5.el7cp.x86_64
  libcephfs1-10.2.2-5.el7cp.x86_64
  python-cephfs-10.2.2-5.el7cp.x86_64
  rhscon-agent-0.0.15-1.el7scon.noarch
  rhscon-core-selinux-0.0.34-1.el7scon.noarch

  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  # ceph-installer task 634ab5ea-5bb5-4dc7-a360-e0824ab1a757
  --> endpoint: /api/osd/configure
  --> succeeded: False
  --> stdout: 
      <<TRUNCATED>>
  TASK: [ceph-osd | prepare osd disk(s)] **************************************** 
  failed: [HOST1] => (item=[{'changed': False, 'end': '2016-07-14 15:07:16.122170', 'failed': False, 'stdout': u'', 'cmd': "parted --script /dev/vdh print | egrep -sq '^ 1.*ceph'", 'rc': 1, 'start': '2016-07-14 15:07:16.099744', 'item': u'/dev/vdh', 'warnings': [], 'delta': '0:00:00.022426', 'invocation': {'module_name': u'shell', 'module_complex_args': {}, 'module_args': u"parted --script /dev/vdh print | egrep -sq '^ 1.*ceph'"}, 'stdout_lines': [], 'failed_when_result': False, 'stderr': u''}, {'changed': False, 'end': '2016-07-14 15:07:06.097457', 'failed': False, 'stdout': u'', 'cmd': "echo '/dev/vdh' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", 'rc': 1, 'start': '2016-07-14 15:07:06.089188', 'item': u'/dev/vdh', 'warnings': [], 'delta': '0:00:00.008269', 'invocation': {'module_name': u'shell', 'module_complex_args': {}, 'module_args': u"echo '/dev/vdh' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'"}, 'stdout_lines': [], 'failed_when_result': False, 'stderr': u''}, u'/dev/vdh', u'/dev/vdc']) => {"changed": false, "cmd": ["ceph-disk", "prepare", "--cluster", "TestClusterA", "/dev/vdh", "/dev/vdc"], "delta": "0:00:05.504853", "end": "2016-07-14 15:07:22.855749", "item": [{"changed": false, "cmd": "parted --script /dev/vdh print | egrep -sq '^ 1.*ceph'", "delta": "0:00:00.022426", "end": "2016-07-14 15:07:16.122170", "failed": false, "failed_when_result": false, "invocation": {"module_args": "parted --script /dev/vdh print | egrep -sq '^ 1.*ceph'", "module_complex_args": {}, "module_name": "shell"}, "item": "/dev/vdh", "rc": 1, "start": "2016-07-14 15:07:16.099744", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}, {"changed": false, "cmd": "echo '/dev/vdh' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", "delta": "0:00:00.008269", "end": "2016-07-14 15:07:06.097457", "failed": false, "failed_when_result": false, "invocation": {"module_args": "echo '/dev/vdh' | egrep '/dev/([hsv]d[a-z]{1,2}|cciss/c[0-9]d[0-9]p|nvme[0-9]n[0-9]p)[0-9]{1,2}$'", "module_complex_args": {}, "module_name": "shell"}, "item": "/dev/vdh", "rc": 1, "start": "2016-07-14 15:07:06.089188", "stderr": "", "stdout": "", "stdout_lines": [], "warnings": []}, "/dev/vdh", "/dev/vdc"], "rc": 1, "start": "2016-07-14 15:07:17.350896", "warnings": []}
  stderr: prepare_device: OSD will not be hot-swappable if journal is not the same device as the osd data
  Could not create partition 1 from 2048 to 10487807
  Unable to set partition 1's name to 'ceph journal'!
  Could not change partition 1's type code to 45b0969e-9b03-4f30-b4c6-b4b80ceff106!
  Error encountered; not saving changes.
  Traceback (most recent call last):
    File "/sbin/ceph-disk", line 9, in <module>
      load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4994, in run
      main(sys.argv[1:])
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4947, in main
      main_catch(args.func, args)
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4972, in main_catch
      func(args)
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1774, in main
      Prepare.factory(args).prepare()
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1762, in prepare
      self.prepare_locked()
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1794, in prepare_locked
      self.data.prepare(self.journal)
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2446, in prepare
      self.prepare_device(*to_prepare_list)
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2622, in prepare_device
      to_prepare.prepare()
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1964, in prepare
      self.prepare_device()
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2054, in prepare_device
      num=num)
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1522, in create_partition
      self.path,
    File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 439, in command_check_call
      return subprocess.check_call(arguments)
    File "/usr/lib64/python2.7/subprocess.py", line 542, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/sbin/sgdisk', '--new=1:0:+5120M', '--change-name=1:ceph journal', '--partition-guid=1:ffb82410-8e2e-41ba-aefc-0a4d3be2b4d1', '--typecode=1:45b0969e-9b03-4f30-b4c6-b4b80ceff106', '--mbrtogpt', '--', '/dev/vdc']' returned non-zero exit status 4
  
  FATAL: all hosts have already failed -- aborting
  
  PLAY RECAP ******************************************************************** 
             to retry, use: --limit @/var/lib/ceph-installer/osd-configure.retry
  
  HOST1 : ok=89   changed=0    unreachable=0    failed=1   
  HOST2 : ok=4    changed=0    unreachable=0    failed=0   
  HOST3 : ok=4    changed=0    unreachable=0    failed=0   
  HOST4 : ok=4    changed=0    unreachable=0    failed=0   
  
  
  --> started: 2016-07-14 15:06:22.282679
  --> request: {"cluster_name":"TestClusterA","cluster_network":"IP/20","devices":{"/dev/vdh":"/dev/vdc"},"fsid":"00e4da1e-472c-4119-a2e4-bd1afb0c32b8","host":"HOST1","journal_size":5120,"monitors":[{"address":"IP2","host":"HOST2"},{"address":"IP3","host":"HOST3"},{"address":"IP4","host":"HOST4"}],"public_network":"IP/20","redhat_storage":true}
  --> exit_code: 2
  --> ended: 2016-07-14 15:07:22.916202
  --> http_method: POST
  --> command: /bin/ansible-playbook -v -u ceph-installer /usr/share/ceph-ansible/osd-configure.yml -i /tmp/634ab5ea-5bb5-4dc7-a360-e0824ab1a757_DYGqPD --extra-vars {"raw_journal_devices": ["/dev/vdc"], "devices": ["/dev/vdh"], "cluster": "TestClusterA", "ceph_stable_rh_storage_cdn_install": true, "public_network": "IP/20", "fetch_directory": "/var/lib/ceph-installer/fetch", "cluster_network": "IP/20", "journal_size": 5120, "raw_multi_journal": true, "fsid": "00e4da1e-472c-4119-a2e4-bd1afb0c32b8", "ceph_stable_rh_storage": true} --skip-tags package-install
  --> user_agent: Go 1.1 package http
  --> stderr: 
  --> identifier: 634ab5ea-5bb5-4dc7-a360-e0824ab1a757
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Failed ceph-disk command is not ignored.

>> VERIFIED

Comment 4 Ken Dreyer (Red Hat) 2016-09-23 17:00:20 UTC
For posterity, this went upstream as https://github.com/ceph/ceph-ansible/pull/837

It shipped downstream in the RH Storage Console 2 GA, https://access.redhat.com/errata/RHEA-2016:1754 (We accidentally failed to attach this particular BZ to that advisory.)


Note You need to log in before you can comment on or make changes to this bug.