Bug 1298620 - udev fails to activate ceph OSD, OSP Director Deployment, RHEL 7.2 images
udev fails to activate ceph OSD, OSP Director Deployment, RHEL 7.2 images
Status: CLOSED DUPLICATE of bug 1297251
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
x86_64 Linux
high Severity high
: ga
: 8.0 (Liberty)
Assigned To: Hugh Brock
yeylon@redhat.com
:
Depends On:
Blocks: 1261979
  Show dependency treegraph
 
Reported: 2016-01-14 10:13 EST by Matt Flusche
Modified: 2016-04-18 03:14 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-08 12:25:02 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
udevd debug output (130.96 KB, text/plain)
2016-01-14 10:13 EST, Matt Flusche
no flags Details

  None (edit)
Description Matt Flusche 2016-01-14 10:13:30 EST
Created attachment 1114843 [details]
udevd debug output

Description of problem:
With the latest RHEL 7.2 deployment images, udev fails to activate ceph OSDs.

This occurs when OSDs are placed on block devices.

Example from /etc/puppet/hieradata/ceph.yaml:

ceph::profile::params::osds:
  '/dev/vdb':
    journal: ''

I also re-deployed a single cluster member with a RHEL 7.1 image and the OSD activated without issue.

Version-Release number of selected component (if applicable):

overcloud-full-7.2.0-46.tar  

How reproducible:
100%

Steps to Reproduce:
1. Overcloud deployment with ceph-storage
2. After deployment, login to a ceph storage node and run ceph -s  and verify no OSDs are online.
3. set udevd into debug logging:  udevadm control --log-priority=debug
4. zap the disk(s): ceph-disk zap /dev/vdb
5. manually run the puppet used during deployment:  puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/*.pp
6. Verify the OSD was created but not activated:  ceph-disk list
7. View udevd debug logs:  journalctl -u systemd-udevd --no-pager -b |less


Actual results:
Ceph OSD is setup but not activated

Expected results:
Ceph OSD to be up and in (ceph osd tree)


Additional info:

traceback from udev debug log.  (log attached to the case)

Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '2016-01-13 14:37:13.147974 7ff73c2b07c0 -1 filestore(/var/lib/ceph/tmp/mnt.ewU5Ko) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.ewU5Ko/journal: (22) Invalid argument'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '2016-01-13 14:37:13.147993 7ff73c2b07c0 -1 OSD::mkfs: ObjectStore::mkfs failed with error -22'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '2016-01-13 14:37:13.148042 7ff73c2b07c0 -1  ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.ewU5Ko: (22) Invalid argument'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) 'ERROR:ceph-disk:Failed to activate'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) 'Traceback (most recent call last):'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '  File "/usr/sbin/ceph-disk", line 2992, in <module>'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '    main()'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '  File "/usr/sbin/ceph-disk", line 2970, in main'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '    args.func(args)'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '  File "/usr/sbin/ceph-disk", line 2180, in main_activate'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '    init=args.mark_init,'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '  File "/usr/sbin/ceph-disk", line 1956, in mount_activate'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '    (osd_id, cluster) = activate(path, activate_key_template, init)'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '  File "/usr/sbin/ceph-disk", line 2119, in activate'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '    keyring=keyring,'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '  File "/usr/sbin/ceph-disk", line 1734, in mkfs'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '    '--keyring', os.path.join(path, 'keyring'),'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '  File "/usr/sbin/ceph-disk", line 323, in command_check_call'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '    return subprocess.check_call(arguments)'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '  File "/lib64/python2.7/subprocess.py", line 542, in check_call'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '    raise CalledProcessError(retcode, cmd)'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) 'subprocess.CalledProcessError: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '2', '--monmap', '/var/lib/ceph/tmp/mnt.ewU5Ko/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.ewU5Ko', '--osd-journal', '/var/lib/ceph/tmp/mnt.ewU5Ko/journal', '--osd-uuid', 'e86d0db8-36ab-4113-89dd-e9f62ced9b53', '--keyring', '/var/lib/ceph/tmp/mnt.ewU5Ko/keyring']' returned non-zero exit status 1'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1' [3144] exit with return code 1
Comment 2 Giulio Fidente 2016-02-08 12:25:02 EST

*** This bug has been marked as a duplicate of bug 1297251 ***

Note You need to log in before you can comment on or make changes to this bug.