Created attachment 1114843 [details]
udevd debug output

Description of problem:
With the latest RHEL 7.2 deployment images, udev fails to activate ceph OSDs.

This occurs when OSDs are placed on block devices.

Example from /etc/puppet/hieradata/ceph.yaml:

ceph::profile::params::osds:
  '/dev/vdb':
    journal: ''

I also re-deployed a single cluster member with a RHEL 7.1 image and the OSD activated without issue.

Version-Release number of selected component (if applicable):

overcloud-full-7.2.0-46.tar  

How reproducible:
100%

Steps to Reproduce:
1. Overcloud deployment with ceph-storage
2. After deployment, login to a ceph storage node and run ceph -s  and verify no OSDs are online.
3. set udevd into debug logging:  udevadm control --log-priority=debug
4. zap the disk(s): ceph-disk zap /dev/vdb
5. manually run the puppet used during deployment:  puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/*.pp
6. Verify the OSD was created but not activated:  ceph-disk list
7. View udevd debug logs:  journalctl -u systemd-udevd --no-pager -b |less


Actual results:
Ceph OSD is setup but not activated

Expected results:
Ceph OSD to be up and in (ceph osd tree)


Additional info:

traceback from udev debug log.  (log attached to the case)

Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '2016-01-13 14:37:13.147974 7ff73c2b07c0 -1 filestore(/var/lib/ceph/tmp/mnt.ewU5Ko) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.ewU5Ko/journal: (22) Invalid argument'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '2016-01-13 14:37:13.147993 7ff73c2b07c0 -1 OSD::mkfs: ObjectStore::mkfs failed with error -22'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '2016-01-13 14:37:13.148042 7ff73c2b07c0 -1  ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.ewU5Ko: (22) Invalid argument'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) 'ERROR:ceph-disk:Failed to activate'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) 'Traceback (most recent call last):'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '  File "/usr/sbin/ceph-disk", line 2992, in <module>'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '    main()'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '  File "/usr/sbin/ceph-disk", line 2970, in main'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '    args.func(args)'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '  File "/usr/sbin/ceph-disk", line 2180, in main_activate'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '    init=args.mark_init,'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '  File "/usr/sbin/ceph-disk", line 1956, in mount_activate'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '    (osd_id, cluster) = activate(path, activate_key_template, init)'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '  File "/usr/sbin/ceph-disk", line 2119, in activate'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '    keyring=keyring,'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '  File "/usr/sbin/ceph-disk", line 1734, in mkfs'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '    '--keyring', os.path.join(path, 'keyring'),'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '  File "/usr/sbin/ceph-disk", line 323, in command_check_call'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '    return subprocess.check_call(arguments)'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '  File "/lib64/python2.7/subprocess.py", line 542, in check_call'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '    raise CalledProcessError(retcode, cmd)'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) 'subprocess.CalledProcessError: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '2', '--monmap', '/var/lib/ceph/tmp/mnt.ewU5Ko/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.ewU5Ko', '--osd-journal', '/var/lib/ceph/tmp/mnt.ewU5Ko/journal', '--osd-uuid', 'e86d0db8-36ab-4113-89dd-e9f62ced9b53', '--keyring', '/var/lib/ceph/tmp/mnt.ewU5Ko/keyring']' returned non-zero exit status 1'
Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1' [3144] exit with return code 1