| Summary: | udev fails to activate ceph OSD, OSP Director Deployment, RHEL 7.2 images | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Matt Flusche <mflusche> | ||||
| Component: | rhosp-director | Assignee: | Hugh Brock <hbrock> | ||||
| Status: | CLOSED DUPLICATE | QA Contact: | yeylon <yeylon> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 7.0 (Kilo) | CC: | alan_bishop, arkady_kanevsky, gfidente, mburns, rhel-osp-director-maint, srevivo | ||||
| Target Milestone: | ga | ||||||
| Target Release: | 8.0 (Liberty) | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2016-02-08 17:25:02 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1261979 | ||||||
| Attachments: |
|
||||||
*** This bug has been marked as a duplicate of bug 1297251 *** |
Created attachment 1114843 [details] udevd debug output Description of problem: With the latest RHEL 7.2 deployment images, udev fails to activate ceph OSDs. This occurs when OSDs are placed on block devices. Example from /etc/puppet/hieradata/ceph.yaml: ceph::profile::params::osds: '/dev/vdb': journal: '' I also re-deployed a single cluster member with a RHEL 7.1 image and the OSD activated without issue. Version-Release number of selected component (if applicable): overcloud-full-7.2.0-46.tar How reproducible: 100% Steps to Reproduce: 1. Overcloud deployment with ceph-storage 2. After deployment, login to a ceph storage node and run ceph -s and verify no OSDs are online. 3. set udevd into debug logging: udevadm control --log-priority=debug 4. zap the disk(s): ceph-disk zap /dev/vdb 5. manually run the puppet used during deployment: puppet apply --detailed-exitcodes /var/lib/heat-config/heat-config-puppet/*.pp 6. Verify the OSD was created but not activated: ceph-disk list 7. View udevd debug logs: journalctl -u systemd-udevd --no-pager -b |less Actual results: Ceph OSD is setup but not activated Expected results: Ceph OSD to be up and in (ceph osd tree) Additional info: traceback from udev debug log. (log attached to the case) Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '2016-01-13 14:37:13.147974 7ff73c2b07c0 -1 filestore(/var/lib/ceph/tmp/mnt.ewU5Ko) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.ewU5Ko/journal: (22) Invalid argument' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '2016-01-13 14:37:13.147993 7ff73c2b07c0 -1 OSD::mkfs: ObjectStore::mkfs failed with error -22' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) '2016-01-13 14:37:13.148042 7ff73c2b07c0 -1 ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.ewU5Ko: (22) Invalid argument' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) 'ERROR:ceph-disk:Failed to activate' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) 'Traceback (most recent call last):' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) ' File "/usr/sbin/ceph-disk", line 2992, in <module>' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) ' main()' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) ' File "/usr/sbin/ceph-disk", line 2970, in main' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) ' args.func(args)' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) ' File "/usr/sbin/ceph-disk", line 2180, in main_activate' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) ' init=args.mark_init,' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) ' File "/usr/sbin/ceph-disk", line 1956, in mount_activate' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) ' (osd_id, cluster) = activate(path, activate_key_template, init)' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) ' File "/usr/sbin/ceph-disk", line 2119, in activate' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) ' keyring=keyring,' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) ' File "/usr/sbin/ceph-disk", line 1734, in mkfs' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) ' '--keyring', os.path.join(path, 'keyring'),' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) ' File "/usr/sbin/ceph-disk", line 323, in command_check_call' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) ' return subprocess.check_call(arguments)' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) ' File "/lib64/python2.7/subprocess.py", line 542, in check_call' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) ' raise CalledProcessError(retcode, cmd)' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1'(err) 'subprocess.CalledProcessError: Command '['/usr/bin/ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', '2', '--monmap', '/var/lib/ceph/tmp/mnt.ewU5Ko/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.ewU5Ko', '--osd-journal', '/var/lib/ceph/tmp/mnt.ewU5Ko/journal', '--osd-uuid', 'e86d0db8-36ab-4113-89dd-e9f62ced9b53', '--keyring', '/var/lib/ceph/tmp/mnt.ewU5Ko/keyring']' returned non-zero exit status 1' Jan 13 14:37:13 overcloud-cephstorage-1.localdomain systemd-udevd[2999]: '/usr/sbin/ceph-disk-activate /dev/vdb1' [3144] exit with return code 1