Bug 1585482
| Summary: | OSPd unable to deploy RHCS3.0 (Bluestore) Error : bluestore mkfs fsck found fatal Input/output error | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | karan singh <karan> |
| Component: | openstack-tripleo | Assignee: | Giulio Fidente <gfidente> |
| Status: | CLOSED DUPLICATE | QA Contact: | Arik Chernetsky <achernet> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 12.0 (Pike) | CC: | bschmaus, gfidente, karan, mburns |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-06-07 18:01:14 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
karan singh
2018-06-03 12:37:34 UTC
Based on email discussion with John Fulton i retried this by performing node cleanup.
1. Set automated_clean=true in ironic.conf, restart services
2. Set nodes to manage, set nodes back to available. Ironic did node cleanup before setting the nodes to available.
(undercloud) [stack@refarch-r220-03 ~]$ openstack baremetal node list
+--------------------------------------+-----------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+-----------+---------------+-------------+--------------------+-------------+
| 90065bea-6a60-4c94-833d-d9c7843fb735 | r630-01 | None | power on | clean wait | False |
| 5866f163-f48c-4a5a-8c81-c59041f31322 | r630-02 | None | power on | clean wait | False |
| 0c6a190e-a14f-4bf0-a8c1-09535311819c | r630-03 | None | power on | clean wait | False |
| 29b252d7-da76-483c-a834-829f7b3c3144 | r220-01 | None | power on | clean wait | False |
| 5581154b-9992-419d-9916-971e299f0673 | r220-08 | None | power on | clean wait | False |
| a51790ee-25ac-4400-ac7b-03de2024b9f4 | r220-09 | None | power on | clean wait | False |
| 9f60fa30-fad2-4d28-8759-be7a1480bb98 | r220-10 | None | power on | clean wait | False |
| 58cfae5f-c650-45a2-ad95-c23c1f71eada | r730xd-01 | None | power off | cleaning | False |
| dda77bfa-291a-40e7-acfb-5c1fae6cf561 | r730xd-02 | None | power off | cleaning | False |
| 87072bba-765a-4e21-83c3-1ec3d07d07b6 | r730xd-03 | None | power off | cleaning | False |
| b26921d4-7a33-4815-9069-d0430eb773fe | r730xd-04 | None | power off | cleaning | False |
| 8c4dfe4f-d0e2-4876-9509-42ae3d29fa27 | r730xd-05 | None | power off | cleaning | False |
+--------------------------------------+-----------+---------------+-------------+--------------------+-------------+
(undercloud) [stack@refarch-r220-03 ~]$
3. Triggered OSPd to deplo OSP and Ceph. But still encountering the exact same issue
[heat-admin@controller-0 ~]$ sudo ceph -s
cluster:
id: b8a0918c-5d05-11e8-962f-2047478cce5e
health: HEALTH_WARN
Reduced data availability: 160 pgs inactive
services:
mon: 1 daemons, quorum controller-0
mgr: controller-0(active)
osd: 60 osds: 0 up, 0 in <<<< No OSD are UP/IN
data:
pools: 5 pools, 160 pgs
objects: 0 objects, 0 bytes
usage: 0 kB used, 0 kB / 0 kB avail
pgs: 100.000% pgs unknown
160 unknown
[heat-admin@controller-0 ~]$
OSD logs
-----------
Jun 03 20:10:46 ceph-storage-1 docker[106640]: Error response from daemon: No such container: ceph-osd-ceph-storage-1-sdh
Jun 03 20:10:46 ceph-storage-1 docker[106673]: Error response from daemon: No such container: ceph-osd-ceph-storage-1-sdh
Jun 03 20:10:46 ceph-storage-1 systemd[1]: Started Ceph OSD.
Jun 03 20:10:47 ceph-storage-1 ceph-osd-run.sh[106706]: Error response from daemon: No such container: expose_partitions_sdh
Jun 03 20:10:49 ceph-storage-1 ceph-osd-run.sh[106706]: 2018-06-03 20:10:49 /entrypoint.sh: static: does not generate config
Jun 03 20:10:49 ceph-storage-1 ceph-osd-run.sh[106706]: main_activate: path = /dev/sdh1
Jun 03 20:10:51 ceph-storage-1 ceph-osd-run.sh[106706]: get_dm_uuid: get_dm_uuid /dev/sdh1 uuid path is /sys/dev/block/8:113/dm/uuid
Jun 03 20:10:51 ceph-storage-1 ceph-osd-run.sh[106706]: command: Running command: /usr/sbin/blkid -o udev -p /dev/sdh1
Jun 03 20:10:51 ceph-storage-1 ceph-osd-run.sh[106706]: command: Running command: /sbin/blkid -p -s TYPE -o value -- /dev/sdh1
Jun 03 20:10:51 ceph-storage-1 ceph-osd-run.sh[106706]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
Jun 03 20:10:52 ceph-storage-1 ceph-osd-run.sh[106706]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
Jun 03 20:10:52 ceph-storage-1 ceph-osd-run.sh[106706]: mount: Mounting /dev/sdh1 on /var/lib/ceph/tmp/mnt.VwM4I_ with options noatime,inode64
Jun 03 20:10:52 ceph-storage-1 ceph-osd-run.sh[106706]: command_check_call: Running command: /usr/bin/mount -t xfs -o noatime,inode64 -- /dev/sdh1 /var/lib/ceph/tmp/mnt.VwM4I_
Jun 03 20:10:52 ceph-storage-1 ceph-osd-run.sh[106706]: command: Running command: /usr/sbin/restorecon /var/lib/ceph/tmp/mnt.VwM4I_
Jun 03 20:10:52 ceph-storage-1 ceph-osd-run.sh[106706]: activate: Cluster uuid is b8a0918c-5d05-11e8-962f-2047478cce5e
Jun 03 20:10:52 ceph-storage-1 ceph-osd-run.sh[106706]: command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
Jun 03 20:10:52 ceph-storage-1 ceph-osd-run.sh[106706]: activate: Cluster name is ceph
Jun 03 20:10:52 ceph-storage-1 ceph-osd-run.sh[106706]: activate: OSD uuid is 03b5dab0-ff57-4645-a9b2-4c106777833e
Jun 03 20:10:52 ceph-storage-1 ceph-osd-run.sh[106706]: activate: OSD id is 35
Jun 03 20:10:52 ceph-storage-1 ceph-osd-run.sh[106706]: activate: Initializing OSD...
Jun 03 20:10:52 ceph-storage-1 ceph-osd-run.sh[106706]: command_check_call: Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/tmp/mnt.VwM4I_/activate.monmap
Jun 03 20:10:53 ceph-storage-1 ceph-osd-run.sh[106706]: got monmap epoch 1
Jun 03 20:10:53 ceph-storage-1 ceph-osd-run.sh[106706]: command_check_call: Running command: /usr/bin/ceph-osd --cluster ceph --mkfs -i 35 --monmap /var/lib/ceph/tmp/mnt.VwM4I_/activate.monmap --osd-data /var/lib/ceph/tmp/mnt.VwM4I_ --osd-uuid 03b5dab0-ff57-4645-a9b2-4c106777833e --setuser ceph --setgroup disk
Jun 03 20:10:53 ceph-storage-1 ceph-osd-run.sh[106706]: 2018-06-03 20:10:53.220245 7f322512ad80 -1 bluestore(/var/lib/ceph/tmp/mnt.VwM4I_/block) _check_or_set_bdev_label bdev /var/lib/ceph/tmp/mnt.VwM4I_/block fsid a1a13449-f826-4899-8a8c-eb8c905ab9c6 does not match our fsid 03b5dab0-ff57-4645-a9b2-4c106777833e
Jun 03 20:10:53 ceph-storage-1 ceph-osd-run.sh[106706]: 2018-06-03 20:10:53.475778 7f322512ad80 -1 bluestore(/var/lib/ceph/tmp/mnt.VwM4I_) mkfs fsck found fatal error: (5) Input/output error
Jun 03 20:10:53 ceph-storage-1 ceph-osd-run.sh[106706]: 2018-06-03 20:10:53.475812 7f322512ad80 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
Jun 03 20:10:53 ceph-storage-1 ceph-osd-run.sh[106706]: 2018-06-03 20:10:53.475912 7f322512ad80 -1 ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.VwM4I_: (5) Input/output error
Jun 03 20:10:53 ceph-storage-1 ceph-osd-run.sh[106706]: mount_activate: Failed to activate
Jun 03 20:10:53 ceph-storage-1 ceph-osd-run.sh[106706]: unmount: Unmounting /var/lib/ceph/tmp/mnt.VwM4I_
|