Bug 1618548

The image fails to mount /tmp (/dev/vg/lv_tmp) and /home (/dev/vg/lv_home)

I overwritten the xfs filesystem on both and was able to successfully boot the image.

Note: The failure was spotted on baremetal setup.

Omri Hochman 2018-08-17 10:11:57 EDT
Flags: qe_test_coverage?

we would need to consider to re-run/convert this automation-job to run on BM and update the test-case accordingly.

I am working on testing that now. Images work on vms and isolated on baremetal, but somehow they are failing integrated into tripleo. I am testing with different approaches until we figure the origin of the problem.

As soon as images are copied into the baremetal servers, i can see volumes being created in the undercloud (this should not happen), and I/O errors there:

LANG=EN pvs
   WARNING: Not using lvmetad because duplicate PVs were found.
   WARNING: Use multipath or vgimportclone to resolve duplicate PVs?
   WARNING: After duplicates are resolved, run "pvscan --cache" to enable lvmetad.
   /dev/vg/lv_root: read failed after 0 of 4096 at 0: Input/output error
   /dev/vg/lv_root: read failed after 0 of 4096 at 6438191104: Input/output error
   /dev/vg/lv_root: read failed after 0 of 4096 at 6438248448: Input/output error
   /dev/vg/lv_root: read failed after 0 of 4096 at 4096: Input/output error

Random Errors when booting, on different volumes:

XFS (dm-4): Mounting V5 Filesystem
systemd-journald[735]: Received request to flush runtime journal from PID 1
systemd-journald[735]: File /var/log/journal/xxxx/system.journal corrupted or uncleanly shutdown, renaming and replacing.
XFS (dm-4): Log inconsistent (didn't find previous header)
XFS (dm-4): failed to find log head
XFS (dm-4): log mount/recovery failed: error -5
XFS (dm-4): log mount failed
Give root password for maintenance

Problem is still there. Things that i tried:
- create a config-2 partition, to see if problem was caused when ironic was creating config drive partition
- automount volumes on init, to check if we were hitting https://bugzilla.redhat.com/show_bug.cgi?id=1364092 . It continues failing after that

When looking at boot image, i can see, when booting:

File /var/log/journal/xxx/system.journal corrupted or uncleanly shut down, renaming and replacing. The images that we generate from diskimage-builder, have /vr/log/journal empty.
Ian, can i have your thoughts there?

Ok, centos-minimal builds do not have a /var/log/journal directory.  It looks like we have in /etc/systemd/journald.conf

---
[Journal]
Storage=persistent
---

which I guess is just the default, I'm not aware that we set that in any way.  So by my reading of everything, the systemd-journal service should be happy to create /var/log/journal.  Indeed that's what I see on a live system, where /var/log/journal which must have been created at boot 

---
-bash-4.2# uptime
 05:55:53 up 36 min,  1 user,  load average: 1.45, 0.88, 0.75
-bash-4.2# date --date="36 minutes ago"
Mon Aug 27 05:20:23 UTC 2018
-bash-4.2# ls -l /var/log/journal/
total 8
drwxr-sr-x+ 2 root systemd-journal 4096 Aug 27 05:19 3b9c77317d8a4694ba3e42dfb4ebec92
---

I don't see anything relating to this in the boot logs on a standard system.

@Yolanda; I'm not following the following comment 

> As soon as images are copied into the baremetal servers, 
> i can see volumes being created in the undercloud (this should not happen), and I/O errors there:

What do you mean by "volumes being created in the undercloud"?  I'm wondering if

---
WARNING: Not using lvmetad because duplicate PVs were found.
---

Is a clue here.  If LVM was somehow getting confused about PV's because they have the same UUID or something, that could lead to it mounting the wrong PV in the wrong place; the upper layers would be all messed up and it would present itself as a filesystem corruption?  But I'm not fully following where we are actually seeing those messages.

Am i also understanding correctly that the image that produced as the output of "openstack overcloud image build" works when booted in isolation as a VM?  There seem to be a lot of transitions to making this a RAW image a rolling it out as the whole-disk images; including the 64mb configuration drive that is added (as mentioned).  If that was getting written in the wrong spot ...

We found the origin of the problem. There is this patch:

https://review.openstack.org/#/c/534442/

That added a conv=sparse to the dd copy of the image. This seems to be causing trouble with empty volumes (our /home volume is empty), and it was causing the failure to boot. When removing that patch, it worked fine.

Tried this patch https://code.engineering.redhat.com/gerrit/#/c/150095/ on OSP13.

After applying it had to restart  openstack-ironic-conductor.service and then was able to boot instance with wholedisk image.

Agree with Aviv. No need for signing a new hot fix.

Yolanda - usually in engineering we are responsible for building the package and support gets the pkg signed for delivery to the customer.

Verified:

Environment:
python-ironic-lib-2.12.1-2.el7ost.noarch

The reported issue doesn't reproduce. Successfully booted machines and deployed OC.

According to our records, this should be resolved by python-ironic-lib-2.12.1-2.el7ost.  This build is available now.