Bug 1814660 - FFU: Data corruption on VMs noticed after FFU completed [NEEDINFO]
Summary: FFU: Data corruption on VMs noticed after FFU completed
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RBD
Version: 3.2
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: rc
: 5.*
Assignee: Jason Dillaman
QA Contact: Madhavi Kasturi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-18 13:23 UTC by David Hill
Modified: 2021-09-30 09:17 UTC (History)
21 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-06-17 12:42:52 UTC
Target Upstream Version:
johfulto: needinfo? (kchamart)

Attachments (Terms of Use)
Add an attachment (proposed patch, testcase, etc.)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHCEPH-1941 0 None None None 2021-09-30 09:17:06 UTC

Description David Hill 2020-03-18 13:23:46 UTC 
Description of problem:
FFU: Data corruption on VMs noticed after FFU completed.   The logs below happened while the upgrade was occuring still:

2020-03-12T09:30:05.397190Z qemu-kvm: Failed to set msg fds.
2020-03-12T09:30:05.397209Z qemu-kvm: vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
2020-03-12T09:30:05.397714Z qemu-kvm: Failed to set msg fds.
2020-03-12T09:30:05.397738Z qemu-kvm: vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11)
2020-03-12T09:30:05.397759Z qemu-kvm: Failed to set msg fds.
2020-03-12T09:30:05.397777Z qemu-kvm: vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
2020-03-12T09:30:05.398288Z qemu-kvm: Failed to set msg fds.
2020-03-12T09:30:05.398310Z qemu-kvm: vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11)
2020-03-12T09:30:05.398332Z qemu-kvm: Failed to set msg fds.
2020-03-12T09:30:05.398352Z qemu-kvm: vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
2020-03-12 15:17:06.472+0000: shutting down, reason=crashed
#168 (Associate) Make PublicPrivateCannot set 'Helps Resolution'  0
  
Created By: Dave Hill  (3/18/2020 8:32 AM) Last Modified By: Dave Hill  (3/18/2020 8:48 AM)
[425221.740140] EXT4-fs (dm-3): mounting ext3 file system using the ext4 subsystem
[425221.741241] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[425221.831076] EXT4-fs (dm-4): mounting ext3 file system using the ext4 subsystem
[425221.944995] EXT4-fs warning (device dm-4): ext4_clear_journal_err:4959: Filesystem error recorded from previous mount: IO failure
[425221.944999] EXT4-fs warning (device dm-4): ext4_clear_journal_err:4960: Marking fs in need of filesystem check.
[425221.945278] EXT4-fs (dm-4): warning: mounting fs with errors, running e2fsck is recommended
[425221.945428] EXT4-fs (dm-4): recovery complete
[425221.946685] EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[425222.066271] EXT4-fs (dm-6): mounting ext3 file system using the ext4 subsystem
[425222.152306] EXT4-fs warning (device dm-6): ext4_clear_journal_err:4959: Filesystem error recorded from previous mount: IO failure
[425222.152311] EXT4-fs warning (device dm-6): ext4_clear_journal_err:4960: Marking fs in need of filesystem check.
[425222.152622] EXT4-fs (dm-6): warning: mounting fs with errors, running e2fsck is recommended
[425222.152741] EXT4-fs (dm-6): recovery complete
[425222.154123] EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[425223.211569] EXT4-fs (dm-5): mounting ext3 file system using the ext4 subsystem
[425223.213064] EXT4-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[425224.331347] EXT4-fs (dm-3): mounting ext3 file system using the ext4 subsystem
[425224.333017] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[425224.350268] EXT4-fs (dm-4): mounting ext3 file system using the ext4 subsystem
[425224.352403] EXT4-fs (dm-4): warning: mounting fs with errors, running e2fsck is recommended
[425224.352728] EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[425224.373719] EXT4-fs (dm-6): mounting ext3 file system using the ext4 subsystem
[425224.375021] EXT4-fs (dm-6): warning: mounting fs with errors, running e2fsck is recommended
[425224.375363] EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[425224.396028] EXT4-fs (dm-5): mounting ext3 file system using the ext4 subsystem
[425224.397502] EXT4-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro

[425499.243202] EXT4-fs (vda1): error count since last fsck: 6
[425499.243208] EXT4-fs (vda1): initial error at time 1584022060: ext4_journal_check_start:56
[425499.243211] EXT4-fs (vda1): last error at time 1584028099: ext4_mb_generate_buddy:757
[425524.843210] EXT4-fs (dm-4): error count since last fsck: 2
[425524.843215] EXT4-fs (dm-4): initial error at time 1584022070: ext4_journal_check_start:56
[425524.843219] EXT4-fs (dm-4): last error at time 1584022070: ext4_journal_check_start:56
[425524.843223] EXT4-fs (dm-6): error count since last fsck: 2
[425524.843225] EXT4-fs (dm-6): initial error at time 1584022075: ext4_journal_check_start:56
[425524.843227] EXT4-fs (dm-6): last error at time 1584022075: ext4_journal_check_start:56


The errors above happened on March 12nd 2020 between 2:07pm and 3:48pm

Version-Release number of selected component (if applicable):
From RHOSP10 -> RHOSP13

How reproducible:
Uncertain

Steps to Reproduce:
1. Need RCA
2.
3.

Actual results:
Rebooting VMs fails

Expected results:
Rebooting VMs succeeds / no missing files

Additional info:
Note You need to log in before you can comment on or make changes to this bug.