Bug 1814660 - FFU: Data corruption on VMs noticed after FFU completed
Summary: FFU: Data corruption on VMs noticed after FFU completed
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat Storage
Component: RBD
Version: 3.2
Hardware: x86_64
OS: Linux
unspecified
urgent
Target Milestone: rc
: 5.*
Assignee: Jason Dillaman
QA Contact: Madhavi Kasturi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-03-18 13:23 UTC by David Hill
Modified: 2024-01-06 04:28 UTC (History)
21 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-06-17 12:42:52 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHCEPH-1941 0 None None None 2021-09-30 09:17:06 UTC

Description David Hill 2020-03-18 13:23:46 UTC
Description of problem:
FFU: Data corruption on VMs noticed after FFU completed.   The logs below happened while the upgrade was occuring still:

2020-03-12T09:30:05.397190Z qemu-kvm: Failed to set msg fds.
2020-03-12T09:30:05.397209Z qemu-kvm: vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
2020-03-12T09:30:05.397714Z qemu-kvm: Failed to set msg fds.
2020-03-12T09:30:05.397738Z qemu-kvm: vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11)
2020-03-12T09:30:05.397759Z qemu-kvm: Failed to set msg fds.
2020-03-12T09:30:05.397777Z qemu-kvm: vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
2020-03-12T09:30:05.398288Z qemu-kvm: Failed to set msg fds.
2020-03-12T09:30:05.398310Z qemu-kvm: vhost VQ 0 ring restore failed: -1: Resource temporarily unavailable (11)
2020-03-12T09:30:05.398332Z qemu-kvm: Failed to set msg fds.
2020-03-12T09:30:05.398352Z qemu-kvm: vhost VQ 1 ring restore failed: -1: Resource temporarily unavailable (11)
2020-03-12 15:17:06.472+0000: shutting down, reason=crashed
#168 (Associate) Make PublicPrivateCannot set 'Helps Resolution'  0
  
Created By: Dave Hill  (3/18/2020 8:32 AM) Last Modified By: Dave Hill  (3/18/2020 8:48 AM)
[425221.740140] EXT4-fs (dm-3): mounting ext3 file system using the ext4 subsystem
[425221.741241] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[425221.831076] EXT4-fs (dm-4): mounting ext3 file system using the ext4 subsystem
[425221.944995] EXT4-fs warning (device dm-4): ext4_clear_journal_err:4959: Filesystem error recorded from previous mount: IO failure
[425221.944999] EXT4-fs warning (device dm-4): ext4_clear_journal_err:4960: Marking fs in need of filesystem check.
[425221.945278] EXT4-fs (dm-4): warning: mounting fs with errors, running e2fsck is recommended
[425221.945428] EXT4-fs (dm-4): recovery complete
[425221.946685] EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[425222.066271] EXT4-fs (dm-6): mounting ext3 file system using the ext4 subsystem
[425222.152306] EXT4-fs warning (device dm-6): ext4_clear_journal_err:4959: Filesystem error recorded from previous mount: IO failure
[425222.152311] EXT4-fs warning (device dm-6): ext4_clear_journal_err:4960: Marking fs in need of filesystem check.
[425222.152622] EXT4-fs (dm-6): warning: mounting fs with errors, running e2fsck is recommended
[425222.152741] EXT4-fs (dm-6): recovery complete
[425222.154123] EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[425223.211569] EXT4-fs (dm-5): mounting ext3 file system using the ext4 subsystem
[425223.213064] EXT4-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[425224.331347] EXT4-fs (dm-3): mounting ext3 file system using the ext4 subsystem
[425224.333017] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[425224.350268] EXT4-fs (dm-4): mounting ext3 file system using the ext4 subsystem
[425224.352403] EXT4-fs (dm-4): warning: mounting fs with errors, running e2fsck is recommended
[425224.352728] EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[425224.373719] EXT4-fs (dm-6): mounting ext3 file system using the ext4 subsystem
[425224.375021] EXT4-fs (dm-6): warning: mounting fs with errors, running e2fsck is recommended
[425224.375363] EXT4-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[425224.396028] EXT4-fs (dm-5): mounting ext3 file system using the ext4 subsystem
[425224.397502] EXT4-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro

[425499.243202] EXT4-fs (vda1): error count since last fsck: 6
[425499.243208] EXT4-fs (vda1): initial error at time 1584022060: ext4_journal_check_start:56
[425499.243211] EXT4-fs (vda1): last error at time 1584028099: ext4_mb_generate_buddy:757
[425524.843210] EXT4-fs (dm-4): error count since last fsck: 2
[425524.843215] EXT4-fs (dm-4): initial error at time 1584022070: ext4_journal_check_start:56
[425524.843219] EXT4-fs (dm-4): last error at time 1584022070: ext4_journal_check_start:56
[425524.843223] EXT4-fs (dm-6): error count since last fsck: 2
[425524.843225] EXT4-fs (dm-6): initial error at time 1584022075: ext4_journal_check_start:56
[425524.843227] EXT4-fs (dm-6): last error at time 1584022075: ext4_journal_check_start:56


The errors above happened on March 12nd 2020 between 2:07pm and 3:48pm

Version-Release number of selected component (if applicable):
From RHOSP10 -> RHOSP13

How reproducible:
Uncertain

Steps to Reproduce:
1. Need RCA
2.
3.

Actual results:
Rebooting VMs fails

Expected results:
Rebooting VMs succeeds / no missing files

Additional info:

Comment 32 Red Hat Bugzilla 2024-01-06 04:28:32 UTC
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days


Note You need to log in before you can comment on or make changes to this bug.