Hide Forgot
Description of problem: When upgrading a 7.6 system that was previously upgraded from RHEL 6, a crash happens in overlayfs (tracked by BZ #1789473). Internally, leapp mounts /proc onto /var/lib/leapp/scratch/mounts/root_proc/root_proc through overlay2, which is an error (overlay2 doesn't support /proc as lower): -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- 2020-01-14 11:06:53.410 DEBUG PID: 5239 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has finished: ['mount', '-t', 'overlay', 'overlay2', '-o', 'lowerdir=/proc,upperdir=/var/lib/leapp/scratch/mounts/root_proc/upper,workdir=/var/lib/leapp/scratch/mounts/root_proc/work', '/var/lib/leapp/scratch/mounts/root_proc/root_proc'] -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- The reason for that is when the system got upgraded from RHEL6, the following entries (only /proc is the culprit here) are still found in /etc/fstab: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Version-Release number of selected component (if applicable): leapp-repository-0.9.0-4.el7.noarch How reproducible: Always Steps to Reproduce: 1. Add entry for "/proc" in /etc/fstab proc /proc proc defaults 0 0 2. Reboot 3. Execute "leapp preupgrade" Actual results: Crash Expected results: No crash Additional info: Christophe Besson proposes the following quick patch: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- --- /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/libraries/overlaygen.py.orig 2020-01-14 11:30:29.926393785 +0100 +++ /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/libraries/overlaygen.py 2020-01-14 11:31:38.804393785 +0100 @@ -65,7 +65,7 @@ current_mount = mounts.pop(current) name = _mount_name(current) # We will not make an overlay over /boot/efi as it is vfat that does not support overlayfs - with _overlay_if(current != '/boot/efi', name=name, source=current, workdir=current_mount.target) as overlay: + with _overlay_if(current not in ('/boot/efi', '/proc', '/sys'), name=name, source=current, workdir=current_mount.target) as overlay: with mounting.BindMount(source=overlay.target, target=os.path.join(root_mount.target, current.lstrip('/'))): with _build_overlay_mount(root_mount, mounts) as mount: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
Thank Renaud for the report. I will create ticket in our JIRA board for this to ensure we will not miss that in upcoming plans.
My initial "quick patch" suggestion is maybe not enough to prevent a bad behavior. Maybe we could have other filesystem in /etc/fstab which are not supported as an overlay lower layer. Using statfs() from C might help to figure out the filesystem type properly. Here is a try, allowing only XFS and EXT fs. import ctypes import ctypes.util # Some constants for filesystem magic EXT2_SUPER_MAGIC = 0xEF53 EXT3_SUPER_MAGIC = 0xEF53 XFS_SUPER_MAGIC = 0x58465342 PROC_SUPER_MAGIC = 0x9fa0 SYSFS_SUPER_MAGIC = 0x62656572 TMPFS_MAGIC = 0x01021994 # Wrap statfs_t struct class statfs_t(ctypes.Structure): _fields_ = [ ("f_type", ctypes.c_long), # type of file system (see below) ("f_bsize", ctypes.c_long), # optimal transfer block size ("f_blocks", ctypes.c_long), # total data blocks in file system ("f_bfree", ctypes.c_long), # free blocks in fs ("f_bavail", ctypes.c_long), # free blocks avail to non-superuser ("f_files", ctypes.c_long), # total file nodes in file system ("f_ffree", ctypes.c_long), # free file nodes in fs ("f_fsid", ctypes.c_int*2), # file system id ("f_namelen", ctypes.c_long), # maximum length of filenames # statfs_t has a bunch of extra padding, we hopefully guess large enough. ("padding", ctypes.c_char*1024), ] # Wrap statfs() libc = ctypes.CDLL(ctypes.util.find_library("c"), use_errno=True) _statfs = libc.statfs _statfs.argtypes = [ctypes.c_char_p, ctypes.POINTER(statfs_t)] _statfs.rettype = ctypes.c_int def statfs(path): buf = statfs_t() err = _statfs(path, ctypes.byref(buf)) if err == -1: errno = ctypes.get_errno() raise OSError(errno, '%s path: %r' % (os.strerror(errno), path)) return buf def main(): import sys if len(sys.argv) > 1: path = sys.argv[1] else: path = sys.argv[0] if statfs(path).f_type in (XFS_SUPER_MAGIC, EXT3_SUPER_MAGIC): sys.exit(0) # OK else: sys.exit(1) # NO if __name__ == "__main__": main()
@Christopher, we have more solutions around. Currently, we know only about XFS without ftype that are not supported. But if you tell us what other FSs have this issue, we can resolve it using the same workround. - currently we are creating for such FSs ext4 fs inside file, which we use lower layer.
In this little piece of code, I've just suggested a check of the fs type to ensure it is supported. At least vfat, procfs and sysfs are not supported as an overlay lower layer. I don't know for the others...
Hi Christophe, sorry for late response. Thanks for information. I meant previously that we already know a way, how to workround cases when some FSs are not supported as lower layer. Well, in case of procfs, we will just want to skip it (maybe same for sysfs). Btw, thanks for the piece of the code. It seems like better way to obtain required information about FSs in python than we are using now.
(In reply to Petr Stodulka from comment #7) > sorry for late response. Thanks for information. I meant previously that we > already know a way, how to workround cases when some FSs are not supported > as lower layer. Well, in case of procfs, we will just want to skip it (maybe > same for sysfs). sysfs should be also skipped, access to files in: /var/lib/leapp/scratch/mounts/root_/system_overlay/sys also triggers kernel panic during upgrade
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:1959