Bug 1790861

Summary: Crash in overlayfs when upgrading a system from 7.6 to 8.1 using leapp
Product: Red Hat Enterprise Linux 7 Reporter: Renaud Métrich <rmetrich>
Component: leapp-repositoryAssignee: Leapp team <leapp-notifications>
Status: CLOSED ERRATA QA Contact: Alois Mahdal <amahdal>
Severity: high Docs Contact:
Priority: high    
Version: 7.6CC: amarecek, cbesson, ghalat, leapp-notifications, mbocek, mreznik, pstodulk
Target Milestone: rcKeywords: Upgrades
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: leapp-repository-0.10.0-2.el7_8 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-04-29 01:45:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Renaud Métrich 2020-01-14 12:07:24 UTC
Description of problem:

When upgrading a 7.6 system that was previously upgraded from RHEL 6, a crash happens in overlayfs (tracked by BZ #1789473).
Internally, leapp mounts /proc onto /var/lib/leapp/scratch/mounts/root_proc/root_proc through overlay2, which is an error (overlay2 doesn't support /proc as lower):

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
2020-01-14 11:06:53.410 DEBUG    PID: 5239 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has finished: ['mount', '-t', 'overlay', 'overlay2', '-o', 'lowerdir=/proc,upperdir=/var/lib/leapp/scratch/mounts/root_proc/upper,workdir=/var/lib/leapp/scratch/mounts/root_proc/work', '/var/lib/leapp/scratch/mounts/root_proc/root_proc']
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

The reason for that is when the system got upgraded from RHEL6, the following entries (only /proc is the culprit here) are still found in /etc/fstab:

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------


Version-Release number of selected component (if applicable):

leapp-repository-0.9.0-4.el7.noarch


How reproducible:

Always


Steps to Reproduce:
1. Add entry for "/proc" in /etc/fstab

proc                    /proc                   proc    defaults        0 0


2. Reboot

3. Execute "leapp preupgrade"

Actual results:

Crash


Expected results:

No crash


Additional info:

Christophe Besson proposes the following quick patch:

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
--- /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/libraries/overlaygen.py.orig	2020-01-14 11:30:29.926393785 +0100
+++ /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/libraries/overlaygen.py	2020-01-14 11:31:38.804393785 +0100
@@ -65,7 +65,7 @@
         current_mount = mounts.pop(current)
         name = _mount_name(current)
         # We will not make an overlay over /boot/efi as it is vfat that does not support overlayfs
-        with _overlay_if(current != '/boot/efi', name=name, source=current, workdir=current_mount.target) as overlay:
+        with _overlay_if(current not in ('/boot/efi', '/proc', '/sys'), name=name, source=current, workdir=current_mount.target) as overlay:
             with mounting.BindMount(source=overlay.target,
                                     target=os.path.join(root_mount.target, current.lstrip('/'))):
                 with _build_overlay_mount(root_mount, mounts) as mount:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Comment 2 Petr Stodulka 2020-01-14 12:37:18 UTC
Thank Renaud for the report. I will create ticket in our JIRA board for this to ensure we will not miss that in upcoming plans.

Comment 4 Christophe Besson 2020-01-15 14:39:23 UTC
My initial "quick patch" suggestion is maybe not enough to prevent a bad behavior. Maybe we could have other filesystem in /etc/fstab which are not supported as an overlay lower layer.

Using statfs() from C might help to figure out the filesystem type properly. Here is a try, allowing only XFS and EXT fs.

import ctypes
import ctypes.util

# Some constants for filesystem magic
EXT2_SUPER_MAGIC       = 0xEF53
EXT3_SUPER_MAGIC       = 0xEF53
XFS_SUPER_MAGIC        = 0x58465342
PROC_SUPER_MAGIC       = 0x9fa0
SYSFS_SUPER_MAGIC      = 0x62656572
TMPFS_MAGIC            = 0x01021994

# Wrap statfs_t struct
class statfs_t(ctypes.Structure):
  _fields_ = [
	("f_type",    ctypes.c_long),  # type of file system (see below)
	("f_bsize",   ctypes.c_long),  # optimal transfer block size
	("f_blocks",  ctypes.c_long),  # total data blocks in file system
	("f_bfree",   ctypes.c_long),  # free blocks in fs
	("f_bavail",  ctypes.c_long),  # free blocks avail to non-superuser
	("f_files",   ctypes.c_long),  # total file nodes in file system
	("f_ffree",   ctypes.c_long),  # free file nodes in fs
	("f_fsid",    ctypes.c_int*2), # file system id
	("f_namelen", ctypes.c_long),  # maximum length of filenames
	# statfs_t has a bunch of extra padding, we hopefully guess large enough.
	("padding",   ctypes.c_char*1024),
	]

# Wrap statfs()
libc = ctypes.CDLL(ctypes.util.find_library("c"), use_errno=True)
_statfs = libc.statfs
_statfs.argtypes = [ctypes.c_char_p, ctypes.POINTER(statfs_t)]
_statfs.rettype = ctypes.c_int
def statfs(path):
	buf = statfs_t()
	err = _statfs(path, ctypes.byref(buf))
	if err == -1:
		errno = ctypes.get_errno()
		raise OSError(errno, '%s path: %r' % (os.strerror(errno), path))
	return buf

def main():
	import sys
	if len(sys.argv) > 1:
		path = sys.argv[1]
	else:
		path = sys.argv[0]
	if statfs(path).f_type in (XFS_SUPER_MAGIC, EXT3_SUPER_MAGIC):
		sys.exit(0) # OK
	else:
		sys.exit(1) # NO

if __name__ == "__main__":
	main()

Comment 5 Petr Stodulka 2020-01-15 15:42:30 UTC
@Christopher,
we have more solutions around. Currently, we know only about XFS without ftype that are not supported. But if you tell us what other FSs have this issue, we can resolve it using the same workround. - currently we are creating for such FSs ext4 fs inside file, which we use lower layer.

Comment 6 Christophe Besson 2020-01-15 16:07:36 UTC
In this little piece of code, I've just suggested a check of the fs type to ensure it is supported.
At least vfat, procfs and sysfs are not supported as an overlay lower layer. I don't know for the others...

Comment 7 Petr Stodulka 2020-01-23 16:19:56 UTC
Hi Christophe,
sorry for late response. Thanks for information. I meant previously that we already know a way, how to workround cases when some FSs are not supported as lower layer. Well, in case of procfs, we will just want to skip it (maybe same for sysfs).

Btw, thanks for the piece of the code. It seems like better way to obtain required information about FSs in python than we are using now.

Comment 8 Grzegorz Halat 2020-01-23 16:48:33 UTC
(In reply to Petr Stodulka from comment #7)
> sorry for late response. Thanks for information. I meant previously that we
> already know a way, how to workround cases when some FSs are not supported
> as lower layer. Well, in case of procfs, we will just want to skip it (maybe
> same for sysfs).

sysfs should be also skipped, access to files in:
/var/lib/leapp/scratch/mounts/root_/system_overlay/sys
also triggers kernel panic during upgrade

Comment 20 errata-xmlrpc 2020-04-29 01:45:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1959