RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1790861 - Crash in overlayfs when upgrading a system from 7.6 to 8.1 using leapp
Summary: Crash in overlayfs when upgrading a system from 7.6 to 8.1 using leapp
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: leapp-repository
Version: 7.6
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Leapp team
QA Contact: Alois Mahdal
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-01-14 12:07 UTC by Renaud Métrich
Modified: 2023-03-24 16:42 UTC (History)
7 users (show)

Fixed In Version: leapp-repository-0.10.0-2.el7_8
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-04-29 01:45:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 4733241 0 None None None 2020-01-14 12:14:06 UTC
Red Hat Product Errata RHBA-2020:1959 0 None None None 2020-04-29 01:46:15 UTC

Description Renaud Métrich 2020-01-14 12:07:24 UTC
Description of problem:

When upgrading a 7.6 system that was previously upgraded from RHEL 6, a crash happens in overlayfs (tracked by BZ #1789473).
Internally, leapp mounts /proc onto /var/lib/leapp/scratch/mounts/root_proc/root_proc through overlay2, which is an error (overlay2 doesn't support /proc as lower):

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
2020-01-14 11:06:53.410 DEBUG    PID: 5239 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: External command has finished: ['mount', '-t', 'overlay', 'overlay2', '-o', 'lowerdir=/proc,upperdir=/var/lib/leapp/scratch/mounts/root_proc/upper,workdir=/var/lib/leapp/scratch/mounts/root_proc/work', '/var/lib/leapp/scratch/mounts/root_proc/root_proc']
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

The reason for that is when the system got upgraded from RHEL6, the following entries (only /proc is the culprit here) are still found in /etc/fstab:

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------


Version-Release number of selected component (if applicable):

leapp-repository-0.9.0-4.el7.noarch


How reproducible:

Always


Steps to Reproduce:
1. Add entry for "/proc" in /etc/fstab

proc                    /proc                   proc    defaults        0 0


2. Reboot

3. Execute "leapp preupgrade"

Actual results:

Crash


Expected results:

No crash


Additional info:

Christophe Besson proposes the following quick patch:

-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
--- /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/libraries/overlaygen.py.orig	2020-01-14 11:30:29.926393785 +0100
+++ /usr/share/leapp-repository/repositories/system_upgrade/el7toel8/libraries/overlaygen.py	2020-01-14 11:31:38.804393785 +0100
@@ -65,7 +65,7 @@
         current_mount = mounts.pop(current)
         name = _mount_name(current)
         # We will not make an overlay over /boot/efi as it is vfat that does not support overlayfs
-        with _overlay_if(current != '/boot/efi', name=name, source=current, workdir=current_mount.target) as overlay:
+        with _overlay_if(current not in ('/boot/efi', '/proc', '/sys'), name=name, source=current, workdir=current_mount.target) as overlay:
             with mounting.BindMount(source=overlay.target,
                                     target=os.path.join(root_mount.target, current.lstrip('/'))):
                 with _build_overlay_mount(root_mount, mounts) as mount:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Comment 2 Petr Stodulka 2020-01-14 12:37:18 UTC
Thank Renaud for the report. I will create ticket in our JIRA board for this to ensure we will not miss that in upcoming plans.

Comment 4 Christophe Besson 2020-01-15 14:39:23 UTC
My initial "quick patch" suggestion is maybe not enough to prevent a bad behavior. Maybe we could have other filesystem in /etc/fstab which are not supported as an overlay lower layer.

Using statfs() from C might help to figure out the filesystem type properly. Here is a try, allowing only XFS and EXT fs.

import ctypes
import ctypes.util

# Some constants for filesystem magic
EXT2_SUPER_MAGIC       = 0xEF53
EXT3_SUPER_MAGIC       = 0xEF53
XFS_SUPER_MAGIC        = 0x58465342
PROC_SUPER_MAGIC       = 0x9fa0
SYSFS_SUPER_MAGIC      = 0x62656572
TMPFS_MAGIC            = 0x01021994

# Wrap statfs_t struct
class statfs_t(ctypes.Structure):
  _fields_ = [
	("f_type",    ctypes.c_long),  # type of file system (see below)
	("f_bsize",   ctypes.c_long),  # optimal transfer block size
	("f_blocks",  ctypes.c_long),  # total data blocks in file system
	("f_bfree",   ctypes.c_long),  # free blocks in fs
	("f_bavail",  ctypes.c_long),  # free blocks avail to non-superuser
	("f_files",   ctypes.c_long),  # total file nodes in file system
	("f_ffree",   ctypes.c_long),  # free file nodes in fs
	("f_fsid",    ctypes.c_int*2), # file system id
	("f_namelen", ctypes.c_long),  # maximum length of filenames
	# statfs_t has a bunch of extra padding, we hopefully guess large enough.
	("padding",   ctypes.c_char*1024),
	]

# Wrap statfs()
libc = ctypes.CDLL(ctypes.util.find_library("c"), use_errno=True)
_statfs = libc.statfs
_statfs.argtypes = [ctypes.c_char_p, ctypes.POINTER(statfs_t)]
_statfs.rettype = ctypes.c_int
def statfs(path):
	buf = statfs_t()
	err = _statfs(path, ctypes.byref(buf))
	if err == -1:
		errno = ctypes.get_errno()
		raise OSError(errno, '%s path: %r' % (os.strerror(errno), path))
	return buf

def main():
	import sys
	if len(sys.argv) > 1:
		path = sys.argv[1]
	else:
		path = sys.argv[0]
	if statfs(path).f_type in (XFS_SUPER_MAGIC, EXT3_SUPER_MAGIC):
		sys.exit(0) # OK
	else:
		sys.exit(1) # NO

if __name__ == "__main__":
	main()

Comment 5 Petr Stodulka 2020-01-15 15:42:30 UTC
@Christopher,
we have more solutions around. Currently, we know only about XFS without ftype that are not supported. But if you tell us what other FSs have this issue, we can resolve it using the same workround. - currently we are creating for such FSs ext4 fs inside file, which we use lower layer.

Comment 6 Christophe Besson 2020-01-15 16:07:36 UTC
In this little piece of code, I've just suggested a check of the fs type to ensure it is supported.
At least vfat, procfs and sysfs are not supported as an overlay lower layer. I don't know for the others...

Comment 7 Petr Stodulka 2020-01-23 16:19:56 UTC
Hi Christophe,
sorry for late response. Thanks for information. I meant previously that we already know a way, how to workround cases when some FSs are not supported as lower layer. Well, in case of procfs, we will just want to skip it (maybe same for sysfs).

Btw, thanks for the piece of the code. It seems like better way to obtain required information about FSs in python than we are using now.

Comment 8 Grzegorz Halat 2020-01-23 16:48:33 UTC
(In reply to Petr Stodulka from comment #7)
> sorry for late response. Thanks for information. I meant previously that we
> already know a way, how to workround cases when some FSs are not supported
> as lower layer. Well, in case of procfs, we will just want to skip it (maybe
> same for sysfs).

sysfs should be also skipped, access to files in:
/var/lib/leapp/scratch/mounts/root_/system_overlay/sys
also triggers kernel panic during upgrade

Comment 20 errata-xmlrpc 2020-04-29 01:45:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1959


Note You need to log in before you can comment on or make changes to this bug.