RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1399487 - grub2-install failed in rear recover with /boot partition formatted with XFS
Summary: grub2-install failed in rear recover with /boot partition formatted with XFS
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: rear
Version: 7.3
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: rc
: ---
Assignee: Jakub Mazanek
QA Contact: Tereza Cerna
URL:
Whiteboard:
Depends On:
Blocks: 1402716 1404167
TreeView+ depends on / blocked
 
Reported: 2016-11-29 07:24 UTC by Masahiro Matsuya
Modified: 2021-12-10 14:48 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Rear was setting the file system UUID twice Consequence: Filesystem metadata ended up inconsistent. grub2-install command failed. Fix: Adjusted rear to set the UUID only once and correctly Result: The grub2 installation works as expected.
Clone Of:
: 1402716 1404167 (view as bug list)
Environment:
Last Closed: 2017-08-01 22:19:55 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:2052 0 normal SHIPPED_LIVE rear bug fix update 2017-08-01 19:33:49 UTC

Description Masahiro Matsuya 2016-11-29 07:24:22 UTC
Description of problem:

rear recover doesn't work when XFS is used for /boot from RHEL7.3.
grub2-install failed while 'rear recover' was executed in rescue.

  grub2-install: error: unknown filesystem.

grub2-probe --device <device file for /boot>
also shows the same error.

The error means that it could not detect the filesystem type of /boot partition.
/boot partition should be formatted with xfs to reproduce it.

From RHEL7.3, CRC feature was enabled for mkfs.xfs. It can be confirmed with the output of mkfs.xfs and xfs_info command.

# mkfs.xfs -f /dev/sdb1
meta-data=/dev/sdb1              isize=512    agcount=4, agsize=655296 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
... snip ...

# mount /dev/sdb1 /mnt
# xfs_info /mnt
meta-data=/dev/sdb1              isize=512    agcount=4, agsize=655296 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0 spinodes=0
... snip ...

In RHEL7.2, it outputs crc=0.

The manpage of mkfs.xfs was changed as below, so the change looks by intent.

  In RHEL7.2:
     By default, mkfs.xfs will not enable metadata CRCs.
  In RHEL7.3:
     By default, mkfs.xfs will enable metadata CRCs.

Also, the following statements are in the manpage of xfs_admin.

----------------------------------
       -U uuid
              Set  the UUID of the filesystem to uuid.  A sample UUID looks like this: "c1b9d5a2-f162-11cf-9ece-0020afc76f16".  The
              uuid may also be nil, which will set the filesystem UUID to the null UUID.  The uuid may also be generate, which will
              generate  a  new  UUID  for the filesystem.  Note that on CRC-enabled filesystems, this will set an incompatible flag
              such that older kernels will not be able to mount the filesystem.  To remove this  incompatible  flag,  use  restore,
              which will restore the original UUID and remove the incompatible feature flag as needed.
-----------------------------------

That says that an incompatible flag is set on CRC enabled.
And, the incompatible flag is checked in grub2-install with the following code, and it fails to detect XFS format.

In grub-core/fs/xfs.c:
static int grub_xfs_sb_valid(struct grub_xfs_data *data)
{
  grub_dprintf("xfs", "Validating superblock\n");
... snip ...
  if ((data->sblock.version & grub_cpu_to_be16_compile_time(XFS_SB_VERSION_NUMBITS)) ==
       grub_cpu_to_be16_compile_time(XFS_SB_VERSION_5))
    {
      grub_dprintf("xfs", "XFS v5 superblock detected\n");
      if (data->sblock.sb_features_incompat &
          grub_cpu_to_be32_compile_time(~XFS_SB_FEAT_INCOMPAT_SUPPORTED))
        {
          grub_error (GRUB_ERR_BAD_FS, "XFS filesystem has unsupported "
                      "incompatible features");
          return 0;
        }
      return 1;
    }

To sum up:
  - CRC feature of XFS was enabled by default from RHEL7.3
  - When CRC feature is enabled, xfs_admin -U command set an incompatible flag.
  - grub2-install fails to detect the filesystem as a valid and supporte XFS format.

In 'rear recover', the following commands are executed to format the /boot partition
  mkfs.xfs -f /dev/sda1
  xfs_admin -U <uuid> /dev/sda1

As a result, an incompatible flag is set in xfs_admin -U command.

If -m uuid=<uuid> option is used with mkfs.xfs, it seems that grub2-install doesn't fail.

XFS is the default filesystem from RHEL7. So, this is a serious regression.
This problem doesn't happen with RHEL7.2, since CRC feature is disabled by default.

Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux 7.3

How reproducible:
Always


Steps to Reproduce:
1. Install RHEL7.3 (XFS for /boot)
2. configure local.conf, and run rear mkbackup
3. boot with rescue ISO (or ramdisk) and execute recovery.

Actual results:
The rear recover fails with grub2-install error.

Expected results:
The rear recover doesn't fail.

Addtional info:
I was not sure which component I should select for this bug, xfsprogs, grub2, and rear. The release note of 7.3 includes the following:
---------------------------
Metadata cyclic redundancy checks (CRCs) and directory entry file types are now enabled by default. To replicate the older mkfs on-disk format used in earlier versions of Red Hat Enterprise Linux 7, use the -m crc=0 -n ftype=0 options on the mkfs.xfs command line.
---------------------------
So, the CRC feature was enabled by intent, and I thought it's not a bug of xfsprogs, at least.

Comment 3 Masahiro Matsuya 2016-11-29 09:11:56 UTC
libxfs/xfs_format.h of xfsprogs package
--------------------------------------------
#define XFS_SB_FEAT_INCOMPAT_FTYPE      (1 << 0)        /* filetype in dirent */
#define XFS_SB_FEAT_INCOMPAT_SPINODES   (1 << 1)        /* sparse inode chunks */
#define XFS_SB_FEAT_INCOMPAT_META_UUID  (1 << 2)        /* metadata UUID */
--------------------------------------------

xfs_admin -U command causes to set the XFS_SB_FEAT_INCOMPAT_META_UUID flag.
XFS_SB_FEAT_INCOMPAT_FTYPE flag only is regarded as supported by grub2-install.

grub-core/fs/xfs.c of grub2 package
--------------------------------------------
#define XFS_SB_FEAT_INCOMPAT_FTYPE      (1 << 0)        /* filetype in dirent */
#define XFS_SB_FEAT_INCOMPAT_SUPPORTED \
        (XFS_SB_FEAT_INCOMPAT_FTYPE)
--------------------------------------------

In upstream grub2, XFS_SB_FEAT_INCOMPAT_META_UUID is included in XFS_SB_FEAT_INCOMPAT_SUPPORTED.
http://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/fs/xfs.c

Changing into grub2...

Comment 7 Masahiro Matsuya 2016-12-01 01:00:46 UTC
This is an easier way to reproduce the problem without rear.

1. install RHEL7.3 with /boot partition fomatted with XFS.
2. Assuming that /boot partition is sda1, the following grub2-probe command should be able to work. (Note: xfs_admin -U command is not executed during installation)

   # grub2-probe --device /dev/sda1
   xfs

3. umount /dev/sda1
4. set an UUID with xfs_admin -U
 
   $ uuidgen 
   78cb73e1-a8b6-4471-a7fc-037ce6191b9e
   # xfs_admin -U 78cb73e1-a8b6-4471-a7fc-037ce6191b9e /dev/sda1

5. modify UUID= for /boot in /etc/fstab
6. mount /dev/sda1
7. the same grub2-probe command fails as below.

   # grub2-probe --device /dev/sda1
   grub2-probe: error: unknown filesystem.

'grub2-install /dev/sda1' fails as well.

   # grub2-install /dev/sda1
   Installing for i386-pc platform.
   grub2-install: error: unknown filesystem.

Comment 18 Gratien D'haese 2017-01-19 10:32:51 UTC
Could you check if this is related to https://github.com/rear/rear/pull/1067 ?

Comment 19 Jakub Mazanek 2017-02-23 15:11:42 UTC
Sorry for long time no update

fix has already been provided as a hotfix.
It was tested by customer and functional.

I will be closing this bugzilla

@gdha

Yes, it was related issue

Comment 20 Gratien D'haese 2017-03-01 11:19:01 UTC
That has been fixed by https://github.com/rear/rear/pull/1067/commits/bc09f447b18ede0218677d68dd7eb06c28152b13 and was released with ReaR v2.00

Comment 24 errata-xmlrpc 2017-08-01 22:19:55 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2052


Note You need to log in before you can comment on or make changes to this bug.