Eric, is this related to bug 1937973?  If so, I thought e.g. RHEL 8 would have backported code to allow this to mount?

Yes, it is related to the new features enabled by default in bug 1937973. One of those features is for Y2038 compatibility.

As far as I know, there has never been a requirement that older RHEL relesases must support all disk formats present in newer RHEL releases. For example, when we added metadata CRC support to XFS in RHEL8 by default, we did not backport that support to RHEL7.

That said, the two features added by default for RHEL9 are less invasive, and most likely could be enabled in RHEL8.  TBH the userspace component would be the most work. We'd need to plan for it, but it's probably not too late to have it by RHEL8.6 if it /is/ considered a requirement.

Another option could be that for OS images we expect to be hosted on on RHEL releases, we *could* turn those features back off with the associated lack of functionality (primarily Y2038 support.)

I have just compared the xfs_info output of the latest 9-stream image with the latest rhel-9 compose:

CentOS-Stream-GenericCloud-9-20211014.0.x86_64.qcow2
meta-data=/dev/nbd0p1            isize=512    agcount=4, agsize=512000 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=1
data     =                       bsize=4096   blocks=2048000, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

rhel-guest-image-9.0-20211013.10.x86_64.qcow2
meta-data=/dev/nbd1p3            isize=512    agcount=4, agsize=648829 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=0
data     =                       bsize=4096   blocks=2595315, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Given that rhel-9 is apparently not enabling bigtime, and the support lifecycle of 9-stream will end well short of 2038, either of these could justify switching off bigtime in 9-stream built images.

Just for anyone's reference who finds this; a 5.10 kernel is sufficient to mount such a partition.  I think that you can mount it on centos 8-stream (but not 8) and we have now switched our builder system to Debian Bullseye which can also mount it.

Still, it is probably good for the image to be mountable on the previous major release, IMO

If it's not on in RHEL9 at this point, that's an oversight.  It's our current plan to enable this, at least by RHEL9 GA.

However:

# cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.0 Beta (Plow)
# rpm -qf /etc/redhat-release 
redhat-release-9.0-2.8.el9.x86_64
# rpm -q xfsprogs
xfsprogs-5.12.0-4.el9.x86_64
# xfs_info / | grep bigtime
         =                       reflink=1    bigtime=1 inobtcount=1

It appears to be turned on in RHEL9 beta.

FWIW, Y2038 timestamps may matter before the year 2038; it's uncommon, but there are cases where times can be set ahead. And boxes don't burn to the ground when the support lifecycle ends. It felt prudent to get ahead of it.

If it's a hard requirement to have RHEL9 filesystems mountable on older RHELs, we need to know that ASAP so we can make plans, either to backport the features to RHEL8 (RHEL8.6 at best, it will not happen for RHEL7), or to disable it by default in RHEL9 and give the customer clear instructions for feature upgrade under RHEL9 when needed...

Assigned to Rick for initial triage per bz process and age of bug created or assigned to virt-maint without triage.

Eric, based on the previous comments, this is not an issue specific to rhel-guest-image. From what I understand, the changes from bug 1937973 has implications across a major release that need to be discussed and worked out by your team.

Are you OK with taking this BZ to move it forward?

I've just hit this issue as well. I'm building CS9 images in our CS8 Koji builders and I need to be able to extract the root fs UUID so I can tell Openstack Ironic where to find it.

I tried to work around this issue by producing a CS9 image with bigtime=0, but now I'm just hitting another problem:

---
> LIBGUESTFS_BACKEND=direct guestfish --ro -a cs9-cloud-20211119-8.x86_64.raw -i cat /etc/fstab
libguestfs: error: inspect_os: lstat: /grub/menu.lst: Input/output error
> LIBGUESTFS_BACKEND=direct guestfish --ro -a cs9-cloud-20211119-8.x86_64.raw

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: ‘help’ for help on commands
      ‘man’ to read the manual
      ‘quit’ to quit the shell

><fs> trace 1
><fs> run
libguestfs: trace: launch
libguestfs: trace: max_disks
libguestfs: trace: max_disks = 255
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: trace: get_cachedir
libguestfs: trace: get_cachedir = "/var/tmp"
libguestfs: trace: get_backend_setting "force_tcg"
libguestfs: trace: get_backend_setting = NULL (error)
libguestfs: trace: get_sockdir
libguestfs: trace: get_sockdir = "/run/user/17954"
libguestfs: trace: get_backend_setting "gdb"
libguestfs: trace: get_backend_setting = NULL (error)
libguestfs: trace: launch = 0
><fs> list-filesystems 
libguestfs: trace: list_filesystems
libguestfs: trace: list_filesystems = ["/dev/sda1", "xfs", "/dev/sda14", "unknown", "/dev/sda15", "vfat"]
/dev/sda1: xfs
/dev/sda14: unknown
/dev/sda15: vfat
><fs> mount /dev/sda1 /
libguestfs: trace: mount "/dev/sda1" "/"
libguestfs: trace: vfs_type "/dev/sda1"
libguestfs: trace: vfs_type = "xfs"
libguestfs: trace: mount = -1 (error)
libguestfs: error: mount: mount exited with status 32: mount: /sysroot: wrong fs type, bad option, bad superblock on /dev/sda1, missing codepage or helper program, or other error.
><fs>
---

What's the other new xfs feature enabled by default that I need to disable to make this work? (I don't have access to bug 1937973)

inobtcount is the other feature that is involved here

(In reply to Rick Barry from comment #7)
> Eric, based on the previous comments, this is not an issue specific to
> rhel-guest-image. From what I understand, the changes from bug 1937973 has
> implications across a major release that need to be discussed and worked out
> by your team.
> 
> Are you OK with taking this BZ to move it forward?

there's nothing for me to do here as far as I can tell, what you have encountered is a new on-disk feature on by default in rhel9 which is not currently supported in RHEL8. That's not a bug, though - it's an intentional and necessary change, with the side-effect of incompatibilities with older kernels.

However, I have opened bug 2022903 and bug 2024201 to enable these features in RHEL8, should land in RHEL8.6.

Brian - you are correct. mkfs.xfs -m bigtime=0,inobtcount=0 should disable both new features and make a rhel8 compatible filesystem.

> (In reply to Rick Barry from comment #7)
> > Eric, based on the previous comments, this is not an issue specific to
> > rhel-guest-image. From what I understand, the changes from bug 1937973 has
> > implications across a major release that need to be discussed and worked out
> > by your team.
> > 
> > Are you OK with taking this BZ to move it forward?
> 
> there's nothing for me to do here as far as I can tell, what you have
> encountered is a new on-disk feature on by default in rhel9 which is not
> currently supported in RHEL8. That's not a bug, though - it's an intentional
> and necessary change, with the side-effect of incompatibilities with older
> kernels.
> 
> However, I have opened bug 2022903 and bug 2024201 to enable these features
> in RHEL8, should land in RHEL8.6.

Thanks, Eric. If those features are available in RHEL 8.6, it seems that would resolve the issue for RHEL, but not for the other use cases mentioned in the BZ description.

I'm including Troy Dawson and Tom Gundersen here. (As of RHEL 8.4 and 9.0, Image builder is used to build the rhel-guest-image, but this is not the case for CentOS Stream 9. I'm not sure if a rhel-guest-image that's built by Image Builder also exhibits the same behavior.)

Troy/Tom, could you review this BZ and see if you think there's anything that should be done from an image build perspective? Or is having the features back-ported to RHEL 8.6 as currently planned the best solution?

Having those features backported to RHEL 8.6 is the ideal case here, if it's feasible.

(Adding Wei Shi from Virt QE)

We have updated the kickstart files for the CentOS Stream 9 kvm and ec2 images to use -m bigtime=0,inobtcount=0
At this time the composes are having some issues, so we don't have anything for you to test yet, but we are working on it.

Thanks. If the built images differ from a default RHEL9 install, I think that would be something that is worth clearly documenting - though I'm not exactly sure where to do so.  I understand the need to alter the install, but differing capabilities depending on the OS source may be confusing, and documentation might help.

The most user-visible difference will be an inability to read or write timestamps beyond 2038.

-Eric

Verified with CentOS-Stream-GenericCloud-9-20220302.0.x86_64.qcow2