After some more investigation it looks like the immutability is lost somewhere between the ignition-ostree-mount-sysroot and the ignition-ostree-mount-var phase . I just printed the output of "lsattr -d /sysroot" in ignition-ostree-mount-sysroot.sh and it is immutable (has the i flag set). I put a print in ignition-ostree-mount-var.sh and the immutability was not present. There is a OSTree Prepare phase in between, but I'm not sure if that could've caused it to be removed.

yup just did a quick test by putting an lsattr in ostree-prepare-root.service in the ExecStartPre and ExecStartPost section and looks like sysroot is mutable after ostree prepare executes

[   14.053634] sh[832]: ----i--------------- /sysroot
[   14.073740] ostree-prepare-root[834]: Resolved OSTree target to: /sysroot/ostree/deploy/rhcos/deploy/aad0d687b4542facdb60c44fe311fe5e9dfe7ef28c787726ccc120768b18c8e1.0
[   14.088788] sh[836]: -------------------- /sysroot


Need to investigate what ostree prepare does.

To clarify, this is *only* s390x, not ppc64le/aarch64 right?

(In reply to Colin Walters from comment #5)
> To clarify, this is *only* s390x, not ppc64le/aarch64 right?

Correct. This is an issue on s390x only.

> yup just did a quick test by putting an lsattr in ostree-prepare-root.service in the ExecStartPre and ExecStartPost section and looks like sysroot is mutable after ostree prepare executes

That is...very weird.  What does this print:

mkdir -p /var/tmp/testdir-{src,target}
chattr +i /var/tmp/testdir-src
mount --bind /var/tmp/testdir-{src,target}
lsattr -d /var/tmp/testdir-target

(In reply to Colin Walters from comment #7)
> > yup just did a quick test by putting an lsattr in ostree-prepare-root.service in the ExecStartPre and ExecStartPost section and looks like sysroot is mutable after ostree prepare executes
> 
> That is...very weird.  What does this print:
> 
> mkdir -p /var/tmp/testdir-{src,target}
> chattr +i /var/tmp/testdir-src
> mount --bind /var/tmp/testdir-{src,target}
> lsattr -d /var/tmp/testdir-target

[root@cosa-devsh core]# mkdir -p /var/tmp/testdir-{src,target}
[root@cosa-devsh core]# chattr +i /var/tmp/testdir-src
[root@cosa-devsh core]# mount --bind /var/tmp/testdir-{src,target}
[root@cosa-devsh core]# lsattr -d /var/tmp/testdir-target
----i--------------- /var/tmp/testdir-target

Ok. After some more printf and syscalls looks like the mount sytemcall which is used when the root mount point is moved is causing the immutability bit to be unset.

https://github.com/ostreedev/ostree/blob/master/src/switchroot/ostree-prepare-root.c#L364

Tried the same test on other arches and the immutability bit is preserved. It might be an issue with the mount system call on s390x.

still going down this rabbit hole..looks like the deploy root , i.e the /sysroot/ostree/deploy/rhcos/deploy/... directory does not have the immutability set and this is being mounted at sysroot.

Is the immutable bit there on upgrade?

If you use e.g. libguestfs to inspect the image, do you see it there to start?

Hmm I wonder if this is an issue with `_ostree_linuxfs_fd_alter_immutable_flag()`.
I see that e2fsprogs is passing as an int:
https://github.com/tytso/e2fsprogs/blob/ae3a1d571beb1b86cedfee6652a327854499da3f/lib/e2p/setflags.c#L62

Ah yes and that might explain it, on LE architectures that's going to be fine, but a BE arch would see zeros.

IOW does this fix it?

diff --git a/src/libostree/ostree-linuxfsutil.c b/src/libostree/ostree-linuxfsutil.c
index 231ecf76..cd1f485c 100644
--- a/src/libostree/ostree-linuxfsutil.c
+++ b/src/libostree/ostree-linuxfsutil.c
@@ -55,7 +55,7 @@ _ostree_linuxfs_fd_alter_immutable_flag (int            fd,
   if (g_atomic_int_get (&no_alter_immutable))
     return TRUE;
 
-  unsigned long flags;
+  int flags;
   int r = ioctl (fd, EXT2_IOC_GETFLAGS, &flags);
   if (r == -1)
     {

Added a test for this https://github.com/coreos/fedora-coreos-config/pull/573

(In reply to Colin Walters from comment #12)
> Is the immutable bit there on upgrade?
> 
> If you use e.g. libguestfs to inspect the image, do you see it there to
> start?
> 
> Hmm I wonder if this is an issue with
> `_ostree_linuxfs_fd_alter_immutable_flag()`.
> I see that e2fsprogs is passing as an int:
> https://github.com/tytso/e2fsprogs/blob/
> ae3a1d571beb1b86cedfee6652a327854499da3f/lib/e2p/setflags.c#L62
> 
> Ah yes and that might explain it, on LE architectures that's going to be
> fine, but a BE arch would see zeros.
> 
> IOW does this fix it?
> 
> diff --git a/src/libostree/ostree-linuxfsutil.c
> b/src/libostree/ostree-linuxfsutil.c
> index 231ecf76..cd1f485c 100644
> --- a/src/libostree/ostree-linuxfsutil.c
> +++ b/src/libostree/ostree-linuxfsutil.c
> @@ -55,7 +55,7 @@ _ostree_linuxfs_fd_alter_immutable_flag (int            fd,
>    if (g_atomic_int_get (&no_alter_immutable))
>      return TRUE;
>  
> -  unsigned long flags;
> +  int flags;
>    int r = ioctl (fd, EXT2_IOC_GETFLAGS, &flags);
>    if (r == -1)
>      {
The immutable bit was not there to begin with. I put an lsattr in create_disk.sh right after ostree deploy and it wasn't there. I will test the fix.Thanks!

https://github.com/ostreedev/ostree/pull/2179

(In reply to Colin Walters from comment #15)
> https://github.com/ostreedev/ostree/pull/2179

the fix works. Thanks for looking into it !

@Pranshanth would you be able to confirm this BZ is fixed and mark it VERIFIED?

I have a new cluster up and can confirm:

[root@amccrae-ocp-h4zk8-master-0 ~]# lsattr -d /sysroot
----i--------------- /sysroot
[root@amccrae-ocp-h4zk8-master-0 ~]# uname -r
4.18.0-193.23.1.el8_2.s390x

Looks good!

FYI..the fix is verified but I can't change state to "VERIFIED"..as there seems to be no "VERIFIED" option...am i missing something...should i choose something else

Hmm, me neither. I suppose only users in a "QE" group see this state, so someone from QE must set the VERIFIED state.

Marking verified per comment #22

Thanks everyone!

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (ostree bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2020:4706