Bug 1771257

A shorter reproducer is to pick up the qcow2 image used by the installer from ~/.cache/openshift-install/libvirt/image/ , unzip it, and then run:
$ TMPDIR=$(pwd) virt-sparsify ./rhcos-43.81.201911081536.0-qemu.x86_64.qcow2 sparse.qcow2

The filesystem on /dev/sda4 is XFS and mounts fine (there are no kernel
messages indicating any problem) but I simply cannot write to it even as root.
All writes return -EPERM but there's no other indication of what is wrong.

$ virt-rescue --ro -a rhcos-43.81.201911081536.0-qemu.x86_64.qcow2

><rescue> file -bsL /dev/sda4
SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)
><rescue> mount /dev/sda4 /sysroot/
><rescue> dmesg | tail
[    0.413665] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    0.413709] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    0.414962] sd 2:0:1:0: [sdb] Attached SCSI disk
[    0.415296]  sda: sda1 sda2 sda3 sda4
[    0.415732] sd 2:0:0:0: [sda] Attached SCSI disk
[    0.421296] EXT4-fs (sdb): mounting ext2 file system using the ext4 subsystem
[    0.423023] EXT4-fs (sdb): mounted filesystem without journal. Opts: 
[   22.706196] SGI XFS with ACLs, security attributes, scrub, no debug enabled
[   22.709669] XFS (sda4): Mounting V5 Filesystem
[   22.715660] XFS (sda4): Ending clean mount

><rescue> ls -la /sysroot
total 8
drwxr-xr-x.  4 root root   66 Nov  8 15:43 .
drwxr-xr-x  19 root root 4096 Nov 12 16:02 ..
-rw-rw-r--.  1 root root  257 Nov  8 15:43 .coreos-aleph-version.json
drwxrwxr-x.  2 root root    6 Nov  8 15:42 boot
drwxrwxr-x.  5 root root   62 Nov  8 15:43 ostree

><rescue> touch /sysroot/hello
touch: setting times of '/sysroot/hello': No such file or directory
><rescue> strace touch /sysroot/hello
execve("/bin/touch", ["touch", "/sysroot/hello"], 0x7ffe6aacbb68 /* 13 vars */) = 0
[...]
openat(AT_FDCWD, "/sysroot/hello", O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK, 0666) = -1 EPERM (Operation not permitted)
utimensat(AT_FDCWD, "/sysroot/hello", NULL, 0) = -1 ENOENT (No such file or directory)

It turns out (thanks Jeff Moyer) that the root directory of that
filesystem is marked as immutable:

><rescue> lsattr -a /sysroot
----i--------------- /sysroot/.

which is what causes the problem.  Because of the way that virt-sparsify
works in copying mode it needs to temporarily write a balloon file to
each filesystem, so an immutable filesystem isn't going to work.

As a workaround for this particular disk you can use the --ignore option, eg:

  virt-sparsify --ignore /dev/sda4 [...]

You can also use in-place sparsification instead, which is much faster.  This
ignores the immutable bit and I've checked that it works on this particular
filesystem.

@Richard Thank you for quick look and providing a workaround. Before we were using `in-place` option but that doesn't work when we specify `-o lazy_refcounts=on` option and that is something we really required to make this qcow2 image work with mac using hyperkit. We will try to use the --ignore option meanwhile and see if resulted image is work for our case. I still not sure why this change is required for Redhat CoreOS side, may be @Micah or @Steve can answer.

Another thing we were discussing is temporarily removing the immutable attribute
around the virt-sparsify command, ie:

$ guestfish -a source.qcow2 -m /dev/sda4 set-e2attrs / i clear:true
$ virt-sparsify source.qcow2 dest.qcow2
$ guestfish -a dest.qcow2 -m /dev/sda4 set-e2attrs / i

Although the command is called set-e2attrs it can in fact set XFS attributes
too, but see also bug 1771587.

Also when I inspect the Redhat CoreOS image for 4.2 and 4.3 side I see the difference in mount points.

- 4.3 images
```
$ guestfish <<_EOF_
 add crc.qcow2
run
list-filesystems
_EOF_

/dev/sda1: ext4
/dev/sda2: vfat
/dev/sda3: unknown
/dev/sda4: xfs
```

- 4.2 images
```
$  guestfish <<_EOF_
>  add crc.qcow2
> run
> list-filesystems
> _EOF_

/dev/sda1: unknown
/dev/sda2: ext4
/dev/sda3: xfs
```

This came from https://github.com/coreos/coreos-assembler/pull/737/commits/bbc61e3bc0b1666ea951284010602e5c2308b7e2

> Another thing we were discussing is temporarily removing the immutable attribute

Makes sense to me!

I don't think there's any RHCOS bug here, CRC should add those libguestfs commands above to their build scripts, right?

(In reply to Colin Walters from comment #9) 
> CRC should add those libguestfs commands above to their build scripts, right?

Mostly done in https://github.com/code-ready/snc/pull/130

@praveen Based on the latest comments and PR to `code-ready`, this doesn't appear to be an actual RHCOS bug anymore.  Would you object to closing it?

Yes this can be closed.