Fedora Account System
Red Hat Associate
Red Hat Customer
Test to check systemd services are started cleanly on boot(FedoraCloudValidation.verify_services_started(https://github.com/microsoft/lisa/blob/main/lisa/microsoft/testsuites/fedora/fedora_cloud_validation.py#L213) Fedora 45 (prerelease), x86_64 — compose 45.20260828.0 Kernel: 7.2.0-61.fc45.x86_64 cloud-init: 26.1 (Python 3.15.0rc1) Platform: Azure westus3 Test failed with assertion error: AssertionError: [System must be running (got degraded). Failed: cloud-init-main.service loaded failed failed Cloud-init: Single Process cloud-init-network.service loaded failed failed Cloud-init: Network Stage] Expected <degraded> to be equal to <running>, but was not. ----- cloud-init-main.service loaded failed failed Cloud-init: Single Process cloud-init-network.service loaded failed failed Cloud-init: Network Stage ----- Reproducible: Always Steps to Reproduce: 1. Deploy VM from the Fedora 45 prerelease x86_64 image on Azure. 2. Wait for first boot to complete. 3. Run systemctl is-system-running and systemctl --all --failed --no-legend --plain --no-pager. Actual Results: System must be running (got degraded). Failed: cloud-init-main.service loaded failed failed Cloud-init: Single Process cloud-init-network.service loaded failed failed Cloud-init: Network Stage Expected Results: systemctl is-system-running should report running. All cloud-init stages should complete without failure.
Created attachment 2155997 [details] cloud-init log
Created attachment 2155999 [details] boot journal log
Created attachment 2156000 [details] dmseg log
What's the exact SKU it's booting? I think the relevant error from the journal is: Aug 29 03:47:49 lisa--479-e0-n0 kernel: ntfs3: Enabled Linux POSIX ACLs support Aug 29 03:47:49 lisa--479-e0-n0 kernel: ntfs3: Read-only LZX/Xpress compression included Aug 29 03:47:49 lisa--479-e0-n0 kernel: ntfs3(sdb1): Primary boot signature is not NTFS. Aug 29 03:47:49 lisa--479-e0-n0 kernel: ntfs3(sdb1): try to read out of volume at offset 0x12bfdffe00 Aug 29 03:47:49 lisa--479-e0-n0 cloud-init[843]: 2026-08-29 03:47:49,827 - log_util.py[WARNING]: Running module mounts (<module 'cloudinit.config.cc_mounts' from '/usr/lib/python3.15/site-packages/cloudinit/config/cc_mounts.py'>) failed So sdb1 is some NTFS-formatted disk. I'm not really up-to-date on kernel NTFS support and what is expected to work. My first guess is a kernel regression, so narrowing down when it started happening would be great.
The failure occurs on the Standard_D2ads_v5 temporary disk. Cloud-init reformats /dev/sdb1 from NTFS to ext4, then immediately runs mount -a with the filesystem type set to auto. Stale NTFS detection causes the kernel’s ntfs3 driver to attempt mounting the new ext4 filesystem, so cloud-init fails.
Nice, I didn't notice that in the cloud-init logs. After some digging, it turns out it's fairly reproducible: [root@f45-x64 jcline]# umount /mnt [root@f45-x64 jcline]# cat /etc/fstab UUID=347f7632-c2b6-427a-bb1a-2f1fa842bf9a / btrfs compress=zstd:1,defaults,subvol=root 0 1 UUID=347f7632-c2b6-427a-bb1a-2f1fa842bf9a /boot btrfs compress=zstd:1,subvol=boot 0 0 UUID=347f7632-c2b6-427a-bb1a-2f1fa842bf9a /home btrfs compress=zstd:1,subvol=home 0 0 UUID=347f7632-c2b6-427a-bb1a-2f1fa842bf9a /var btrfs compress=zstd:1,subvol=var 0 0 UUID=5DDE-11AE /boot/efi vfat defaults,umask=0077,shortname=winnt 0 0 /dev/disk/cloud/azure_resource-part1 /mnt auto defaults,nofail,x-systemd.after=cloud-init-network.service,_netdev,comment=cloudconfig 0 2 [root@f45-x64 jcline]# mkfs.ntfs -Q /dev/disk/cloud/azure_resource-part1 Cluster size has been automatically set to 4096 bytes. Creating NTFS volume structures. mkntfs completed successfully. Have a nice day. [root@f45-x64 jcline]# mkfs.ext4 /dev/disk/cloud/azure_resource-part1 && mount -a mke2fs 1.47.4 (6-Mar-2025) /dev/disk/cloud/azure_resource-part1 contains a ntfs file system Proceed anyway? (y,N) y Discarding device blocks: done Creating filesystem with 19660288 4k blocks and 4915200 inodes Filesystem UUID: 03100853-e448-4662-9611-5a8d03c92194 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424 Allocating group tables: done Writing inode tables: done Creating journal (131072 blocks): done Writing superblocks and filesystem accounting information: done mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdb1, missing codepage or helper program, or other error. dmesg(1) may have more information after failed mount system call. [root@f45-x64 jcline]# mount -a [root@f45-x64 jcline]# mount <snip> /dev/sdb1 on /mnt type ext4 (rw,relatime,seclabel,x-systemd.after=cloud-init-network.service,_netdev) [root@f45-x64 jcline]# dmesg shows: [ 991.780357] ntfs3(sdb1): Primary boot signature is not NTFS. [ 991.781902] ntfs3(sdb1): try to read out of volume at offset 0x12bfdffe00 [ 1011.215351] EXT4-fs (sdb1): mounted filesystem 8e8bda1b-1aca-4590-9384-08d93f3ccef8 r/w with ordered data mode. Quota mode: none. so it tries to mount it as ntfs after it's been formatted, but when run later it successfully mounts it as ext4. This same reproducer works fine in Fedora 44. I haven't confirmed it, but I wonder if it's util-linux 2.42 vs 2.41, and in particular this release note: - libmount now reads filesystem information from udevd (with fallback to classic libblkid-based detection). Presumably the mount -a is running fast enough that udevd has the old filesystem value. In fact, this can be reproduced with any two filesystems it seems: [root@f45-x64 jcline]# umount /mnt [root@f45-x64 jcline]# mkfs.xfs -f /dev/disk/cloud/azure_resource-part1 meta-data=/dev/disk/cloud/azure_resource-part1 isize=512 agcount=4, agsize=4915072 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=1, rmapbt=1 = reflink=1 bigtime=1 inobtcount=1 nrext64=1 = exchange=1 metadir=0 data = bsize=4096 blocks=19660288, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0, ftype=1, parent=1 log =internal log bsize=4096 blocks=16384, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 = rgcount=0 rgsize=0 extents = zoned=0 start=0 reserved=0 Discarding blocks...Done. [root@f45-x64 jcline]# mkfs.ext4 /dev/disk/cloud/azure_resource-part1 && mount -a mke2fs 1.47.4 (6-Mar-2025) /dev/disk/cloud/azure_resource-part1 contains a xfs file system Proceed anyway? (y,N) y Discarding device blocks: done Creating filesystem with 19660288 4k blocks and 4915200 inodes Filesystem UUID: c47743b9-455b-4c7e-aca3-bf1eeac65aa7 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424 Allocating group tables: done Writing inode tables: done Creating journal (131072 blocks): done Writing superblocks and filesystem accounting information: done mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdb1, missing codepage or helper program, or other error. dmesg(1) may have more information after failed mount system call. [ 1747.257776] SGI XFS with ACLs, security attributes, realtime, scrub, repair, quota, no debug enabled [ 1747.263634] XFS (sdb1): Invalid superblock magic number [ 1806.467245] EXT4-fs (sdb1): mounted filesystem c47743b9-455b-4c7e-aca3-bf1eeac65aa7 r/w with ordered data mode. Quota mode: none. I think it's probably ideal if cloud-init wrote out its fstab entry with the filesystem type it created, rather than letting it be automatically discovered. Alternatively, it can wait for udev to settle with udevadm settle, as this seems to work fine: [root@f45-x64 jcline]# mkfs.ext4 /dev/disk/cloud/azure_resource-part1 && udevadm settle && mount -a mke2fs 1.47.4 (6-Mar-2025) /dev/disk/cloud/azure_resource-part1 contains a xfs file system Proceed anyway? (y,N) y Discarding device blocks: done Creating filesystem with 19660288 4k blocks and 4915200 inodes Filesystem UUID: bf1c5747-708b-462a-b042-0acb29d3b79c Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424 Allocating group tables: done Writing inode tables: done Creating journal (131072 blocks): done Writing superblocks and filesystem accounting information: done [root@f45-x64 jcline]# mount <snip> /dev/sdb1 on /mnt type ext4 (rw,relatime,seclabel,x-systemd.after=cloud-init-network.service,_netdev)
An upstream fix has been posted to wait for pending udev events before running mount -a: https://github.com/canonical/cloud-init/pull/7082