Bug 2525858

Created attachment 2155997 [details]
cloud-init log

Created attachment 2155999 [details]
boot journal log

Created attachment 2156000 [details]
dmseg log

What's the exact SKU it's booting? I think the relevant error from the journal is:

Aug 29 03:47:49 lisa--479-e0-n0 kernel: ntfs3: Enabled Linux POSIX ACLs support
Aug 29 03:47:49 lisa--479-e0-n0 kernel: ntfs3: Read-only LZX/Xpress compression included
Aug 29 03:47:49 lisa--479-e0-n0 kernel: ntfs3(sdb1): Primary boot signature is not NTFS.
Aug 29 03:47:49 lisa--479-e0-n0 kernel: ntfs3(sdb1): try to read out of volume at offset 0x12bfdffe00
Aug 29 03:47:49 lisa--479-e0-n0 cloud-init[843]: 2026-08-29 03:47:49,827 - log_util.py[WARNING]: Running module mounts (<module 'cloudinit.config.cc_mounts' from '/usr/lib/python3.15/site-packages/cloudinit/config/cc_mounts.py'>) failed

So sdb1 is some NTFS-formatted disk. I'm not really up-to-date on kernel NTFS support and what is expected to work. My first guess is a kernel regression, so narrowing down when it started happening would be great.

The failure occurs on the Standard_D2ads_v5 temporary disk. 
Cloud-init reformats /dev/sdb1 from NTFS to ext4, then immediately runs mount -a with the filesystem type set to auto. 
Stale NTFS detection causes the kernel’s ntfs3 driver to attempt mounting the new ext4 filesystem, so cloud-init fails.

Nice, I didn't notice that in the cloud-init logs. After some digging, it turns out it's fairly reproducible:

[root@f45-x64 jcline]# umount /mnt
[root@f45-x64 jcline]# cat /etc/fstab 
UUID=347f7632-c2b6-427a-bb1a-2f1fa842bf9a / btrfs compress=zstd:1,defaults,subvol=root 0 1
UUID=347f7632-c2b6-427a-bb1a-2f1fa842bf9a /boot btrfs compress=zstd:1,subvol=boot 0 0
UUID=347f7632-c2b6-427a-bb1a-2f1fa842bf9a /home btrfs compress=zstd:1,subvol=home 0 0
UUID=347f7632-c2b6-427a-bb1a-2f1fa842bf9a /var btrfs compress=zstd:1,subvol=var 0 0
UUID=5DDE-11AE /boot/efi vfat defaults,umask=0077,shortname=winnt 0 0
/dev/disk/cloud/azure_resource-part1	/mnt	auto	defaults,nofail,x-systemd.after=cloud-init-network.service,_netdev,comment=cloudconfig	0	2
[root@f45-x64 jcline]# mkfs.ntfs -Q /dev/disk/cloud/azure_resource-part1
Cluster size has been automatically set to 4096 bytes.
Creating NTFS volume structures.
mkntfs completed successfully. Have a nice day.
[root@f45-x64 jcline]# mkfs.ext4 /dev/disk/cloud/azure_resource-part1 && mount -a
mke2fs 1.47.4 (6-Mar-2025)
/dev/disk/cloud/azure_resource-part1 contains a ntfs file system
Proceed anyway? (y,N) y
Discarding device blocks: done                            
Creating filesystem with 19660288 4k blocks and 4915200 inodes
Filesystem UUID: 03100853-e448-4662-9611-5a8d03c92194
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: done   

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdb1, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
[root@f45-x64 jcline]# mount -a
[root@f45-x64 jcline]# mount
<snip>
/dev/sdb1 on /mnt type ext4 (rw,relatime,seclabel,x-systemd.after=cloud-init-network.service,_netdev)
[root@f45-x64 jcline]# 


dmesg shows:
[  991.780357] ntfs3(sdb1): Primary boot signature is not NTFS.
[  991.781902] ntfs3(sdb1): try to read out of volume at offset 0x12bfdffe00
[ 1011.215351] EXT4-fs (sdb1): mounted filesystem 8e8bda1b-1aca-4590-9384-08d93f3ccef8 r/w with ordered data mode. Quota mode: none.

so it tries to mount it as ntfs after it's been formatted, but when run later it successfully mounts it as ext4.

This same reproducer works fine in Fedora 44. I haven't confirmed it, but I wonder if it's util-linux 2.42 vs 2.41, and in particular this release note:

- libmount now reads filesystem information from udevd (with fallback to classic libblkid-based detection).

Presumably the mount -a is running fast enough that udevd has the old filesystem value. In fact, this can be reproduced with any two filesystems it seems:

[root@f45-x64 jcline]# umount /mnt
[root@f45-x64 jcline]# mkfs.xfs -f /dev/disk/cloud/azure_resource-part1
meta-data=/dev/disk/cloud/azure_resource-part1 isize=512    agcount=4, agsize=4915072 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=1
         =                       reflink=1    bigtime=1 inobtcount=1 nrext64=1
         =                       exchange=1   metadir=0
data     =                       bsize=4096   blocks=19660288, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1, parent=1
log      =internal log           bsize=4096   blocks=16384, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
         =                       rgcount=0    rgsize=0 extents
         =                       zoned=0      start=0 reserved=0
Discarding blocks...Done.
[root@f45-x64 jcline]# mkfs.ext4 /dev/disk/cloud/azure_resource-part1 && mount -a
mke2fs 1.47.4 (6-Mar-2025)
/dev/disk/cloud/azure_resource-part1 contains a xfs file system
Proceed anyway? (y,N) y
Discarding device blocks: done                            
Creating filesystem with 19660288 4k blocks and 4915200 inodes
Filesystem UUID: c47743b9-455b-4c7e-aca3-bf1eeac65aa7
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: done   

mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdb1, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.

[ 1747.257776] SGI XFS with ACLs, security attributes, realtime, scrub, repair, quota, no debug enabled
[ 1747.263634] XFS (sdb1): Invalid superblock magic number
[ 1806.467245] EXT4-fs (sdb1): mounted filesystem c47743b9-455b-4c7e-aca3-bf1eeac65aa7 r/w with ordered data mode. Quota mode: none.

I think it's probably ideal if cloud-init wrote out its fstab entry with the filesystem type it created, rather than letting it be automatically discovered. Alternatively, it can wait for udev to settle with udevadm settle, as this seems to work fine:

[root@f45-x64 jcline]# mkfs.ext4 /dev/disk/cloud/azure_resource-part1 && udevadm settle && mount -a
mke2fs 1.47.4 (6-Mar-2025)
/dev/disk/cloud/azure_resource-part1 contains a xfs file system
Proceed anyway? (y,N) y
Discarding device blocks: done                            
Creating filesystem with 19660288 4k blocks and 4915200 inodes
Filesystem UUID: bf1c5747-708b-462a-b042-0acb29d3b79c
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: done   

[root@f45-x64 jcline]# mount
<snip>
/dev/sdb1 on /mnt type ext4 (rw,relatime,seclabel,x-systemd.after=cloud-init-network.service,_netdev)

An upstream fix has been posted to wait for pending udev events before
running mount -a:

https://github.com/canonical/cloud-init/pull/7082