I can't reproduce this.

Can you provide a bzip2'd e2image -r and a testcase for filling it?

the repdocuer is simple, see below
3.18.7-100.fc20.x86_64

i will reboot this VM without mountig /boot to get the e2image and attach it in the next response to this bugreport
___________________________________________________

[root@arrakisvm:/boot]$ cat /usr/local/bin/zerofill.sh
#!/usr/bin/bash
nice -n 19 dd if=/dev/zero of=zero.fill bs=16M
sync
nice -n 19 rm -f zero.fill

[root@arrakisvm:/boot]$ zerofill.sh
dd: Fehler beim Schreiben von „zero.fill“: Auf dem Gerät ist kein Speicherplatz mehr verfügbar
29+0 Datensätze ein
28+0 Datensätze aus
479096832 Bytes (479 MB) kopiert, 1,30679 s, 367 MB/s

[root@arrakisvm:/boot]$ dmesg -c
[21878.108691] EXT4-fs (sda1): Delayed block allocation failed for inode 16 at logical offset 419377 with max blocks 3 with error 28
[21878.108844] EXT4-fs (sda1): This should not happen!! Data will be lost
[21878.108944] EXT4-fs (sda1): Total free blocks count 0
[21878.109016] EXT4-fs (sda1): Free/Dirty block details
[21878.109087] EXT4-fs (sda1): free_blocks=0
[21878.109150] EXT4-fs (sda1): dirty_blocks=147
[21878.109214] EXT4-fs (sda1): Block reservation details
[21878.109439] EXT4-fs (sda1): i_reserved_data_blocks=147
[21878.635678] EXT4-fs (sda1): Delayed block allocation failed for inode 16 at logical offset 424036 with max blocks 32 with error 28
[21878.635978] EXT4-fs (sda1): This should not happen!! Data will be lost

> [root@arrakisvm:/boot]$ dmesg -c
> [21878.108691] EXT4-fs (sda1): Delayed block allocation failed for inode 16 at logical offset 419377 with max blocks 3 with error 28

I wish we had the rest of the dmesg too, to see what the driver says when the filesystem mounts.

-Eric

Created attachment 992266 [details]
e2image and dmesg of full boot

your wish is my command :-)

see attachment, it contains the e2image of /dev/sda1 and after another reboot the "dmesg" output, different machine but same issue (as said all clones)

Something seems wrong with that image.

[ 2590.979127] EXT4-fs (loop1): mounting ext2 file system using the ext4 subsystem
[ 2590.986429] EXT4-fs (loop1): bad geometry: block count 521215 exceeds size of device (16404 blocks)

# debugfs e2image.bin 
debugfs 1.41.12 (17-May-2010)
e2image.bin: Can't read an inode bitmap while reading inode bitmap

# e2image -r e2image.bin new.img
e2image 1.41.12 (17-May-2010)
Illegal block number passed to ext2fs_mark_block_bitmap #4294967295 for in-use block map
Illegal block number passed to ext2fs_mark_block_bitmap #4294967295 for in-use block map
Illegal block number passed to ext2fs_mark_block_bitmap #4294967295 for in-use block map
e2image: Can't read next inode while getting next inode

So I can't analyze or even mount that image for investigation...

Tell me more about what you mean by cloning, too, please.  How were these clones made?

by just clone the whole VMware guest from the goldenmaster created in 2008 from which all other machines are derived

there is a dedicated vritual disk for /boot

well, and as grub2 was introduced i instaleld on one machine gparted, moved the (one and only) /dev/sda1 to make space for grub2 in front, unmounted /boot, made a dd-image of the whole /dev/sda, copied it with scp too all other guests and restored it there 

this should not be an issue because as said the virtual disks are all the same because only one time inserted the Fedora installer in 2008, cloned all other servers and from then dist-upgrades until current F20

what i don't get is why e2image has prolems with that FS
bseides the "/dev/sda1 is in use" (see kernel-list-thread) the filesystem seems to bee fine, the machines boots from it and i used "e2image /dev/sda1 /data/etimage.bin" as command after re-boot the machine without mount /boot

[root@master:~]$ umount /dev/sda1; /usr/sbin/fsck.ext4 -n /dev/sda1; mount /boot
e2fsck 1.42.12 (29-Aug-2014)
Warning!  /dev/sda1 is in use.
boot was not cleanly unmounted, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
boot: 340/130560 files (22.9% non-contiguous), 51645/521215 blocks

Ok, at any rate, the image you provided is corrupt.  (did you use the "-r" option?)

The other thing that's odd is that it says:

> Default mount options:    journal_data_writeback user_xattr acl
> Journal backup:           inode blocks

did this fs once have a journal?

Creating a new filesystem with the same geometry and running your fill-with-dd test, I don't get the error you see.

i did not use any option - if i should have done so please specify exactly commands and options you need

yes that FS was ext4 and the journal removed years ago 

now that you say it the "journal_data_writeback" annoys me too *and* the reason for all that checks yesterday was the for whatever reason "needs_recovery" came back *on some* of the machines leading in a emergency shell because i have to "rebott -f" one machine caused by https://bugzilla.redhat.com/show_bug.cgi?id=1185278

i am really tempted to snapshot the golden master, reformat with an origin ext2, manage somehow to get the kernel and grub2 back on the drive and after that restore a the new dd-image on all the other machines

From comment #1 ;)

> provide a bzip2'd e2image -r

Please use "-r" so I can have a skeleton image I can work with.

thanks,
-Eric

Since this is a own virtual disk only 500 MB large with just /boot on it what about a dd-image where you could have the whole /dev/sda including the partition table and MBR

Xz compressed that shouldn't be that large (only one kernel and the free space filled up with zeros)

You can try it if you like, see how small it is.  e2image -r really should work too.

you can download the dd-image from https://access.thelounge.net/harry/sda-image.img.tar.xz because it becomes 700 KB too large for bugzilla

hohum, ok, that's the *entire disk* .. after extracting the partition, and running e2image -r on it, and bzip2-ing it, it's 46k.

Which is why I asked for an e2image -r, not a device dd image ... you're making it kind of hard to help you efficiently.  ;)

the idea was more that you can make a backup of the entire dd-image and even fireup the zero-fill in a mounted copy and strace, gdb or whatever on the real unchanged data :-)

Right, but I do not need a full dd image in order to do that.

Which is why I asked for e2image -r....



Anyway, something is strange; you don't get "mounting ext2 file system using the ext4 subsystem" message; at least, it's not present in dmesg.txt.  But you do get a "mounted filesystem without journal" message, which makes me think it's getting mounted as ext4.

(I do get the "mounting ext2 file system" message when I mount your image, on 3.18.6-100.fc20).

So I'm not sure it's actually being mounted as ext2, which would explain a lot.

Can you triple-check the fs type in /proc/mounts?

[root@master:~]$ cat /proc/mounts | grep sda
/dev/sda1 /boot ext4 rw,relatime,commit=30 0 0

smells like it is mounted as ext4, on the other hand 2 years ago the ext4 driver was changed to handle ext2/ext3

after explicit chnage /etc/fstab to "ext2" it looks different and the "Delayed block allocation" error messages are also gone - this must be a regression because the change back to ext2 without journal was made years ago and in the meantime i did for sure the zerofill multiple times on all /boot partitions of the cluster

[ 4952.280126] EXT4-fs (sda1): mounting ext2 file system using the ext4 subsystem

[root@master:~]$ cat /proc/mounts | grep sda
/dev/sda1 /boot ext4 rw,relatime,commit=30 0 0

smells like it is mounted as ext4, on the other hand 2 years ago the ext4 driver was changed to handle ext2/ext3

after explicit chnage /etc/fstab to "ext2" it looks different and the "Delayed block allocation" error messages are also gone - this must be a regression because the change back to ext2 without journal was made years ago and in the meantime i did for sure the zerofill multiple times on all /boot partitions of the cluster

[ 4952.280126] EXT4-fs (sda1): mounting ext2 file system using the ext4 subsystem

sorry for the double submit before (network issues on my or bugzillas side)

well, the "is in use" issue is really annyoing
on the master no problem, "umount /boot", change fstab, mount -> done

on at least two other machines no way to mount it again :-(

[root@asterisk:~]$ mount /boot/
mount: /dev/sda1 ist bereits eingehängt oder /boot wird gerade benutzt

[root@asterisk:~]$ df
Dateisystem    Typ  Größe Benutzt Verf. Verw% Eingehängt auf
/dev/sdb1      ext4  5,8G    885M  4,9G   15% /

[root@asterisk:~]$ mount /boot
mount: /dev/sda1 ist bereits eingehängt oder /boot wird gerade benutzt

[root@asterisk:~]$ df
Dateisystem    Typ  Größe Benutzt Verf. Verw% Eingehängt auf
/dev/sdb1      ext4  5,8G    885M  4,9G   15% /

> smells like it is mounted as ext4, on the other hand 2 years ago the ext4 driver was changed to handle ext2/ext3

Yes, it was, but even if we use ext4, it will report ext2 if mounted as ext2.

So this is nothing but user error ... let's open another bug for the in use error, and see if we can resolve it in fewer than 20 comments.  ;)

Also, you owe me a beer.  ;)

if you ever come to Vienna ping me and prepare for get drunken :-)

however, fstab was unchanged for years and that issues where never present until now, but i can live with specify "etx2" in fstab

see https://bugzilla.redhat.com/show_bug.cgi?id=1193203 for "in use" issue