Bug 2067215
| Summary: | Data loss: thin type volume can suddenly become full of zeroes - device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Mai Ling <mailinglists35> | ||||||||
| Component: | lvm2 | Assignee: | LVM and device-mapper development team <lvm-team> | ||||||||
| lvm2 sub component: | Thin Provisioning | QA Contact: | cluster-qe <cluster-qe> | ||||||||
| Status: | CLOSED NOTABUG | Docs Contact: | |||||||||
| Severity: | high | ||||||||||
| Priority: | unspecified | CC: | agk, heinzm, jbrassow, msnitzer, prajnoha, thornber, zkabelac | ||||||||
| Version: | 8.4 | Flags: | pm-rhel:
mirror+
|
||||||||
| Target Milestone: | rc | ||||||||||
| Target Release: | --- | ||||||||||
| Hardware: | x86_64 | ||||||||||
| OS: | Linux | ||||||||||
| Whiteboard: | |||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||
| Doc Text: | Story Points: | --- | |||||||||
| Clone Of: | Environment: | ||||||||||
| Last Closed: | 2022-05-11 13:30:43 UTC | Type: | Bug | ||||||||
| Regression: | --- | Mount Type: | --- | ||||||||
| Documentation: | --- | CRM: | |||||||||
| Verified Versions: | Category: | --- | |||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||
| Embargoed: | |||||||||||
| Attachments: |
|
||||||||||
Two boots before, the system appeared to have issues unmounting the /var/log as well as closing the luks tmp device: journalctl -b -3|egrep -i 'mount|tmp'|grep -v cups Jan 29 08:40:07 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Jan 30 08:40:12 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Jan 31 08:40:33 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Feb 01 08:40:41 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Feb 02 08:40:56 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Feb 03 08:41:22 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Feb 04 08:41:23 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Feb 05 08:41:41 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Feb 06 08:41:46 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Feb 07 08:42:13 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Feb 08 08:42:31 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Feb 08 10:32:02 localhost.localdomain systemd[1]: Stopping /run/user/1000 mount wrapper... Feb 08 10:32:02 localhost.localdomain systemd[1]: run-user-1000.mount: Succeeded. Feb 08 10:32:02 localhost.localdomain systemd[1]: Stopped /run/user/1000 mount wrapper. Feb 08 10:32:02 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.timer: Succeeded. Feb 08 10:32:05 localhost.localdomain systemd[1]: Stopping Update UTMP about System Boot/Shutdown... Feb 08 10:32:05 localhost.localdomain systemd[1]: Stopping Cryptography Setup for tmp... Feb 08 10:32:05 localhost.localdomain systemd[1]: systemd-update-utmp.service: Succeeded. Feb 08 10:32:05 localhost.localdomain systemd[1]: Stopped Update UTMP about System Boot/Shutdown. Feb 08 10:32:05 localhost.localdomain systemd-cryptsetup[2398346]: Device tmp is still in use. Feb 08 10:32:05 localhost.localdomain systemd[1]: systemd-cryptsetup: Control process exited, code=exited status=1 Feb 08 10:32:05 localhost.localdomain systemd[1]: systemd-cryptsetup: Failed with result 'exit-code'. Feb 08 10:32:05 localhost.localdomain systemd[1]: Stopped Cryptography Setup for tmp. Feb 08 10:32:05 localhost.localdomain systemd[1]: systemd-tmpfiles-setup.service: Succeeded. Feb 08 10:32:05 localhost.localdomain kernel: audit: type=1131 audit(1644309125.472:54920): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Feb 08 10:32:05 localhost.localdomain systemd[1]: Unmounting /tmp... Feb 08 10:32:05 localhost.localdomain systemd[1]: Unmounting /boot... Feb 08 10:32:05 localhost.localdomain systemd[1]: Unmounting /home... Feb 08 10:32:05 localhost.localdomain systemd[1]: Unmounting /var/log... Feb 08 10:32:05 localhost.localdomain umount[2398362]: umount: /var/log: target is busy. Feb 08 10:32:05 localhost.localdomain systemd[1]: var-log.mount: Mount process exited, code=exited status=32 Feb 08 10:32:05 localhost.localdomain systemd[1]: Failed unmounting /var/log. Feb 08 10:32:05 localhost.localdomain systemd[1]: tmp.mount: Succeeded. Feb 08 10:32:05 localhost.localdomain systemd[1]: Unmounted /tmp. Feb 08 10:32:05 localhost.localdomain systemd[1]: boot.mount: Succeeded. Feb 08 10:32:05 localhost.localdomain systemd[1]: Unmounted /boot. Feb 08 10:32:05 localhost.localdomain systemd[1]: home.mount: Succeeded. Feb 08 10:32:05 localhost.localdomain systemd[1]: Unmounted /home. Feb 08 10:32:05 localhost.localdomain systemd[1]: systemd-tmpfiles-setup-dev.service: Succeeded. Feb 08 10:32:05 localhost.localdomain systemd[1]: systemd-remount-fs.service: Succeeded. Feb 08 10:32:05 localhost.localdomain systemd[1]: Stopped Remount Root and Kernel File Systems. Feb 08 10:32:05 localhost.localdomain kernel: audit: type=1131 audit(1644309125.523:54925): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Feb 08 10:32:05 localhost.localdomain kernel: audit: type=1131 audit(1644309125.524:54926): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Feb 08 10:32:05 localhost.localdomain systemd[1]: Reached target Unmount All Filesystems. the next boot (one boot before unbootable machine) appears to have put the tmp volume filesystem in read only mode: [root@localhost ~]# journalctl -b -2|egrep -i 'mount|tmp'|grep -v cups Feb 08 10:32:22 localhost.localdomain kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.4.17-2102.202.5.el8uek.x86_64 ro crashkernel=auto resume=/dev/mapper/swap rd.lvm.lv=plj/root rd.lvm.lv=plj/swap rd.lvm.lv=plj/home rd.lvm.lv=plj/tmp panic=60 quiet rhgb rootflags=x-systemd.device-timeout=30 sysrq_always_enabled root=/dev/mapper/root zswap.compressor=zstd zswap.enabled=1 rd.luks.name=8746df75-bb21-45bb-8266-9aff93e756fe=root rd.luks.name=fd8f2dc2-5d80-4fe9-a7be-d497a152a552=swap rd.luks.name=3d72d2b7-ba83-4e87-ad42-3651566e8d85=tmp rd.luks.name=b47f7980-c029-4e10-9a6e-40b195ebcb9a=home rd.luks.key=/root/random_data_keyfile1 rd.luks.options=discard Feb 08 10:32:22 localhost.localdomain kernel: Kernel command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.4.17-2102.202.5.el8uek.x86_64 ro crashkernel=auto resume=/dev/mapper/swap rd.lvm.lv=plj/root rd.lvm.lv=plj/swap rd.lvm.lv=plj/home rd.lvm.lv=plj/tmp panic=60 quiet rhgb rootflags=x-systemd.device-timeout=30 sysrq_always_enabled root=/dev/mapper/root zswap.compressor=zstd zswap.enabled=1 rd.luks.name=8746df75-bb21-45bb-8266-9aff93e756fe=root rd.luks.name=fd8f2dc2-5d80-4fe9-a7be-d497a152a552=swap rd.luks.name=3d72d2b7-ba83-4e87-ad42-3651566e8d85=tmp rd.luks.name=b47f7980-c029-4e10-9a6e-40b195ebcb9a=home rd.luks.key=/root/random_data_keyfile1 rd.luks.options=discard Feb 08 10:32:22 localhost.localdomain kernel: Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) Feb 08 10:32:22 localhost.localdomain kernel: Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear) Feb 08 10:32:22 localhost.localdomain kernel: devtmpfs: initialized Feb 08 10:32:22 localhost.localdomain systemd[1]: systemd 239 (239-45.0.1.el8) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) Feb 08 10:32:22 localhost.localdomain dracut-cmdline[198]: Using kernel command line parameters: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.4.17-2102.202.5.el8uek.x86_64 ro crashkernel=auto resume=/dev/mapper/swap rd.lvm.lv=plj/root rd.lvm.lv=plj/swap rd.lvm.lv=plj/home rd.lvm.lv=plj/tmp panic=60 quiet rhgb rootflags=x-systemd.device-timeout=30 sysrq_always_enabled root=/dev/mapper/root zswap.compressor=zstd zswap.enabled=1 rd.luks.name=8746df75-bb21-45bb-8266-9aff93e756fe=root rd.luks.name=fd8f2dc2-5d80-4fe9-a7be-d497a152a552=swap rd.luks.name=3d72d2b7-ba83-4e87-ad42-3651566e8d85=tmp rd.luks.name=b47f7980-c029-4e10-9a6e-40b195ebcb9a=home rd.luks.key=/root/random_data_keyfile1 rd.luks.options=discard Feb 08 10:32:24 localhost.localdomain systemd[1]: Mounting Kernel Configuration File System... Feb 08 10:32:24 localhost.localdomain systemd[1]: Mounted Kernel Configuration File System. Feb 08 10:32:26 localhost.localdomain dracut-initqueue[387]: Scanning devices sda2 for LVM logical volumes plj/root plj/swap plj/home plj/tmp Feb 08 10:32:26 localhost.localdomain dracut-initqueue[387]: inactive '/dev/plj/tmp' [16.00 GiB] inherit Feb 08 10:32:27 localhost.localdomain systemd[1]: Starting Cryptography Setup for tmp... Feb 08 10:32:42 localhost.localdomain systemd[1]: Started Cryptography Setup for tmp. Feb 08 10:32:42 localhost.localdomain systemd[1]: Mounting /sysroot... Feb 08 10:32:43 localhost.localdomain systemd[1]: Mounted /sysroot. Feb 08 10:32:43 localhost.localdomain kernel: EXT4-fs (dm-7): mounted filesystem with ordered data mode. Opts: (null) Feb 08 10:32:43 localhost.localdomain systemd[1]: systemd-tmpfiles-setup.service: Succeeded. Feb 08 10:32:43 localhost.localdomain systemd[1]: systemd-tmpfiles-setup-dev.service: Succeeded. Feb 08 10:32:44 localhost.localdomain kernel: SELinux: Permission watch_mount in class file not defined in policy. Feb 08 10:32:44 localhost.localdomain kernel: SELinux: Permission watch_mount in class dir not defined in policy. Feb 08 10:32:44 localhost.localdomain kernel: SELinux: Permission watch_mount in class lnk_file not defined in policy. Feb 08 10:32:44 localhost.localdomain kernel: SELinux: Permission watch_mount in class chr_file not defined in policy. Feb 08 10:32:44 localhost.localdomain kernel: SELinux: Permission watch_mount in class blk_file not defined in policy. Feb 08 10:32:44 localhost.localdomain kernel: SELinux: Permission watch_mount in class sock_file not defined in policy. Feb 08 10:32:44 localhost.localdomain kernel: SELinux: Permission watch_mount in class fifo_file not defined in policy. Feb 08 10:32:44 localhost.localdomain systemd[1]: systemd 239 (239-45.0.1.el8) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) Feb 08 10:32:44 localhost.localdomain kernel: EXT4-fs (dm-7): re-mounted. Opts: (null) Feb 08 10:32:44 localhost.localdomain systemd[1]: Started Remount Root and Kernel File Systems. Feb 08 10:32:44 localhost.localdomain systemd[1]: Mounted Huge Pages File System. Feb 08 10:32:44 localhost.localdomain systemd[1]: Mounted POSIX Message Queue File System. Feb 08 10:32:44 localhost.localdomain systemd[1]: Mounted Kernel Debug File System. Feb 08 10:32:45 localhost.localdomain systemd[1]: Mounting /tmp... Feb 08 10:32:45 localhost.localdomain kernel: EXT4-fs (dm-10): mounted filesystem with ordered data mode. Opts: (null) Feb 08 10:32:45 localhost.localdomain systemd[1]: Mounted /tmp. Feb 08 10:32:45 localhost.localdomain systemd[1]: Mounting /boot... Feb 08 10:32:45 localhost.localdomain systemd[1]: Mounting /home... Feb 08 10:32:45 localhost.localdomain kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null) Feb 08 10:32:45 localhost.localdomain systemd[1]: Mounted /boot. Feb 08 10:32:45 localhost.localdomain kernel: EXT4-fs (dm-9): mounted filesystem with ordered data mode. Opts: (null) Feb 08 10:32:45 localhost.localdomain systemd[1]: Mounted /home. Feb 08 10:32:49 localhost.localdomain systemd[1]: var-log.mount: Directory /var/log to mount over is not empty, mounting anyway. Feb 08 10:32:49 localhost.localdomain systemd[1]: Mounting /var/log... Feb 08 10:32:49 localhost.localdomain kernel: EXT4-fs (dm-13): mounted filesystem with ordered data mode. Opts: (null) Feb 08 10:32:49 localhost.localdomain systemd[1]: Mounted /var/log. Feb 08 10:32:50 localhost.localdomain systemd[1]: Starting Update UTMP about System Boot/Shutdown... Feb 08 10:32:50 localhost.localdomain systemd[1]: Started Update UTMP about System Boot/Shutdown. Feb 08 10:32:58 localhost.localdomain systemd[1]: Starting Update UTMP about System Runlevel Changes... Feb 08 10:32:58 localhost.localdomain systemd[1]: systemd-update-utmp-runlevel.service: Succeeded. Feb 08 10:32:58 localhost.localdomain systemd[1]: Started Update UTMP about System Runlevel Changes. Feb 08 10:33:40 localhost.localdomain systemd[1]: Started /run/user/1000 mount wrapper. Feb 08 10:47:22 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Feb 09 10:47:54 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #652801: block 2621472: comm systemd-tmpfile: unable to read itable block Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130564: lblock 0: comm systemd-tmpfile: error -5 reading directory block Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #261121: block 1048608: comm systemd-tmpfile: unable to read itable block Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #522241: block 2097184: comm systemd-tmpfile: unable to read itable block Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130563: lblock 0: comm systemd-tmpfile: error -5 reading directory block Feb 09 10:47:54 localhost.localdomain systemd-tmpfiles[122395]: stat(/tmp/.Test-unix) failed: Input/output error Feb 09 10:47:54 localhost.localdomain systemd-tmpfiles[122395]: stat(/tmp/.XIM-unix) failed: Input/output error Feb 09 10:47:54 localhost.localdomain systemd-tmpfiles[122395]: stat(/tmp/.font-unix) failed: Input/output error Feb 09 10:47:55 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Feb 09 14:21:22 localhost.localdomain kernel: EXT4-fs (dm-10): Remounting filesystem read-only Feb 10 10:47:58 localhost.localdomain systemd-tmpfiles[241501]: stat(/tmp/.Test-unix) failed: Input/output error Feb 10 10:47:58 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #652801: block 2621472: comm systemd-tmpfile: unable to read itable block Feb 10 10:47:59 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130564: lblock 0: comm systemd-tmpfile: error -5 reading directory block Feb 10 10:47:59 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #261121: block 1048608: comm systemd-tmpfile: unable to read itable block Feb 10 10:47:59 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #522241: block 2097184: comm systemd-tmpfile: unable to read itable block Feb 10 10:47:59 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130563: lblock 0: comm systemd-tmpfile: error -5 reading directory block Feb 10 10:47:58 localhost.localdomain systemd-tmpfiles[241501]: stat(/tmp/.XIM-unix) failed: Input/output error Feb 10 10:47:58 localhost.localdomain systemd-tmpfiles[241501]: stat(/tmp/.font-unix) failed: Input/output error Feb 10 10:47:58 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.L4aA3W): Read-only file system Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.xSUoCX): Read-only file system Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.II2YDW): Read-only file system Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.Xys93k): Read-only file system Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.WR2ohY): Read-only file system Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.u5tGAm): Read-only file system Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.LptauX): Read-only file system Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: stat(/tmp/.Test-unix) failed: Input/output error Feb 11 10:48:14 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #652801: block 2621472: comm systemd-tmpfile: unable to read itable block Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.Qq8DpW): Read-only file system Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.9H33dW): Read-only file system Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.EMXDqV): Read-only file system Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.2JipUY): Read-only file system Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.rgnOnU): Read-only file system Feb 11 10:48:14 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130564: lblock 0: comm systemd-tmpfile: error -5 reading directory block Feb 11 10:48:14 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #261121: block 1048608: comm systemd-tmpfile: unable to read itable block Feb 11 10:48:14 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #522241: block 2097184: comm systemd-tmpfile: unable to read itable block Feb 11 10:48:14 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130563: lblock 0: comm systemd-tmpfile: error -5 reading directory block Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.3yW5AY): Read-only file system Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: stat(/tmp/.XIM-unix) failed: Input/output error Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.KzXBVW): Read-only file system Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: stat(/tmp/.font-unix) failed: Input/output error Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.3pdTiW): Read-only file system Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: utimensat(/tmp): Read-only file system Feb 11 10:48:14 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.wac9MK): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.L4aA3W): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.xSUoCX): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.II2YDW): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.Xys93k): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.WR2ohY): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.u5tGAm): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.UWjlMN): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.LptauX): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: stat(/tmp/.Test-unix) failed: Input/output error Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.Qq8DpW): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.9H33dW): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.EMXDqV): Read-only file system Feb 12 10:48:22 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #652801: block 2621472: comm systemd-tmpfile: unable to read itable block Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.2JipUY): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.oEI6GK): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.rgnOnU): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.3yW5AY): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: stat(/tmp/.XIM-unix) failed: Input/output error Feb 12 10:48:22 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130564: lblock 0: comm systemd-tmpfile: error -5 reading directory block Feb 12 10:48:22 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #261121: block 1048608: comm systemd-tmpfile: unable to read itable block Feb 12 10:48:23 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #522241: block 2097184: comm systemd-tmpfile: unable to read itable block Feb 12 10:48:23 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130563: lblock 0: comm systemd-tmpfile: error -5 reading directory block Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.KzXBVW): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: stat(/tmp/.font-unix) failed: Input/output error Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.3pdTiW): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.8FCvoK): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: utimensat(/tmp): Read-only file system Feb 12 10:48:22 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.wac9MK): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.L4aA3W): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.xSUoCX): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.II2YDW): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.Xys93k): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.WR2ohY): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.eaCRiK): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.QT4SnG): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.u5tGAm): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.AN3PDz): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.U3ppqK): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.UWjlMN): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.LptauX): Read-only file system Feb 13 10:48:46 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #652801: block 2621472: comm systemd-tmpfile: unable to read itable block Feb 13 10:48:47 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130564: lblock 0: comm systemd-tmpfile: error -5 reading directory block Feb 13 10:48:47 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #261121: block 1048608: comm systemd-tmpfile: unable to read itable block Feb 13 10:48:47 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #522241: block 2097184: comm systemd-tmpfile: unable to read itable block Feb 13 10:48:47 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130563: lblock 0: comm systemd-tmpfile: error -5 reading directory block Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: stat(/tmp/.Test-unix) failed: Input/output error Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.Qq8DpW): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.DSWKEG): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.9H33dW): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.EMXDqV): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.2JipUY): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.iDJZDA): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.h3jnsJ): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.oEI6GK): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.XgkciD): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.c07zOD): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.NuIRRz): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.rgnOnU): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.3yW5AY): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: stat(/tmp/.XIM-unix) failed: Input/output error Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.KzXBVW): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.VaONYG): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.3cMbFD): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.VApVAB): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: stat(/tmp/.font-unix) failed: Input/output error Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.3pdTiW): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.8FCvoK): Read-only file system Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: utimensat(/tmp): Read-only file system Feb 13 10:48:47 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Feb 14 09:22:06 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.timer: Succeeded. Feb 14 09:22:06 localhost.localdomain systemd[1]: Stopping /run/user/1000 mount wrapper... Feb 14 09:22:06 localhost.localdomain systemd[1]: run-user-1000.mount: Succeeded. Feb 14 09:22:06 localhost.localdomain systemd[1]: Unmounted /run/user/1000. Feb 14 09:22:06 localhost.localdomain systemd[1]: Stopped /run/user/1000 mount wrapper. Feb 14 09:22:37 localhost.localdomain systemd[1]: Stopping Cryptography Setup for tmp... Feb 14 09:22:37 localhost.localdomain systemd[1]: Stopping Update UTMP about System Boot/Shutdown... Feb 14 09:22:37 localhost.localdomain systemd[1]: systemd-update-utmp.service: Succeeded. Feb 14 09:22:37 localhost.localdomain systemd[1]: Stopped Update UTMP about System Boot/Shutdown. Feb 14 09:22:37 localhost.localdomain systemd-cryptsetup[709545]: Device tmp is still in use. Feb 14 09:22:37 localhost.localdomain systemd[1]: systemd-cryptsetup: Control process exited, code=exited status=1 Feb 14 09:22:37 localhost.localdomain systemd[1]: systemd-cryptsetup: Failed with result 'exit-code'. Feb 14 09:22:37 localhost.localdomain systemd[1]: Stopped Cryptography Setup for tmp. Feb 14 09:22:37 localhost.localdomain systemd[1]: systemd-tmpfiles-setup.service: Succeeded. Feb 14 09:22:37 localhost.localdomain kernel: audit: type=1131 audit(1644823357.217:2687): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Feb 14 09:22:37 localhost.localdomain systemd[1]: Unmounting /var/log... Feb 14 09:22:37 localhost.localdomain systemd[1]: Unmounting /home... Feb 14 09:22:37 localhost.localdomain systemd[1]: Unmounting /boot... Feb 14 09:22:37 localhost.localdomain systemd[1]: Unmounting /tmp... Feb 14 09:22:37 localhost.localdomain umount[709560]: umount: /var/log: target is busy. Feb 14 09:22:37 localhost.localdomain systemd[1]: var-log.mount: Mount process exited, code=exited status=32 Feb 14 09:22:37 localhost.localdomain systemd[1]: Failed unmounting /var/log. Feb 14 09:22:37 localhost.localdomain systemd[1]: tmp.mount: Succeeded. Feb 14 09:22:37 localhost.localdomain systemd[1]: Unmounted /tmp. Feb 14 09:22:37 localhost.localdomain systemd[1]: boot.mount: Succeeded. Feb 14 09:22:37 localhost.localdomain systemd[1]: Unmounted /boot. Feb 14 09:22:37 localhost.localdomain systemd[1]: home.mount: Succeeded. Feb 14 09:22:37 localhost.localdomain systemd[1]: Unmounted /home. Feb 14 09:22:37 localhost.localdomain systemd[1]: systemd-tmpfiles-setup-dev.service: Succeeded. Feb 14 09:22:37 localhost.localdomain kernel: audit: type=1131 audit(1644823357.262:2692): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Feb 14 09:22:37 localhost.localdomain systemd[1]: systemd-remount-fs.service: Succeeded. Feb 14 09:22:37 localhost.localdomain systemd[1]: Stopped Remount Root and Kernel File Systems. Feb 14 09:22:37 localhost.localdomain kernel: audit: type=1131 audit(1644823357.263:2693): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Feb 14 09:22:37 localhost.localdomain systemd[1]: Reached target Unmount All Filesystems. then system became unbootable and booted to dracut emergency shell then system brought to service, where I booted a system rescue iso and don't remember what I did to it (vgchange -ay refused to activate, but then later I don't remember what I did and I brought back all volumes except tmp) Created attachment 1868031 [details]
kernel message of boot -3
Created attachment 1868032 [details]
kernel message of boot -2
Created attachment 1868033 [details]
kernel message of boot -1
hardware details: https://linux-hardware.org/?probe=3dc84dc8ff ooops, the s.m.a.r.t. reports issues with the storage (Reallocated_Sector_Ct,Reported_Uncorrect,Offline_Uncorrectable). I think that could be the cause and not the software! Sorry for the storm. 5 Reallocated_Sector_Ct PO--CK 100 100 005 - 19 187 Reported_Uncorrect -O--CK 100 100 000 - 19 198 Offline_Uncorrectable ----CK 100 100 000 - 4 Unfortunately thin-pool metadata can't work with failing storage - user should use 'raid' for metadata to avoid data loose. If you have broken metadata - you can try to upload them for examination and possibly better recovery. But as such I'm closing BZ - since thin-pool is not protected agains storage corruption (use of raid is suggested). |
Description of problem: a thin lv goes all zeroes without any reason Version-Release number of selected component (if applicable): Version : 2.03.11 How reproducible: It happened once, but I have the same image deployed into several machines so if reproductible I can send you the full image (about 10GB) Steps to Reproduce: 1. do nothing, just let the os run 2. 3. Actual results: [root@localhost ~]# journalctl --since '2022-02-09 10:47:54' --until '2022-02-09 10:47:56' --no-pager -- Logs begin at Fri 2021-04-09 13:02:56 EEST, end at Wed 2022-03-23 16:31:30 EET. -- Feb 09 10:47:54 localhost.localdomain systemd[1]: Starting Cleanup of Temporary Directories... Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94 Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94 Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: thin: process_cell: dm_thin_find_block() failed: error = -15 Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94 Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94 Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: thin: process_cell: dm_thin_find_block() failed: error = -15 Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94 Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94 Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: thin: process_cell: dm_thin_find_block() failed: error = -15 Feb 09 10:47:54 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #652801: block 2621472: comm systemd-tmpfile: unable to read itable block Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94 Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94 Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: thin: process_cell: dm_thin_find_block() failed: error = -15 Feb 09 10:47:55 localhost.localdomain kernel: Buffer I/O error on dev dm-10, logical block 0, lost sync page write Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs (dm-10): I/O error while writing superblock Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94 Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94 Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130564: lblock 0: comm systemd-tmpfile: error -5 reading directory block Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94 Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94 Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94 Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94 Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94 Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94 Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #261121: block 1048608: comm systemd-tmpfile: unable to read itable block Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94 Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94 Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: thin: process_cell: dm_thin_find_block() failed: error = -15 Feb 09 10:47:55 localhost.localdomain kernel: Buffer I/O error on dev dm-10, logical block 0, lost sync page write Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs (dm-10): I/O error while writing superblock Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94 Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94 Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #522241: block 2097184: comm systemd-tmpfile: unable to read itable block Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: thin: process_cell: dm_thin_find_block() failed: error = -15 Feb 09 10:47:55 localhost.localdomain kernel: Buffer I/O error on dev dm-10, logical block 0, lost sync page write Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs (dm-10): I/O error while writing superblock Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130563: lblock 0: comm systemd-tmpfile: error -5 reading directory block Feb 09 10:47:54 localhost.localdomain systemd-tmpfiles[122395]: stat(/tmp/.Test-unix) failed: Input/output error Feb 09 10:47:54 localhost.localdomain systemd-tmpfiles[122395]: stat(/tmp/.XIM-unix) failed: Input/output error Feb 09 10:47:54 localhost.localdomain systemd-tmpfiles[122395]: stat(/tmp/.font-unix) failed: Input/output error Feb 09 10:47:55 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded. Feb 09 10:47:55 localhost.localdomain systemd[1]: Started Cleanup of Temporary Directories. Expected results: keep working Additional info: this is a EL clone (oracle linux). disk layout and info: [root@localhost ~]# parted /dev/sda p Model: ATA HP SSD S700 120G (scsi) Disk /dev/sda: 120GB Sector size (logical/physical): 512B/4096B Partition Table: msdos Disk Flags: Number Start End Size Type File system Flags 1 1049kB 1075MB 1074MB primary ext4 boot 2 1075MB 120GB 119GB primary lvm [root@localhost ~]# lvs -a -o +devices LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert Devices home plj Vwi-aotz-- 16.00g thin 3.97 [lvol1_pmspare] plj ewi------- 12.00m /dev/sda2(2668) root plj Vwi-aotz-- 16.00g thin 37.30 swap plj Vwi-aotz-- 4.00g thin 0.39 thin plj twi-aotz-- <10.40g 79.16 45.61 thin_tdata(0) thin_meta0 plj -wi-a----- 12.00m /dev/sda2(1025) thin_meta0 plj -wi-a----- 12.00m /dev/sda2(2004) [thin_tdata] plj Twi-ao---- <10.40g /dev/sda2(1) [thin_tdata] plj Twi-ao---- <10.40g /dev/sda2(1028) [thin_tdata] plj Twi-ao---- <10.40g /dev/sda2(2006) [thin_tmeta] plj ewi-ao---- 12.00m /dev/sda2(0) [thin_tmeta] plj ewi-ao---- 12.00m /dev/sda2(1027) [thin_tmeta] plj ewi-ao---- 12.00m /dev/sda2(2005) tmp plj Vwi-a-tz-- 16.00g thin 0.00 varlog plj Vwi-aotz-- 16.00g thin 10.07 [root@localhost ~]# lsblk -f NAME FSTYPE LABEL UUID MOUNTPOINT sda ├─sda1 ext4 cb170528-4d10-48ac-959f-cb24feca2baa /boot └─sda2 LVM2_member sZVdNK-9FYl-rwMJ-7uj0-687t-wYA2-1ojz1K ├─plj-thin_tmeta │ └─plj-thin-tpool │ ├─plj-root crypto_LUKS 8746df75-bb21-45bb-8266-9aff93e756fe │ │ └─root ext4 root e6bab103-3c7c-4577-a9e3-7c319c3d2d8d / │ ├─plj-swap crypto_LUKS fd8f2dc2-5d80-4fe9-a7be-d497a152a552 │ │ └─swap swap swap 7e750545-b4d9-47e8-8d66-6bbfcdf8578a [SWAP] │ ├─plj-home crypto_LUKS b47f7980-c029-4e10-9a6e-40b195ebcb9a │ │ └─home ext4 home ecffaf62-4ac3-4688-bee1-654d6498b2f0 /home │ ├─plj-tmp │ ├─plj-thin │ └─plj-varlog crypto_LUKS varlog bc13b95d-fbea-4be3-a0ef-7bf1130bbd3f │ └─varlog ext4 varlog fa50103a-9aaf-4117-a250-1047b2a9afb8 /var/log ├─plj-thin_tdata │ └─plj-thin-tpool │ ├─plj-root crypto_LUKS 8746df75-bb21-45bb-8266-9aff93e756fe │ │ └─root ext4 root e6bab103-3c7c-4577-a9e3-7c319c3d2d8d / │ ├─plj-swap crypto_LUKS fd8f2dc2-5d80-4fe9-a7be-d497a152a552 │ │ └─swap swap swap 7e750545-b4d9-47e8-8d66-6bbfcdf8578a [SWAP] │ ├─plj-home crypto_LUKS b47f7980-c029-4e10-9a6e-40b195ebcb9a │ │ └─home ext4 home ecffaf62-4ac3-4688-bee1-654d6498b2f0 /home │ ├─plj-tmp │ ├─plj-thin │ └─plj-varlog crypto_LUKS varlog bc13b95d-fbea-4be3-a0ef-7bf1130bbd3f │ └─varlog ext4 varlog fa50103a-9aaf-4117-a250-1047b2a9afb8 /var/log └─plj-thin_meta0 sr0 [root@localhost ~]# grep _autoextend_ /etc/lvm/lvm.conf|grep -v \# snapshot_autoextend_threshold = 100 snapshot_autoextend_percent = 20 thin_pool_autoextend_threshold = 95 thin_pool_autoextend_percent = 10 vdo_pool_autoextend_threshold = 100 tmp lv is now useless zeroes: [root@localhost ~]# cmp -b /dev/plj/tmp /dev/zero cmp: EOF on /dev/plj/tmp after byte 17179869184, in line 1 attempting to overuse the space does correctly result in the thin pool autoextending: [root@localhost ~]# LD_PRELOAD=/usr/lib64/nosync/nosync.so rsync -aqxPHAX /usr /home/ Mar 23 16:16:46 localhost.localdomain lvm[1422]: WARNING: Thin pool plj-thin-tpool data is now 83.80% full. Mar 23 16:16:56 localhost.localdomain lvm[1422]: WARNING: Thin pool plj-thin-tpool data is now 87.32% full. Mar 23 16:17:06 localhost.localdomain lvm[1422]: WARNING: Thin pool plj-thin-tpool data is now 91.74% full. Mar 23 16:17:12 localhost.localdomain kernel: device-mapper: thin: 252:2: reached low water mark for data device: sending event. Mar 23 16:17:12 localhost.localdomain lvm[1422]: Size of logical volume plj/thin_tdata changed from <10.40 GiB (2662 extents) to 11.44 GiB (2929 extents). Mar 23 16:17:12 localhost.localdomain kernel: device-mapper: thin: 252:2: growing the data device from 170368 to 187456 blocks Mar 23 16:17:12 localhost.localdomain lvm[1422]: Logical volume plj/thin_tdata successfully resized. Mar 23 16:17:16 localhost.localdomain lvm[1422]: WARNING: Thin pool plj-thin-tpool data is now 88.06% full. Mar 23 16:17:26 localhost.localdomain lvm[1422]: WARNING: Thin pool plj-thin-tpool data is now 92.06% full. Mar 23 16:17:38 localhost.localdomain kernel: device-mapper: thin: 252:2: reached low water mark for data device: sending event. Mar 23 16:17:38 localhost.localdomain lvm[1422]: Rounding size to boundary between physical extents: 16.00 MiB. Mar 23 16:17:38 localhost.localdomain lvm[1422]: Size of logical volume plj/thin_tmeta changed from 12.00 MiB (3 extents) to 16.00 MiB (4 extents). Mar 23 16:17:38 localhost.localdomain kernel: device-mapper: thin: 252:2: switching pool to out-of-data-space (queue IO) mode Mar 23 16:17:38 localhost.localdomain kernel: device-mapper: thin: 252:2: switching pool to write mode Mar 23 16:17:38 localhost.localdomain kernel: device-mapper: thin: 252:2: growing the metadata device from 3072 to 4096 blocks Mar 23 16:17:38 localhost.localdomain kernel: device-mapper: thin: 252:2: reached low water mark for data device: sending event. Mar 23 16:17:38 localhost.localdomain kernel: device-mapper: thin: 252:2: switching pool to out-of-data-space (queue IO) mode Mar 23 16:17:39 localhost.localdomain lvm[1422]: Size of logical volume plj/thin_tdata changed from 11.44 GiB (2929 extents) to <12.59 GiB (3222 extents). Mar 23 16:17:39 localhost.localdomain kernel: device-mapper: thin: 252:2: switching pool to write mode Mar 23 16:17:39 localhost.localdomain kernel: device-mapper: thin: 252:2: growing the data device from 187456 to 206208 blocks Mar 23 16:17:39 localhost.localdomain lvm[1422]: Logical volume plj/thin_tdata successfully resized. Mar 23 16:17:46 localhost.localdomain lvm[1422]: WARNING: Thin pool plj-thin-tpool data is now 88.03% full. Mar 23 16:18:06 localhost.localdomain lvm[1422]: WARNING: Thin pool plj-thin-tpool data is now 90.61% full.