RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2067215 - Data loss: thin type volume can suddenly become full of zeroes - device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94
Summary: Data loss: thin type volume can suddenly become full of zeroes - device-mappe...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: lvm2
Version: 8.4
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: ---
Assignee: LVM and device-mapper development team
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-03-23 14:43 UTC by Mai Ling
Modified: 2022-05-11 13:30 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-05-11 13:30:43 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+

Attachments (Terms of Use)
kernel message of boot -3 (223.76 KB, text/plain)
2022-03-24 07:30 UTC, Mai Ling 		no flags Details
kernel message of boot -2 (131.12 KB, text/plain)
2022-03-24 07:30 UTC, Mai Ling 		no flags Details
kernel message of boot -1 (1.19 MB, text/plain)
2022-03-24 07:30 UTC, Mai Ling 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-116566 0 None None None 2022-03-23 19:40:51 UTC

Description Mai Ling 2022-03-23 14:43:31 UTC 
Description of problem:
a thin lv goes all zeroes without any reason

Version-Release number of selected component (if applicable):
Version     : 2.03.11

How reproducible:
It happened once, but I have the same image deployed into several machines so if reproductible I can send you the full image (about 10GB)

Steps to Reproduce:
1. do nothing, just let the os run
2.
3.

Actual results:
[root@localhost ~]# journalctl --since '2022-02-09 10:47:54' --until '2022-02-09 10:47:56' --no-pager
-- Logs begin at Fri 2021-04-09 13:02:56 EEST, end at Wed 2022-03-23 16:31:30 EET. --
Feb 09 10:47:54 localhost.localdomain systemd[1]: Starting Cleanup of Temporary Directories...
Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94
Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94
Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: thin: process_cell: dm_thin_find_block() failed: error = -15
Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94
Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94
Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: thin: process_cell: dm_thin_find_block() failed: error = -15
Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94
Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94
Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: thin: process_cell: dm_thin_find_block() failed: error = -15
Feb 09 10:47:54 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #652801: block 2621472: comm systemd-tmpfile: unable to read itable block
Feb 09 10:47:54 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94
Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94
Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: thin: process_cell: dm_thin_find_block() failed: error = -15
Feb 09 10:47:55 localhost.localdomain kernel: Buffer I/O error on dev dm-10, logical block 0, lost sync page write
Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs (dm-10): I/O error while writing superblock
Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94
Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94
Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130564: lblock 0: comm systemd-tmpfile: error -5 reading directory block
Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94
Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94
Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94
Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94
Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94
Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94
Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #261121: block 1048608: comm systemd-tmpfile: unable to read itable block
Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94
Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94
Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: thin: process_cell: dm_thin_find_block() failed: error = -15
Feb 09 10:47:55 localhost.localdomain kernel: Buffer I/O error on dev dm-10, logical block 0, lost sync page write
Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs (dm-10): I/O error while writing superblock
Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: btree spine: node_check failed: blocknr 10012793332687714485 != wanted 94
Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: block manager: btree_node validator check failed for block 94
Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #522241: block 2097184: comm systemd-tmpfile: unable to read itable block
Feb 09 10:47:55 localhost.localdomain kernel: device-mapper: thin: process_cell: dm_thin_find_block() failed: error = -15
Feb 09 10:47:55 localhost.localdomain kernel: Buffer I/O error on dev dm-10, logical block 0, lost sync page write
Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs (dm-10): I/O error while writing superblock
Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130563: lblock 0: comm systemd-tmpfile: error -5 reading directory block
Feb 09 10:47:54 localhost.localdomain systemd-tmpfiles[122395]: stat(/tmp/.Test-unix) failed: Input/output error
Feb 09 10:47:54 localhost.localdomain systemd-tmpfiles[122395]: stat(/tmp/.XIM-unix) failed: Input/output error
Feb 09 10:47:54 localhost.localdomain systemd-tmpfiles[122395]: stat(/tmp/.font-unix) failed: Input/output error
Feb 09 10:47:55 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Feb 09 10:47:55 localhost.localdomain systemd[1]: Started Cleanup of Temporary Directories.

Expected results:
keep working

Additional info:
this is a EL clone (oracle linux).

disk layout and info:

[root@localhost ~]# parted /dev/sda p
Model: ATA HP SSD S700 120G (scsi)
Disk /dev/sda: 120GB
Sector size (logical/physical): 512B/4096B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
 1 1049kB 1075MB 1074MB primary ext4 boot
 2 1075MB 120GB 119GB primary lvm

[root@localhost ~]# lvs -a -o +devices
  LV              VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
  home            plj Vwi-aotz--  16.00g thin        3.97
  [lvol1_pmspare] plj ewi-------  12.00m                                                     /dev/sda2(2668)
  root            plj Vwi-aotz--  16.00g thin        37.30
  swap            plj Vwi-aotz--   4.00g thin        0.39
  thin            plj twi-aotz-- <10.40g             79.16  45.61                            thin_tdata(0)
  thin_meta0      plj -wi-a-----  12.00m                                                     /dev/sda2(1025)
  thin_meta0      plj -wi-a-----  12.00m                                                     /dev/sda2(2004)
  [thin_tdata]    plj Twi-ao---- <10.40g                                                     /dev/sda2(1)
  [thin_tdata]    plj Twi-ao---- <10.40g                                                     /dev/sda2(1028)
  [thin_tdata]    plj Twi-ao---- <10.40g                                                     /dev/sda2(2006)
  [thin_tmeta]    plj ewi-ao----  12.00m                                                     /dev/sda2(0)
  [thin_tmeta]    plj ewi-ao----  12.00m                                                     /dev/sda2(1027)
  [thin_tmeta]    plj ewi-ao----  12.00m                                                     /dev/sda2(2005)
  tmp             plj Vwi-a-tz--  16.00g thin        0.00
  varlog          plj Vwi-aotz--  16.00g thin        10.07

[root@localhost ~]# lsblk -f
NAME FSTYPE LABEL UUID MOUNTPOINT
sda
├─sda1 ext4 cb170528-4d10-48ac-959f-cb24feca2baa /boot
└─sda2 LVM2_member sZVdNK-9FYl-rwMJ-7uj0-687t-wYA2-1ojz1K
  ├─plj-thin_tmeta
  │ └─plj-thin-tpool
  │ ├─plj-root crypto_LUKS 8746df75-bb21-45bb-8266-9aff93e756fe
  │ │ └─root ext4 root e6bab103-3c7c-4577-a9e3-7c319c3d2d8d /
  │ ├─plj-swap crypto_LUKS fd8f2dc2-5d80-4fe9-a7be-d497a152a552
  │ │ └─swap swap swap 7e750545-b4d9-47e8-8d66-6bbfcdf8578a [SWAP]
  │ ├─plj-home crypto_LUKS b47f7980-c029-4e10-9a6e-40b195ebcb9a
  │ │ └─home ext4 home ecffaf62-4ac3-4688-bee1-654d6498b2f0 /home
  │ ├─plj-tmp
  │ ├─plj-thin
  │ └─plj-varlog crypto_LUKS varlog bc13b95d-fbea-4be3-a0ef-7bf1130bbd3f
  │ └─varlog ext4 varlog fa50103a-9aaf-4117-a250-1047b2a9afb8 /var/log
  ├─plj-thin_tdata
  │ └─plj-thin-tpool
  │ ├─plj-root crypto_LUKS 8746df75-bb21-45bb-8266-9aff93e756fe
  │ │ └─root ext4 root e6bab103-3c7c-4577-a9e3-7c319c3d2d8d /
  │ ├─plj-swap crypto_LUKS fd8f2dc2-5d80-4fe9-a7be-d497a152a552
  │ │ └─swap swap swap 7e750545-b4d9-47e8-8d66-6bbfcdf8578a [SWAP]
  │ ├─plj-home crypto_LUKS b47f7980-c029-4e10-9a6e-40b195ebcb9a
  │ │ └─home ext4 home ecffaf62-4ac3-4688-bee1-654d6498b2f0 /home
  │ ├─plj-tmp
  │ ├─plj-thin
  │ └─plj-varlog crypto_LUKS varlog bc13b95d-fbea-4be3-a0ef-7bf1130bbd3f
  │ └─varlog ext4 varlog fa50103a-9aaf-4117-a250-1047b2a9afb8 /var/log
  └─plj-thin_meta0
sr0

[root@localhost ~]# grep _autoextend_ /etc/lvm/lvm.conf|grep -v \#
        snapshot_autoextend_threshold = 100
        snapshot_autoextend_percent = 20
        thin_pool_autoextend_threshold = 95
        thin_pool_autoextend_percent = 10
        vdo_pool_autoextend_threshold = 100

tmp lv is now useless zeroes:

[root@localhost ~]# cmp -b /dev/plj/tmp /dev/zero
cmp: EOF on /dev/plj/tmp after byte 17179869184, in line 1


attempting to overuse the space does correctly result in the thin pool autoextending:

[root@localhost ~]# LD_PRELOAD=/usr/lib64/nosync/nosync.so rsync -aqxPHAX /usr /home/
Mar 23 16:16:46 localhost.localdomain lvm[1422]: WARNING: Thin pool plj-thin-tpool data is now 83.80% full.
Mar 23 16:16:56 localhost.localdomain lvm[1422]: WARNING: Thin pool plj-thin-tpool data is now 87.32% full.
Mar 23 16:17:06 localhost.localdomain lvm[1422]: WARNING: Thin pool plj-thin-tpool data is now 91.74% full.
Mar 23 16:17:12 localhost.localdomain kernel: device-mapper: thin: 252:2: reached low water mark for data device: sending event.
Mar 23 16:17:12 localhost.localdomain lvm[1422]: Size of logical volume plj/thin_tdata changed from <10.40 GiB (2662 extents) to 11.44 GiB (2929 extents).
Mar 23 16:17:12 localhost.localdomain kernel: device-mapper: thin: 252:2: growing the data device from 170368 to 187456 blocks
Mar 23 16:17:12 localhost.localdomain lvm[1422]: Logical volume plj/thin_tdata successfully resized.
Mar 23 16:17:16 localhost.localdomain lvm[1422]: WARNING: Thin pool plj-thin-tpool data is now 88.06% full.
Mar 23 16:17:26 localhost.localdomain lvm[1422]: WARNING: Thin pool plj-thin-tpool data is now 92.06% full.
Mar 23 16:17:38 localhost.localdomain kernel: device-mapper: thin: 252:2: reached low water mark for data device: sending event.
Mar 23 16:17:38 localhost.localdomain lvm[1422]: Rounding size to boundary between physical extents: 16.00 MiB.
Mar 23 16:17:38 localhost.localdomain lvm[1422]: Size of logical volume plj/thin_tmeta changed from 12.00 MiB (3 extents) to 16.00 MiB (4 extents).
Mar 23 16:17:38 localhost.localdomain kernel: device-mapper: thin: 252:2: switching pool to out-of-data-space (queue IO) mode
Mar 23 16:17:38 localhost.localdomain kernel: device-mapper: thin: 252:2: switching pool to write mode
Mar 23 16:17:38 localhost.localdomain kernel: device-mapper: thin: 252:2: growing the metadata device from 3072 to 4096 blocks
Mar 23 16:17:38 localhost.localdomain kernel: device-mapper: thin: 252:2: reached low water mark for data device: sending event.
Mar 23 16:17:38 localhost.localdomain kernel: device-mapper: thin: 252:2: switching pool to out-of-data-space (queue IO) mode
Mar 23 16:17:39 localhost.localdomain lvm[1422]: Size of logical volume plj/thin_tdata changed from 11.44 GiB (2929 extents) to <12.59 GiB (3222 extents).
Mar 23 16:17:39 localhost.localdomain kernel: device-mapper: thin: 252:2: switching pool to write mode
Mar 23 16:17:39 localhost.localdomain kernel: device-mapper: thin: 252:2: growing the data device from 187456 to 206208 blocks
Mar 23 16:17:39 localhost.localdomain lvm[1422]: Logical volume plj/thin_tdata successfully resized.
Mar 23 16:17:46 localhost.localdomain lvm[1422]: WARNING: Thin pool plj-thin-tpool data is now 88.03% full.
Mar 23 16:18:06 localhost.localdomain lvm[1422]: WARNING: Thin pool plj-thin-tpool data is now 90.61% full.
Comment 1 Mai Ling 2022-03-24 07:21:23 UTC 
Two boots before, the system appeared to have issues unmounting the /var/log as well as closing the luks tmp device:

journalctl -b -3|egrep -i 'mount|tmp'|grep -v cups

Jan 29 08:40:07 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Jan 30 08:40:12 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Jan 31 08:40:33 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Feb 01 08:40:41 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Feb 02 08:40:56 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Feb 03 08:41:22 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Feb 04 08:41:23 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Feb 05 08:41:41 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Feb 06 08:41:46 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Feb 07 08:42:13 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Feb 08 08:42:31 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Feb 08 10:32:02 localhost.localdomain systemd[1]: Stopping /run/user/1000 mount wrapper...
Feb 08 10:32:02 localhost.localdomain systemd[1]: run-user-1000.mount: Succeeded.
Feb 08 10:32:02 localhost.localdomain systemd[1]: Stopped /run/user/1000 mount wrapper.
Feb 08 10:32:02 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.timer: Succeeded.
Feb 08 10:32:05 localhost.localdomain systemd[1]: Stopping Update UTMP about System Boot/Shutdown...
Feb 08 10:32:05 localhost.localdomain systemd[1]: Stopping Cryptography Setup for tmp...
Feb 08 10:32:05 localhost.localdomain systemd[1]: systemd-update-utmp.service: Succeeded.
Feb 08 10:32:05 localhost.localdomain systemd[1]: Stopped Update UTMP about System Boot/Shutdown.
Feb 08 10:32:05 localhost.localdomain systemd-cryptsetup[2398346]: Device tmp is still in use.
Feb 08 10:32:05 localhost.localdomain systemd[1]: systemd-cryptsetup: Control process exited, code=exited status=1
Feb 08 10:32:05 localhost.localdomain systemd[1]: systemd-cryptsetup: Failed with result 'exit-code'.
Feb 08 10:32:05 localhost.localdomain systemd[1]: Stopped Cryptography Setup for tmp.
Feb 08 10:32:05 localhost.localdomain systemd[1]: systemd-tmpfiles-setup.service: Succeeded.
Feb 08 10:32:05 localhost.localdomain kernel: audit: type=1131 audit(1644309125.472:54920): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Feb 08 10:32:05 localhost.localdomain systemd[1]: Unmounting /tmp...
Feb 08 10:32:05 localhost.localdomain systemd[1]: Unmounting /boot...
Feb 08 10:32:05 localhost.localdomain systemd[1]: Unmounting /home...
Feb 08 10:32:05 localhost.localdomain systemd[1]: Unmounting /var/log...
Feb 08 10:32:05 localhost.localdomain umount[2398362]: umount: /var/log: target is busy.
Feb 08 10:32:05 localhost.localdomain systemd[1]: var-log.mount: Mount process exited, code=exited status=32
Feb 08 10:32:05 localhost.localdomain systemd[1]: Failed unmounting /var/log.
Feb 08 10:32:05 localhost.localdomain systemd[1]: tmp.mount: Succeeded.
Feb 08 10:32:05 localhost.localdomain systemd[1]: Unmounted /tmp.
Feb 08 10:32:05 localhost.localdomain systemd[1]: boot.mount: Succeeded.
Feb 08 10:32:05 localhost.localdomain systemd[1]: Unmounted /boot.
Feb 08 10:32:05 localhost.localdomain systemd[1]: home.mount: Succeeded.
Feb 08 10:32:05 localhost.localdomain systemd[1]: Unmounted /home.
Feb 08 10:32:05 localhost.localdomain systemd[1]: systemd-tmpfiles-setup-dev.service: Succeeded.
Feb 08 10:32:05 localhost.localdomain systemd[1]: systemd-remount-fs.service: Succeeded.
Feb 08 10:32:05 localhost.localdomain systemd[1]: Stopped Remount Root and Kernel File Systems.
Feb 08 10:32:05 localhost.localdomain kernel: audit: type=1131 audit(1644309125.523:54925): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Feb 08 10:32:05 localhost.localdomain kernel: audit: type=1131 audit(1644309125.524:54926): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Feb 08 10:32:05 localhost.localdomain systemd[1]: Reached target Unmount All Filesystems.


the next boot (one boot before unbootable machine) appears to have put the tmp volume filesystem in read only mode:

[root@localhost ~]# journalctl -b -2|egrep -i 'mount|tmp'|grep -v cups
Feb 08 10:32:22 localhost.localdomain kernel: Command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.4.17-2102.202.5.el8uek.x86_64 ro crashkernel=auto resume=/dev/mapper/swap rd.lvm.lv=plj/root rd.lvm.lv=plj/swap rd.lvm.lv=plj/home rd.lvm.lv=plj/tmp panic=60 quiet rhgb rootflags=x-systemd.device-timeout=30 sysrq_always_enabled root=/dev/mapper/root zswap.compressor=zstd zswap.enabled=1 rd.luks.name=8746df75-bb21-45bb-8266-9aff93e756fe=root rd.luks.name=fd8f2dc2-5d80-4fe9-a7be-d497a152a552=swap rd.luks.name=3d72d2b7-ba83-4e87-ad42-3651566e8d85=tmp rd.luks.name=b47f7980-c029-4e10-9a6e-40b195ebcb9a=home rd.luks.key=/root/random_data_keyfile1 rd.luks.options=discard
Feb 08 10:32:22 localhost.localdomain kernel: Kernel command line: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.4.17-2102.202.5.el8uek.x86_64 ro crashkernel=auto resume=/dev/mapper/swap rd.lvm.lv=plj/root rd.lvm.lv=plj/swap rd.lvm.lv=plj/home rd.lvm.lv=plj/tmp panic=60 quiet rhgb rootflags=x-systemd.device-timeout=30 sysrq_always_enabled root=/dev/mapper/root zswap.compressor=zstd zswap.enabled=1 rd.luks.name=8746df75-bb21-45bb-8266-9aff93e756fe=root rd.luks.name=fd8f2dc2-5d80-4fe9-a7be-d497a152a552=swap rd.luks.name=3d72d2b7-ba83-4e87-ad42-3651566e8d85=tmp rd.luks.name=b47f7980-c029-4e10-9a6e-40b195ebcb9a=home rd.luks.key=/root/random_data_keyfile1 rd.luks.options=discard
Feb 08 10:32:22 localhost.localdomain kernel: Mount-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
Feb 08 10:32:22 localhost.localdomain kernel: Mountpoint-cache hash table entries: 4096 (order: 3, 32768 bytes, linear)
Feb 08 10:32:22 localhost.localdomain kernel: devtmpfs: initialized
Feb 08 10:32:22 localhost.localdomain systemd[1]: systemd 239 (239-45.0.1.el8) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
Feb 08 10:32:22 localhost.localdomain dracut-cmdline[198]: Using kernel command line parameters: BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.4.17-2102.202.5.el8uek.x86_64 ro crashkernel=auto resume=/dev/mapper/swap rd.lvm.lv=plj/root rd.lvm.lv=plj/swap rd.lvm.lv=plj/home rd.lvm.lv=plj/tmp panic=60 quiet rhgb rootflags=x-systemd.device-timeout=30 sysrq_always_enabled root=/dev/mapper/root zswap.compressor=zstd zswap.enabled=1 rd.luks.name=8746df75-bb21-45bb-8266-9aff93e756fe=root rd.luks.name=fd8f2dc2-5d80-4fe9-a7be-d497a152a552=swap rd.luks.name=3d72d2b7-ba83-4e87-ad42-3651566e8d85=tmp rd.luks.name=b47f7980-c029-4e10-9a6e-40b195ebcb9a=home rd.luks.key=/root/random_data_keyfile1 rd.luks.options=discard
Feb 08 10:32:24 localhost.localdomain systemd[1]: Mounting Kernel Configuration File System...
Feb 08 10:32:24 localhost.localdomain systemd[1]: Mounted Kernel Configuration File System.
Feb 08 10:32:26 localhost.localdomain dracut-initqueue[387]: Scanning devices sda2  for LVM logical volumes plj/root plj/swap plj/home plj/tmp
Feb 08 10:32:26 localhost.localdomain dracut-initqueue[387]: inactive '/dev/plj/tmp' [16.00 GiB] inherit
Feb 08 10:32:27 localhost.localdomain systemd[1]: Starting Cryptography Setup for tmp...
Feb 08 10:32:42 localhost.localdomain systemd[1]: Started Cryptography Setup for tmp.
Feb 08 10:32:42 localhost.localdomain systemd[1]: Mounting /sysroot...
Feb 08 10:32:43 localhost.localdomain systemd[1]: Mounted /sysroot.
Feb 08 10:32:43 localhost.localdomain kernel: EXT4-fs (dm-7): mounted filesystem with ordered data mode. Opts: (null)
Feb 08 10:32:43 localhost.localdomain systemd[1]: systemd-tmpfiles-setup.service: Succeeded.
Feb 08 10:32:43 localhost.localdomain systemd[1]: systemd-tmpfiles-setup-dev.service: Succeeded.
Feb 08 10:32:44 localhost.localdomain kernel: SELinux:  Permission watch_mount in class file not defined in policy.
Feb 08 10:32:44 localhost.localdomain kernel: SELinux:  Permission watch_mount in class dir not defined in policy.
Feb 08 10:32:44 localhost.localdomain kernel: SELinux:  Permission watch_mount in class lnk_file not defined in policy.
Feb 08 10:32:44 localhost.localdomain kernel: SELinux:  Permission watch_mount in class chr_file not defined in policy.
Feb 08 10:32:44 localhost.localdomain kernel: SELinux:  Permission watch_mount in class blk_file not defined in policy.
Feb 08 10:32:44 localhost.localdomain kernel: SELinux:  Permission watch_mount in class sock_file not defined in policy.
Feb 08 10:32:44 localhost.localdomain kernel: SELinux:  Permission watch_mount in class fifo_file not defined in policy.
Feb 08 10:32:44 localhost.localdomain systemd[1]: systemd 239 (239-45.0.1.el8) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy)
Feb 08 10:32:44 localhost.localdomain kernel: EXT4-fs (dm-7): re-mounted. Opts: (null)
Feb 08 10:32:44 localhost.localdomain systemd[1]: Started Remount Root and Kernel File Systems.
Feb 08 10:32:44 localhost.localdomain systemd[1]: Mounted Huge Pages File System.
Feb 08 10:32:44 localhost.localdomain systemd[1]: Mounted POSIX Message Queue File System.
Feb 08 10:32:44 localhost.localdomain systemd[1]: Mounted Kernel Debug File System.
Feb 08 10:32:45 localhost.localdomain systemd[1]: Mounting /tmp...
Feb 08 10:32:45 localhost.localdomain kernel: EXT4-fs (dm-10): mounted filesystem with ordered data mode. Opts: (null)
Feb 08 10:32:45 localhost.localdomain systemd[1]: Mounted /tmp.
Feb 08 10:32:45 localhost.localdomain systemd[1]: Mounting /boot...
Feb 08 10:32:45 localhost.localdomain systemd[1]: Mounting /home...
Feb 08 10:32:45 localhost.localdomain kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
Feb 08 10:32:45 localhost.localdomain systemd[1]: Mounted /boot.
Feb 08 10:32:45 localhost.localdomain kernel: EXT4-fs (dm-9): mounted filesystem with ordered data mode. Opts: (null)
Feb 08 10:32:45 localhost.localdomain systemd[1]: Mounted /home.
Feb 08 10:32:49 localhost.localdomain systemd[1]: var-log.mount: Directory /var/log to mount over is not empty, mounting anyway.
Feb 08 10:32:49 localhost.localdomain systemd[1]: Mounting /var/log...
Feb 08 10:32:49 localhost.localdomain kernel: EXT4-fs (dm-13): mounted filesystem with ordered data mode. Opts: (null)
Feb 08 10:32:49 localhost.localdomain systemd[1]: Mounted /var/log.
Feb 08 10:32:50 localhost.localdomain systemd[1]: Starting Update UTMP about System Boot/Shutdown...
Feb 08 10:32:50 localhost.localdomain systemd[1]: Started Update UTMP about System Boot/Shutdown.
Feb 08 10:32:58 localhost.localdomain systemd[1]: Starting Update UTMP about System Runlevel Changes...
Feb 08 10:32:58 localhost.localdomain systemd[1]: systemd-update-utmp-runlevel.service: Succeeded.
Feb 08 10:32:58 localhost.localdomain systemd[1]: Started Update UTMP about System Runlevel Changes.
Feb 08 10:33:40 localhost.localdomain systemd[1]: Started /run/user/1000 mount wrapper.
Feb 08 10:47:22 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Feb 09 10:47:54 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #652801: block 2621472: comm systemd-tmpfile: unable to read itable block
Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130564: lblock 0: comm systemd-tmpfile: error -5 reading directory block
Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #261121: block 1048608: comm systemd-tmpfile: unable to read itable block
Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #522241: block 2097184: comm systemd-tmpfile: unable to read itable block
Feb 09 10:47:55 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130563: lblock 0: comm systemd-tmpfile: error -5 reading directory block
Feb 09 10:47:54 localhost.localdomain systemd-tmpfiles[122395]: stat(/tmp/.Test-unix) failed: Input/output error
Feb 09 10:47:54 localhost.localdomain systemd-tmpfiles[122395]: stat(/tmp/.XIM-unix) failed: Input/output error
Feb 09 10:47:54 localhost.localdomain systemd-tmpfiles[122395]: stat(/tmp/.font-unix) failed: Input/output error
Feb 09 10:47:55 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Feb 09 14:21:22 localhost.localdomain kernel: EXT4-fs (dm-10): Remounting filesystem read-only
Feb 10 10:47:58 localhost.localdomain systemd-tmpfiles[241501]: stat(/tmp/.Test-unix) failed: Input/output error
Feb 10 10:47:58 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #652801: block 2621472: comm systemd-tmpfile: unable to read itable block
Feb 10 10:47:59 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130564: lblock 0: comm systemd-tmpfile: error -5 reading directory block
Feb 10 10:47:59 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #261121: block 1048608: comm systemd-tmpfile: unable to read itable block
Feb 10 10:47:59 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #522241: block 2097184: comm systemd-tmpfile: unable to read itable block
Feb 10 10:47:59 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130563: lblock 0: comm systemd-tmpfile: error -5 reading directory block
Feb 10 10:47:58 localhost.localdomain systemd-tmpfiles[241501]: stat(/tmp/.XIM-unix) failed: Input/output error
Feb 10 10:47:58 localhost.localdomain systemd-tmpfiles[241501]: stat(/tmp/.font-unix) failed: Input/output error
Feb 10 10:47:58 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.L4aA3W): Read-only file system
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.xSUoCX): Read-only file system
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.II2YDW): Read-only file system
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.Xys93k): Read-only file system
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.WR2ohY): Read-only file system
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.u5tGAm): Read-only file system
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.LptauX): Read-only file system
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: stat(/tmp/.Test-unix) failed: Input/output error
Feb 11 10:48:14 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #652801: block 2621472: comm systemd-tmpfile: unable to read itable block
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.Qq8DpW): Read-only file system
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.9H33dW): Read-only file system
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.EMXDqV): Read-only file system
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.2JipUY): Read-only file system
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.rgnOnU): Read-only file system
Feb 11 10:48:14 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130564: lblock 0: comm systemd-tmpfile: error -5 reading directory block
Feb 11 10:48:14 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #261121: block 1048608: comm systemd-tmpfile: unable to read itable block
Feb 11 10:48:14 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #522241: block 2097184: comm systemd-tmpfile: unable to read itable block
Feb 11 10:48:14 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130563: lblock 0: comm systemd-tmpfile: error -5 reading directory block
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.3yW5AY): Read-only file system
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: stat(/tmp/.XIM-unix) failed: Input/output error
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.KzXBVW): Read-only file system
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: stat(/tmp/.font-unix) failed: Input/output error
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: unlink(/tmp/.com.microsoft.Edge.3pdTiW): Read-only file system
Feb 11 10:48:14 localhost.localdomain systemd-tmpfiles[360187]: utimensat(/tmp): Read-only file system
Feb 11 10:48:14 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.wac9MK): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.L4aA3W): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.xSUoCX): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.II2YDW): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.Xys93k): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.WR2ohY): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.u5tGAm): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.UWjlMN): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.LptauX): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: stat(/tmp/.Test-unix) failed: Input/output error
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.Qq8DpW): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.9H33dW): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.EMXDqV): Read-only file system
Feb 12 10:48:22 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #652801: block 2621472: comm systemd-tmpfile: unable to read itable block
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.2JipUY): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.oEI6GK): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.rgnOnU): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.3yW5AY): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: stat(/tmp/.XIM-unix) failed: Input/output error
Feb 12 10:48:22 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130564: lblock 0: comm systemd-tmpfile: error -5 reading directory block
Feb 12 10:48:22 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #261121: block 1048608: comm systemd-tmpfile: unable to read itable block
Feb 12 10:48:23 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #522241: block 2097184: comm systemd-tmpfile: unable to read itable block
Feb 12 10:48:23 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130563: lblock 0: comm systemd-tmpfile: error -5 reading directory block
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.KzXBVW): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: stat(/tmp/.font-unix) failed: Input/output error
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.3pdTiW): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: unlink(/tmp/.com.microsoft.Edge.8FCvoK): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd-tmpfiles[478862]: utimensat(/tmp): Read-only file system
Feb 12 10:48:22 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.wac9MK): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.L4aA3W): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.xSUoCX): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.II2YDW): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.Xys93k): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.WR2ohY): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.eaCRiK): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.QT4SnG): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.u5tGAm): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.AN3PDz): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.U3ppqK): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.UWjlMN): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.LptauX): Read-only file system
Feb 13 10:48:46 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #652801: block 2621472: comm systemd-tmpfile: unable to read itable block
Feb 13 10:48:47 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130564: lblock 0: comm systemd-tmpfile: error -5 reading directory block
Feb 13 10:48:47 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #261121: block 1048608: comm systemd-tmpfile: unable to read itable block
Feb 13 10:48:47 localhost.localdomain kernel: EXT4-fs error (device dm-10): __ext4_get_inode_loc:4713: inode #522241: block 2097184: comm systemd-tmpfile: unable to read itable block
Feb 13 10:48:47 localhost.localdomain kernel: EXT4-fs warning (device dm-10): htree_dirblock_to_tree:997: inode #130563: lblock 0: comm systemd-tmpfile: error -5 reading directory block
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: stat(/tmp/.Test-unix) failed: Input/output error
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.Qq8DpW): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.DSWKEG): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.9H33dW): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.EMXDqV): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.2JipUY): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.iDJZDA): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.h3jnsJ): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.oEI6GK): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.XgkciD): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.c07zOD): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.NuIRRz): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.rgnOnU): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.3yW5AY): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: stat(/tmp/.XIM-unix) failed: Input/output error
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.KzXBVW): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.VaONYG): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.3cMbFD): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.VApVAB): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: stat(/tmp/.font-unix) failed: Input/output error
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.3pdTiW): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: unlink(/tmp/.com.microsoft.Edge.8FCvoK): Read-only file system
Feb 13 10:48:46 localhost.localdomain systemd-tmpfiles[597431]: utimensat(/tmp): Read-only file system
Feb 13 10:48:47 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.service: Succeeded.
Feb 14 09:22:06 localhost.localdomain systemd[1]: systemd-tmpfiles-clean.timer: Succeeded.
Feb 14 09:22:06 localhost.localdomain systemd[1]: Stopping /run/user/1000 mount wrapper...
Feb 14 09:22:06 localhost.localdomain systemd[1]: run-user-1000.mount: Succeeded.
Feb 14 09:22:06 localhost.localdomain systemd[1]: Unmounted /run/user/1000.
Feb 14 09:22:06 localhost.localdomain systemd[1]: Stopped /run/user/1000 mount wrapper.
Feb 14 09:22:37 localhost.localdomain systemd[1]: Stopping Cryptography Setup for tmp...
Feb 14 09:22:37 localhost.localdomain systemd[1]: Stopping Update UTMP about System Boot/Shutdown...
Feb 14 09:22:37 localhost.localdomain systemd[1]: systemd-update-utmp.service: Succeeded.
Feb 14 09:22:37 localhost.localdomain systemd[1]: Stopped Update UTMP about System Boot/Shutdown.
Feb 14 09:22:37 localhost.localdomain systemd-cryptsetup[709545]: Device tmp is still in use.
Feb 14 09:22:37 localhost.localdomain systemd[1]: systemd-cryptsetup: Control process exited, code=exited status=1
Feb 14 09:22:37 localhost.localdomain systemd[1]: systemd-cryptsetup: Failed with result 'exit-code'.
Feb 14 09:22:37 localhost.localdomain systemd[1]: Stopped Cryptography Setup for tmp.
Feb 14 09:22:37 localhost.localdomain systemd[1]: systemd-tmpfiles-setup.service: Succeeded.
Feb 14 09:22:37 localhost.localdomain kernel: audit: type=1131 audit(1644823357.217:2687): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Feb 14 09:22:37 localhost.localdomain systemd[1]: Unmounting /var/log...
Feb 14 09:22:37 localhost.localdomain systemd[1]: Unmounting /home...
Feb 14 09:22:37 localhost.localdomain systemd[1]: Unmounting /boot...
Feb 14 09:22:37 localhost.localdomain systemd[1]: Unmounting /tmp...
Feb 14 09:22:37 localhost.localdomain umount[709560]: umount: /var/log: target is busy.
Feb 14 09:22:37 localhost.localdomain systemd[1]: var-log.mount: Mount process exited, code=exited status=32
Feb 14 09:22:37 localhost.localdomain systemd[1]: Failed unmounting /var/log.
Feb 14 09:22:37 localhost.localdomain systemd[1]: tmp.mount: Succeeded.
Feb 14 09:22:37 localhost.localdomain systemd[1]: Unmounted /tmp.
Feb 14 09:22:37 localhost.localdomain systemd[1]: boot.mount: Succeeded.
Feb 14 09:22:37 localhost.localdomain systemd[1]: Unmounted /boot.
Feb 14 09:22:37 localhost.localdomain systemd[1]: home.mount: Succeeded.
Feb 14 09:22:37 localhost.localdomain systemd[1]: Unmounted /home.
Feb 14 09:22:37 localhost.localdomain systemd[1]: systemd-tmpfiles-setup-dev.service: Succeeded.
Feb 14 09:22:37 localhost.localdomain kernel: audit: type=1131 audit(1644823357.262:2692): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Feb 14 09:22:37 localhost.localdomain systemd[1]: systemd-remount-fs.service: Succeeded.
Feb 14 09:22:37 localhost.localdomain systemd[1]: Stopped Remount Root and Kernel File Systems.
Feb 14 09:22:37 localhost.localdomain kernel: audit: type=1131 audit(1644823357.263:2693): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-remount-fs comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Feb 14 09:22:37 localhost.localdomain systemd[1]: Reached target Unmount All Filesystems.

then system became unbootable and booted to dracut emergency shell

then system brought to service, where I booted a system rescue iso and don't remember what I did to it (vgchange -ay refused to activate, but then later I don't remember what I did and I brought back all volumes except tmp)
Comment 2 Mai Ling 2022-03-24 07:30:10 UTC 
Created attachment 1868031 [details]
kernel message of boot -3
Comment 3 Mai Ling 2022-03-24 07:30:31 UTC 
Created attachment 1868032 [details]
kernel message of boot -2
Comment 4 Mai Ling 2022-03-24 07:30:52 UTC 
Created attachment 1868033 [details]
kernel message of boot -1
Comment 5 Mai Ling 2022-03-24 07:57:14 UTC 
hardware details: https://linux-hardware.org/?probe=3dc84dc8ff
Comment 6 Mai Ling 2022-03-24 08:03:07 UTC 
ooops, the s.m.a.r.t. reports issues with the storage (Reallocated_Sector_Ct,Reported_Uncorrect,Offline_Uncorrectable).
I think that could be the cause and not the software!

Sorry for the storm.

  5 Reallocated_Sector_Ct   PO--CK   100   100   005    -    19
187 Reported_Uncorrect      -O--CK   100   100   000    -    19
198 Offline_Uncorrectable   ----CK   100   100   000    -    4
Comment 7 Zdenek Kabelac 2022-05-11 13:30:43 UTC 
Unfortunately thin-pool metadata can't work with  failing storage - user should use 'raid' for metadata to avoid data loose.

If you have broken metadata - you can try to upload them for examination and possibly better recovery.

But as such I'm closing BZ - since thin-pool is not protected agains storage corruption (use of raid is suggested).
Note You need to log in before you can comment on or make changes to this bug.