Bug 1474726
| Summary: | fsfreeze -f / will cause system to stuck on readonly operation | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Marcel Gazdík <mgazdik> |
| Component: | kernel | Assignee: | fs-maint |
| kernel sub component: | VFS | QA Contact: | Filesystem QE <fs-qe> |
| Status: | CLOSED NOTABUG | Docs Contact: | |
| Severity: | high | ||
| Priority: | medium | CC: | aviro, bfoster, esandeen, mszeredi, socketpair, vfeenstr |
| Version: | 7.3 | ||
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-07-25 15:13:56 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Marcel Gazdík
2017-07-25 09:18:58 UTC
(In reply to Marcel Gazdík from comment #0) > Description of problem: > We are using rsync to make a backup of a system. To make sure, the backup > will not be corrupted, the / is being frozen. This seems like a slightly odd use case for freeze. Why not use a snapshot or remount read-only? The expected behavior for any incidental write operation on a frozen fs is going to be to block indefinitely, and not all writes are explicit from userspace. > strace -Ttfv cat /var/log/messages will get stuck on close operation, > please have a look at additional info bellow. > I'm assuming /var is part of the root fs..? Also, what is the purpose of the 'cat /var/log/messages' here? Wouldn't writes to this file block? Does the blocking behavior occur without this? ... > [root@leapp-tests-centos7-guest-httpd ~]# ps > PID TTY TIME CMD > 1029 pts/1 00:00:00 bash > 1997 pts/1 00:00:00 strace > 1998 pts/1 00:00:00 tail > 2003 pts/1 00:00:00 cat > 2004 pts/1 00:00:00 ps > > > 2003: > [root@leapp-tests-centos7-guest-httpd 2003]# cat stack > [<ffffffff8120153e>] __sb_start_write+0xde/0x110 > [<ffffffffa020b114>] xfs_trans_alloc+0x24/0x40 [xfs] > [<ffffffffa01e9fe1>] xfs_free_eofblocks+0x111/0x260 [xfs] > [<ffffffffa0200aee>] xfs_release+0x9e/0x170 [xfs] > [<ffffffffa01f0ce5>] xfs_file_release+0x15/0x20 [xfs] > [<ffffffff812007d9>] __fput+0xe9/0x260 > [<ffffffff81200a8e>] ____fput+0xe/0x10 > [<ffffffff810ad1e7>] task_work_run+0xa7/0xe0 > [<ffffffff8109d7a5>] ptrace_notify+0xb5/0xc0 > [<ffffffff81039075>] tracehook_report_syscall_exit+0x45/0xe0 > [<ffffffff81039447>] syscall_trace_leave+0x77/0x110 > [<ffffffff81697aa2>] int_check_syscall_exit_work+0x34/0x3d > [<ffffffffffffffff>] 0xffffffffffffffff > As the stack above shows, release is not necessarily a read-only operation. XFS can allocate post-eof blocks at write time to help reduce fragmentation. These blocks may be truncated on file release, which is an fs modification and thus requires a transaction (which is where a frozen fs blocks write operations). ... > [root@leapp-tests-centos7-guest-httpd 1994]# cat stack > [<ffffffffa01fc645>] xfs_iget+0x175/0x750 [xfs] > [<ffffffffa0205557>] xfs_lookup+0xf7/0x140 [xfs] > [<ffffffffa02021fb>] xfs_vn_lookup+0x7b/0xd0 [xfs] > [<ffffffff81208cad>] lookup_real+0x1d/0x50 > [<ffffffff81209622>] __lookup_hash+0x42/0x60 > [<ffffffff81684412>] lookup_slow+0x42/0xa7 > [<ffffffff8120cb43>] path_lookupat+0x773/0x7a0 > [<ffffffff8120cb9b>] filename_lookup+0x2b/0xc0 > [<ffffffff812105b7>] user_path_at_empty+0x67/0xc0 > [<ffffffff81210621>] user_path_at+0x11/0x20 > [<ffffffff81203a93>] vfs_fstatat+0x63/0xc0 > [<ffffffff81204061>] SYSC_newlstat+0x31/0x60 > [<ffffffff812042ee>] SyS_newlstat+0xe/0x10 > [<ffffffff81697809>] system_call_fastpath+0x16/0x1b > [<ffffffffffffffff>] 0xffffffffffffffff > It's not clear to me where this is blocked. What task is this? Manually freezing the root filesystem is fraught with danger; you really should not expect this to be robust in any way. As Brian says, a volume snapshot would be a much saner way to get a point in time view of the filesystem, if that's what you need. From atime updates to eof block releases, etc, there are all kinds of things at any given time which may need write access, and will block, possibly blocking more things behind them. This doesn't look at all like an xfs bug, this looks like a misapplication of the fsfreeze tool. > This seems like a slightly odd use case for freeze. Why not use a snapshot > or remount read-only? The expected behavior for any incidental write > operation on a frozen fs is going to be to block indefinitely, and not all > writes are explicit from userspace. The purpose is to keep the machine up as long as possible and without any physical interventions from the user (like i.e. booting to a rescue mode). We cannot mount / as read only while it is being used. LVM and other snapshots is one of our future plan, but so far we are trying to use fsfreeze only. > I'm assuming /var is part of the root fs..? Also, what is the purpose of the > 'cat /var/log/messages' here? Wouldn't writes to this file block? Does the > blocking behavior occur without this? Yeah, you are right, /var is part of /. And the purpose of cat command is simply just a show, that we are blocked on close call. We tried multiple files, all of them had same issue, no matter whether we tried to write into them or not. > As the stack above shows, release is not necessarily a read-only operation. > XFS can allocate post-eof blocks at write time to help reduce fragmentation. > These blocks may be truncated on file release, which is an fs modification > and thus requires a transaction (which is where a frozen fs blocks write > operations). OK, so does this means for us, that it is intended behavior even when the fs should be frozen? > It's not clear to me where this is blocked. What task is this? task 1994 is: rsync --server --sender -logDtpAXre.iLsfxC . / I included this stack in case it will be somehow helpful, so if it doesn't contain any information relevant to you, you can simply ignore that. I'm going to close this NOTABUG, as there is no evidence of any flaw in freeze handling or logic. Operations such as "cat" or "close" may seem read-only from the user perspective, but they may in fact require metadata writes which will block. A manual freeze of the root filesystem is dangerous and is almost never the right tool for the job. Thanks, -Eric Kernel 6.3.8. Exactly the same bug: rsync + XFS + close() hangs. https://bugzilla.kernel.org/show_bug.cgi?id=205833 the same bug |