Bug 1535429 - ioprocess does use fsync in writefile
Summary: ioprocess does use fsync in writefile
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: ovirt-distribution
Classification: oVirt
Component: ioprocess
Version: 4.2.1
Hardware: Unspecified
OS: Unspecified
unspecified
high vote
Target Milestone: ovirt-4.2.1
: ---
Assignee: Nir Soffer
QA Contact: Lilach Zitnitski
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-17 11:19 UTC by Nir Soffer
Modified: 2018-03-06 17:18 UTC (History)
3 users (show)

Fixed In Version: ioprocess-1.0.2-1.fc27
Doc Type: Bug Fix
Doc Text:
Cause: ioprocess was not using fsync() after writing a file. Consequence: The written data was buffered in the host page cache, so other hosts may not see the new data when the write completed. If the host or the storage had fatal error before the data was flushed to storage, the data may be lost. Fix: ioprocess use now fsync() after writing a file. Result: If the write complete successfully, other hosts in the cluster will see the new data when reading the file.
Clone Of:
Environment:
Last Closed: 2018-02-22 10:01:06 UTC
oVirt Team: Storage
rule-engine: ovirt-4.2?
tnisan: planning_ack?
tnisan: devel_ack+
rule-engine: testing_ack+


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
oVirt gerrit 86452 None None None 2018-01-17 12:47:00 UTC

Description Nir Soffer 2018-01-17 11:19:36 UTC
Description of problem:

When IProcessClient.writefile() returns, the write was completed, but the data may
be in the host page cache instead of storage. Other hosts will see stale data
until the kernel write the dirty pages to storage. If the storage becomes
non responsive, or a fatal error occur, some or all of the data may be lost.

Callers of this method in vdsm assume that the data was transferred to storage,
and other hosts can see the data once writefile returned successfully.

Version-Release number of selected component (if applicable):
Any

How reproducible:
Always

Steps to Reproduce:
1. Run ioprocess with strace -f
2. Call IOProcessClinet.writefile()

Actual results:
In strace output we see that ioprocess doe snot call fsync on the file descriptor.

Expected results:
ioprocess should call fsync() before returning.

Comment 1 Nir Soffer 2018-01-21 14:47:57 UTC
Tal, this is very bad issue in storage infrastructure, there is no way we can
defer this to 4.3.

Since we have an approved and tested trivial fix, suggesting for 4.2.1.

Comment 2 Fedora Update System 2018-01-30 17:03:42 UTC
ioprocess-1.0.0-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-fbe8141dd2

Comment 3 Fedora Update System 2018-01-31 22:45:24 UTC
ioprocess-1.0.0-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-fbe8141dd2

Comment 4 Fedora Update System 2018-02-04 13:25:55 UTC
ioprocess-1.0.2-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-5fc2a37e8a

Comment 7 Fedora Update System 2018-02-04 20:18:41 UTC
ioprocess-1.0.2-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-5fc2a37e8a

Comment 8 Lilach Zitnitski 2018-02-19 09:17:14 UTC
Full tier1 automation ran using ioprocess-1.0.2-1.el7ev.x86_64, no errors or regressions occurred related to the new ioprocess.

moving to verify.

Comment 9 Sandro Bonazzola 2018-02-22 10:01:06 UTC
This bugzilla is included in oVirt 4.2.1 release, published on Feb 12th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.

Comment 10 Fedora Update System 2018-03-06 17:18:53 UTC
ioprocess-1.0.2-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.