Bug 1535429

Summary:	ioprocess does use fsync in writefile
Product:	[oVirt] ovirt-distribution	Reporter:	Nir Soffer <nsoffer>
Component:	ioprocess	Assignee:	Nir Soffer <nsoffer>
ioprocess sub component:	General	QA Contact:	Lilach Zitnitski <lzitnits>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	unspecified	CC:	amureini, ratamir, tnisan
Version:	4.2.1	Flags:	rule-engine: ovirt-4.2? tnisan: planning_ack? tnisan: devel_ack+ rule-engine: testing_ack+
Target Milestone:	ovirt-4.2.1
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	ioprocess-1.0.2-1.fc27	Doc Type:	Bug Fix
Doc Text:	Cause: ioprocess was not using fsync() after writing a file. Consequence: The written data was buffered in the host page cache, so other hosts may not see the new data when the write completed. If the host or the storage had fatal error before the data was flushed to storage, the data may be lost. Fix: ioprocess use now fsync() after writing a file. Result: If the write complete successfully, other hosts in the cluster will see the new data when reading the file.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-02-22 10:01:06 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	Storage	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Nir Soffer 2018-01-17 11:19:36 UTC

Description of problem:

When IProcessClient.writefile() returns, the write was completed, but the data may
be in the host page cache instead of storage. Other hosts will see stale data
until the kernel write the dirty pages to storage. If the storage becomes
non responsive, or a fatal error occur, some or all of the data may be lost.

Callers of this method in vdsm assume that the data was transferred to storage,
and other hosts can see the data once writefile returned successfully.

Version-Release number of selected component (if applicable):
Any

How reproducible:
Always

Steps to Reproduce:
1. Run ioprocess with strace -f
2. Call IOProcessClinet.writefile()

Actual results:
In strace output we see that ioprocess doe snot call fsync on the file descriptor.

Expected results:
ioprocess should call fsync() before returning.

Comment 1 Nir Soffer 2018-01-21 14:47:57 UTC

Tal, this is very bad issue in storage infrastructure, there is no way we can
defer this to 4.3.

Since we have an approved and tested trivial fix, suggesting for 4.2.1.

Comment 2 Fedora Update System 2018-01-30 17:03:42 UTC

ioprocess-1.0.0-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-fbe8141dd2

Comment 3 Fedora Update System 2018-01-31 22:45:24 UTC

ioprocess-1.0.0-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-fbe8141dd2

Comment 4 Fedora Update System 2018-02-04 13:25:55 UTC

ioprocess-1.0.2-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-5fc2a37e8a

Comment 7 Fedora Update System 2018-02-04 20:18:41 UTC

ioprocess-1.0.2-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-5fc2a37e8a

Comment 8 Lilach Zitnitski 2018-02-19 09:17:14 UTC

Full tier1 automation ran using ioprocess-1.0.2-1.el7ev.x86_64, no errors or regressions occurred related to the new ioprocess.

moving to verify.

Comment 9 Sandro Bonazzola 2018-02-22 10:01:06 UTC

This bugzilla is included in oVirt 4.2.1 release, published on Feb 12th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.1 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.

Comment 10 Fedora Update System 2018-03-06 17:18:53 UTC

ioprocess-1.0.2-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.