Bug 1521071

Summary:	Since upgrade to RHEL 7.4 the following error appears in dovecot logs: Error: Broken file
Product:	Red Hat Enterprise Linux 7	Reporter:	SHAURYA <sshaurya>
Component:	nfs-utils	Assignee:	Steve Dickson <steved>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	Yongcheng Yang <yoyang>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	7.4	CC:	dwysocha, rob.beglinger, sshaurya, tcrider, xzhou, yoyang
Target Milestone:	rc
Target Release:	---
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2019-06-06 19:09:02 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description SHAURYA 2017-12-05 18:22:03 UTC

Description of problem:

We are experiencing lots of following error messages in dovecot main.log since Upgrade to RHEL 7.4

Dec 01 12:36:42 imap(***replaced_mailadress***): Error: Broken file /srv/vmail/j/o/***replaced_mailadress***//maildir/dovecot-uidlist line 159: Invalid data: 
Dec 01 12:38:16 imap(***replaced_mailadress***): Error: Broken or unexpectedly changed file /srv/vmail/j/o/***replaced_mailadress***//maildir/dovecot-uidlist line 161: Invalid data:  - re-reading from beginning
Dec 01 12:38:16 imap(***replaced_mailadress***): Error: Broken file /srv/vmail/j/o/***replaced_mailadress***//maildir/dovecot-uidlist line 161: Invalid data: 
Dec 01 12:43:48 imap(***replaced_mailadress***): Error: Broken or unexpectedly changed file /srv/vmail/j/o/***replaced_mailadress***//maildir/dovecot-uidlist line 1: Invalid data:  - re-reading from beginning
Dec 01 12:43:48 imap(***replaced_mailadress***): Error: Broken file /srv/vmail/j/o/***replaced_mailadress***//maildir/dovecot-uidlist line 162: Invalid data: 
Dec 01 12:44:18 imap(***replaced_mailadress***): Error: /srv/vmail/j/o/***replaced_mailadress***//maildir/dovecot-uidlist: next_uid was lowered (70224 -> 70223, hdr=70223)
Dec 01 12:46:41 imap(***replaced_mailadress***): Error: Broken or unexpectedly changed file /srv/vmail/j/o/***replaced_mailadress***//maildir/dovecot-uidlist line 1: Invalid data:  - re-reading from beginning
Dec 01 12:46:41 imap(***replaced_mailadress***): Error: Broken file /srv/vmail/j/o/***replaced_mailadress***//maildir/dovecot-uidlist line 163: Invalid data:

Version-Release number of selected component (if applicable):

nfs-utils-1.3.0-0.48.el7_4.x86_64 

How reproducible:

Always

Steps to Reproduce:
RHEL 7.4 with dovecot and NFS Shares, happens on both our mail servers
For further information please see also the mails to this topic on the dovecot mailing list:
http://dovecot.markmail.org/message/x43nti7mixddhgi6?q=nfs+order:date-backward&page=2#query:nfs%20order%3Adate-backward+page:2+mid:mpvmpxfy33qdrh3j+state:results
This user thinks it has something to do with NFS, so we are currently thinking about downgrading nfs-utils.


Additional info: Please let me know for other logs

Comment 2 Yongcheng Yang 2017-12-06 07:38:36 UTC

(In reply to SHAURYA from comment #0)
 
> nfs-utils-1.3.0-0.48.el7_4.x86_64 

The above package is to fix Bug 1498959.

However I cannot find anything FNS relative from the above logs.
Would it be possible to have a test downgrading to
nfs-utils-1.3.0-0.48.el7 and check if it can solve this problem?

If really it's nfs issue, please help to log some debug on the
server side, such as:
# systemctl status nfs
# showmount -e localhost
# exportfs -v

Comment 10 Rob Beglinger 2019-10-28 13:49:28 UTC

I am running into this same issue and love to work with you to see if we could troubleshoot the issue to get it resolved.  I recently upgraded from CentOS 7.3 to 7.7 while I realize it is not the official RHEL, it should be close enough to troubleshoot if you are willing to work with me.

Comment 11 Rob Beglinger 2019-10-29 13:46:39 UTC

As a follow up, I believe the issue is in the kernel, not the NFS packages, here's how I came to that conclusion.

After updating all packages to the latest available release:
kernel-3.10.0-1062.1.1.el7.x86_64
libnfsidmap-0.25-19.el7.x86_64
nfs-utils-1.3.0-0.65.el7.x86_64
dovecot-2.2.36-3.el7_7.1.x86_64

I started seeing the above errors in our maillog file.

As a first step, I rolled back dovecot to dovecot-2.2.10-8.el7.x86_64, rebooted and the problem persisted.

Next step, I rolled back nfs-utils to nfs-utils-1.3.0-0.33.el7_3.x86_64 and libnfsidmap to libnfsidmap-0.25-15.el7.x86_64, rebooted and the problem persisted.

Next, I switched the server to boot off of kernel-3.10.0-514.10.2.el7.x86_64 and rebooted.  The problem went away.

Next, I reinstalled the nfs updates nfs-utils-1.3.0-0.65.el7.x86_64 and libnfsidmap-0.25-19.el7.x86_64, rebooted and still no issues.

Finally I updated to dovecot-2.2.36-3.el7_7.1.x86_64, rebooted and still no signs of the issue.

So currently we are running with:
kernel-3.10.0-514.10.2.el7.x86_64
libnfsidmap-0.25-19.el7.x86_64
nfs-utils-1.3.0-0.65.el7.x86_64
dovecot-2.2.36-3.el7_7.1.x86_64

and our maillog is not showing the issues.

If you have the desire to look into this further, please let me know.