Bug 1576765

Summary: automount leaves FDs in half-open state
Product: [Fedora] Fedora Reporter: Brian J. Murrell <brian>
Component: autofsAssignee: Ian Kent <ikent>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 27CC: ikent
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: autofs-5.1.4-8.fc27 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1577700 (view as bug list) Environment:
Last Closed: 2018-06-05 14:09:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1577700    
Attachments:
Description Flags
Patch - fix fd leak in rpc_do_create_client() none

Description Brian J. Murrell 2018-05-10 11:20:01 UTC 
Description of problem:
automount is opening and leaving FDs in a half-open state numbering into the thousands.

Version-Release number of selected component (if applicable):
autofs-5.1.4-7.fc27.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Configure autofs to mount some exports
2. Use the configured automount points
3.

Actual results:
automount continues to amass half-open connections, counting into the thousands

Expected results:
automount should not leave connections half-open

Additional info:
$ ps -ef | grep automount
root      2075     1  0 May04 ?        00:00:40 /usr/sbin/automount --foreground --dont-check-daemon
# lsof -n -p 2075 | grep CLOSE_WAIT | wc -l
2711

Not sure if this is related, and if not, I can open another ticket but automount is also opening a boat-load of UDP sockets such as:

automount 2075 root 5404u  IPv4           18768127      0t0      UDP *:37151

# lsof -n -p 2075 | grep UDP\ \\*: | wc -l
2664

Just those two together amount to over 5000 open file-descriptors.
Comment 1 Ian Kent 2018-05-11 00:58:02 UTC 
(In reply to Brian J. Murrell from comment #0)
> Description of problem:
> automount is opening and leaving FDs in a half-open state numbering into the
> thousands.

Umm ... oops!

> 
> Version-Release number of selected component (if applicable):
> autofs-5.1.4-7.fc27.x86_64
> 
> How reproducible:
> 100%
> 
> Steps to Reproduce:
> 1. Configure autofs to mount some exports
> 2. Use the configured automount points
> 3.
> 
> Actual results:
> automount continues to amass half-open connections, counting into the
> thousands

I see it's happening (now i'm looking) but it's not at all
obvious where the mistake is.

I'll continue to investigate.

Ian
Comment 2 Ian Kent 2018-05-11 01:34:28 UTC 
(In reply to Ian Kent from comment #1)
> 
> I see it's happening (now i'm looking) but it's not at all
> obvious where the mistake is.
> 
> I'll continue to investigate.

I'm not certain this is actually in automount.

It appears to happening in the availability probe code so while
I'm working out what's going on you can avoid calling the probe
(most of the time) by setting mount_wait to a value other than
-1 (60 is probably ok) in the autofs configuration.
Comment 3 Ian Kent 2018-05-11 08:39:36 UTC 
Created attachment 1434809 [details]
Patch - fix fd leak in rpc_do_create_client()
Comment 4 Fedora Update System 2018-05-11 08:49:21 UTC 
autofs-5.1.4-8.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-cafb315302
Comment 5 Ian Kent 2018-05-11 08:56:40 UTC 
(In reply to Fedora Update System from comment #4)
> autofs-5.1.4-8.fc27 has been submitted as an update to Fedora 27.
> https://bodhi.fedoraproject.org/updates/FEDORA-2018-cafb315302

Can you check this fixes the problem when it reaches the
mirrors please.
Comment 6 Brian J. Murrell 2018-05-11 11:20:27 UTC 
Installed.  Open FDs initially at 27 total.  Will let it run for a while and see if that grows significantly with half-open TCP sockets and/or UDP sockets.  If everything looks good after a while I will leave feedback on bodhi.

Thanks very much for the super-quick turnaround on this one!
Comment 7 Fedora Update System 2018-05-11 17:37:23 UTC 
autofs-5.1.4-8.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-cafb315302
Comment 8 Fedora Update System 2018-05-17 01:25:13 UTC 
autofs-5.1.4-17.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-0d73915a38
Comment 9 Fedora Update System 2018-05-17 15:27:18 UTC 
autofs-5.1.4-17.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-0d73915a38
Comment 10 Fedora Update System 2018-06-05 13:42:48 UTC 
autofs-5.1.4-17.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.
Comment 11 Fedora Update System 2018-06-05 14:09:42 UTC 
autofs-5.1.4-8.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.