Bug 1609128

Summary: autofs reload is unable to activate new map entries, it is autofs restart which shows new map entries.
Product: Red Hat Enterprise Linux 7 Reporter: Ashima Rawat <arawat>
Component: autofsAssignee: Ian Kent <ikent>
Status: CLOSED ERRATA QA Contact: Kun Wang <kunwan>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 7.2CC: arawat, ikent, kunwan, rhandlin, swhiteho, xzhou
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: autofs-5.0.7-94.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1611866 (view as bug list) Environment:
Last Closed: 2018-10-30 11:41:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1611866    
Attachments:
Description Flags
Patch - fix update_negative_cache() map source usage
none
Patch - fix update_negative_cache() map source usage (updated)
none
Patch - fix update_negative_cache() map source usage (update 2) none

Description Ashima Rawat 2018-07-27 05:04:28 UTC
Description of problem:

autofs reload is unable to activate new map entries, it is autofs restart which shows new map entries.

There was errata released as part of Bugzilla:
https://bugzilla.redhat.com/show_bug.cgi?id=1156662

Wherein the issue got fixed in autofs-5.0.7-48.el7.x86_64.rpm.
However issue still persist when using indirect maps.


Version-Release number of selected component (if applicable):
RHEL 7.2 
autofs-5.0.7-83.el7.x86_64


How reproducible:


Steps to Reproduce:
1. exported share from nfs server as per customer environment.
2. autofs indirect map entries as per customer's environment.

# cat etc/auto.master 
# Managed by CFEngine
/n		/etc/autofs/n

[root@rhel7u2-2 /]# cat /etc/autofs/n
algsim_sources  -rw,intr,hard    10.10.188.239:/nc_algsim_sources
functionaltestdata  -rw,intr,hard    10.10.188.239:/nc_func_tdata
pcni_large_files         -rw,intr,hard   10.10.188.239:/nc_pcni_large_files

3. Reloaded autofs service:

[root@rhel7u2-2 /]# service autofs reload
Redirecting to /bin/systemctl reload  autofs.service
[root@rhel7u2-2 /]# cd /n/pcni_large_files
-bash: cd: /n/pcni_large_files: No such file or directory

4. enabled autofs debugging to see as to why autofs reload is failing.

specific autofs debugging logs with respect to autofs reload not working::

Jul 27 11:35:22 rhel7u2-2 automount[3105]: handle_packet_missing_indirect: token 31, name pcni_large_files, request pid 2058
Jul 27 11:35:22 rhel7u2-2 automount[3105]: attempting to mount entry /n/pcni_large_files
Jul 27 11:35:22 rhel7u2-2 automount[3105]: lookup_mount: lookup(file): looking up pcni_large_files
Jul 27 11:35:22 rhel7u2-2 automount[3105]: key "pcni_large_files" not found in map source(s).
Jul 27 11:35:22 rhel7u2-2 automount[3105]: dev_ioctl_send_fail: token = 31
Jul 27 11:35:22 rhel7u2-2 automount[3105]: failed to mount /n/pcni_large_files
Jul 27 11:35:22 rhel7u2-2 automount[3105]: handle_packet: type = 3
Jul 27 11:35:22 rhel7u2-2 automount[3105]: handle_packet_missing_indirect: token 32, name pcni_large_files, request pid 2058
Jul 27 11:35:22 rhel7u2-2 automount[3105]: dev_ioctl_send_fail: token = 32
Jul 27 11:35:22 rhel7u2-2 automount[3105]: handle_packet: type = 3
Jul 27 11:35:22 rhel7u2-2 automount[3105]: handle_packet_missing_indirect: token 33, name pcni_large_files, request pid 2058
Jul 27 11:35:22 rhel7u2-2 automount[3105]: dev_ioctl_send_fail: token = 33
Jul 27 11:35:44 rhel7u2-2 automount[3105]: st_expire: state 1 path /misc
Jul 27 11:35:44 rhel7u2-2 automount[3105]: expire_proc: exp_proc = 140057420781312 path /misc
Jul 27 11:35:44 rhel7u2-2 automount[3105]: expire_cleanup: got thid 140057420781312 path /misc stat 0
Jul 27 11:35:44 rhel7u2-2 automount[3105]: expire_cleanup: sigchld: exp 140057420781312 finished, switching from 2 to 1
Jul 27 11:35:44 rhel7u2-2 automount[3105]: st_ready: st_ready(): state = 2 path /misc
Jul 27 11:35:47 rhel7u2-2 automount[3105]: st_expire: state 1 path /n
Jul 27 11:35:47 rhel7u2-2 automount[3105]: expire_proc: exp_proc = 140057420781312 path /n
Jul 27 11:35:47 rhel7u2-2 automount[3105]: expire_cleanup: got thid 140057420781312 path /n stat 0
Jul 27 11:35:47 rhel7u2-2 automount[3105]: expire_cleanup: sigchld: exp 140057420781312 finished, switching from 2 to 1
Jul 27 11:35:47 rhel7u2-2 automount[3105]: st_ready: st_ready(): state = 2 path /n

[5] autofs restart works:

[root@rhel7u2-2 /]# systemctl restart autofs.service
[root@rhel7u2-2 /]# cd /n/pcni_large_files
[root@rhel7u2-2 pcni_large_files]# df -h
Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/myvg-rootvol            8.3G  1.7G  6.7G  21% /
devtmpfs                            487M     0  487M   0% /dev
tmpfs                               497M     0  497M   0% /dev/shm
tmpfs                               497M  6.6M  490M   2% /run
tmpfs                               497M     0  497M   0% /sys/fs/cgroup
/dev/vda1                           190M  104M   77M  58% /boot
/dev/mapper/myvg-homevol            497M   26M  472M   6% /home
tmpfs                               100M     0  100M   0% /run/user/0
10.10.188.239:/nc_pcni_large_files  8.5G  1.6G  7.0G  18% /n/pcni_large_files

Comment 4 Ian Kent 2018-07-27 10:53:46 UTC
(In reply to Ashima Rawat from comment #0)
> Description of problem:
> 
> autofs reload is unable to activate new map entries, it is autofs restart
> which shows new map entries.
> 
> There was errata released as part of Bugzilla:
> https://bugzilla.redhat.com/show_bug.cgi?id=1156662

That doesn't look related at all.

> 
> Wherein the issue got fixed in autofs-5.0.7-48.el7.x86_64.rpm.
> However issue still persist when using indirect maps.
> 
> 
> Version-Release number of selected component (if applicable):
> RHEL 7.2 
> autofs-5.0.7-83.el7.x86_64
> 
> 
> How reproducible:
> 
> 
> Steps to Reproduce:
> 1. exported share from nfs server as per customer environment.
> 2. autofs indirect map entries as per customer's environment.
> 
> # cat etc/auto.master 
> # Managed by CFEngine
> /n		/etc/autofs/n
> 
> [root@rhel7u2-2 /]# cat /etc/autofs/n
> algsim_sources  -rw,intr,hard    10.10.188.239:/nc_algsim_sources
> functionaltestdata  -rw,intr,hard    10.10.188.239:/nc_func_tdata
> pcni_large_files         -rw,intr,hard   10.10.188.239:/nc_pcni_large_files
> 
> 3. Reloaded autofs service:
> 
> [root@rhel7u2-2 /]# service autofs reload
> Redirecting to /bin/systemctl reload  autofs.service
> [root@rhel7u2-2 /]# cd /n/pcni_large_files
> -bash: cd: /n/pcni_large_files: No such file or directory

I'm not able to reproduce it revision 83.

My setup is a little different but that shouldn't matter
because the log fragment provided implies that the new
master map entry is seen and mounted but the file map
lookup fails to find the map key. My test map just has
different mount locations, the master map entry and the
map keys are the same.

Lets start by providing a full debug log from startup,
through reload and the mount attempt please.

Also check the ownership and permissions along the path
/etc/autofs/n (probably not the problem but we must check
it).

I might need to check the kernel too, what revision is in
use here?

Ian

Comment 6 Ian Kent 2018-08-01 03:39:44 UTC
(In reply to Ian Kent from comment #4)
> > 
> > How reproducible:
> > 
> > 
> > Steps to Reproduce:
> > 1. exported share from nfs server as per customer environment.
> > 2. autofs indirect map entries as per customer's environment.
> > 
> > # cat etc/auto.master 
> > # Managed by CFEngine
> > /n		/etc/autofs/n
> > 
> > [root@rhel7u2-2 /]# cat /etc/autofs/n
> > algsim_sources  -rw,intr,hard    10.10.188.239:/nc_algsim_sources
> > functionaltestdata  -rw,intr,hard    10.10.188.239:/nc_func_tdata
> > pcni_large_files         -rw,intr,hard   10.10.188.239:/nc_pcni_large_files
> > 
> > 3. Reloaded autofs service:
> > 
> > [root@rhel7u2-2 /]# service autofs reload
> > Redirecting to /bin/systemctl reload  autofs.service
> > [root@rhel7u2-2 /]# cd /n/pcni_large_files
> > -bash: cd: /n/pcni_large_files: No such file or directory
> 
> I'm not able to reproduce it revision 83.

I have been able to identify a use case that produces the
symptoms you describe.

But producing it needs a much more specific procedure than
described above.

First in your description it's unclear if the thing that's
being updated is the master map or the map used by the
master map entry.

Assuming it is the map used by the given master map entry
and a lookup is done for the map key that is about to be
added, followed by adding the key to the map and doing a
map reload, then lookups for this key continue to fail
as described.

Ian

Comment 9 Ian Kent 2018-08-01 06:40:56 UTC
A build of autofs that may resolve the problem here is
located at:
http://people.redhat.com/~ikent/autofs-5.0.7-83.bz1609128.el7/

Please test this rpm to see if it helps.
Ian

Comment 10 Ian Kent 2018-08-01 06:42:19 UTC
Created attachment 1472003 [details]
Patch - fix update_negative_cache() map source usage

Comment 11 Ian Kent 2018-08-02 01:01:41 UTC
Created attachment 1472214 [details]
Patch - fix update_negative_cache() map source usage (updated)

The patch has been updated.

The change to the patch is for completeness, it should make
no difference to the functionality because the matching in
map instance lookups is lazy and there is only one instance
in this case.

I will post an updated build too.

Comment 12 Ian Kent 2018-08-02 01:04:15 UTC
(In reply to Ian Kent from comment #9)
> A build of autofs that may resolve the problem here is
> located at:
> http://people.redhat.com/~ikent/autofs-5.0.7-83.bz1609128.el7/

There is an updated build at the location above.

Please use:
autofs-5.0.7-83.bz1609128.2.el7.x86_64.rpm
autofs-debuginfo-5.0.7-83.bz1609128.2.el7.x86_64.rpm
autofs-5.0.7-83.bz1609128.2.el7.src.rpm

for testing.

Comment 13 Steve Whitehouse 2018-08-02 10:49:37 UTC
Are you able to test Ian's build and let us know whether it fixes the problem?

Comment 15 Ashima Rawat 2018-08-02 11:57:17 UTC
Hi Ian,

customer mentioned that these test autofs packages solved the issue @ his end.

Thanks 
Ashima

Comment 19 Ian Kent 2018-08-04 01:10:08 UTC
Created attachment 1473241 [details]
Patch - fix update_negative_cache() map source usage (update 2)

Update CHANGELOG hunk to reflect difference between rev 83
(test revision) and current RHEL-7.6.

Comment 31 errata-xmlrpc 2018-10-30 11:41:29 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3283