Bug 2033552
| Summary: | Using -hosts option does not work after upgrading from 8.4 to 8.5 | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Rohan Sable <rsable> | ||||||
| Component: | autofs | Assignee: | Ian Kent <ikent> | ||||||
| Status: | CLOSED ERRATA | QA Contact: | Kun Wang <kunwan> | ||||||
| Severity: | high | Docs Contact: | |||||||
| Priority: | urgent | ||||||||
| Version: | 8.5 | CC: | ajmitchell, cmarshal, dwysocha, esandeen, fsorenso, ikent, jke, micmurph, rbergant, rsable, satsingh, wshaikh, xzhou | ||||||
| Target Milestone: | rc | Keywords: | CustomerScenariosInitiative, Regression, Triaged | ||||||
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | autofs-5.1.4-81.el8 | Doc Type: | If docs needed, set a value | ||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | |||||||||
| : | 2056320 (view as bug list) | Environment: | |||||||
| Last Closed: | 2022-05-10 15:29:06 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 2056320 | ||||||||
| Attachments: |
|
||||||||
|
Description
Rohan Sable
2021-12-17 08:38:43 UTC
There is a bug for this already. See https://bugzilla.redhat.com/show_bug.cgi?id=2024793. I'm tempted to make that bug a duplicate of this but the crash that was seen in that bug continues to worry me. I wasn't able to reproduce it because the NFS server (on a Fedora install I think) wouldn't start if there were duplicate export entries. So a workaround for that bug was to fix the server export list. But the report here looks a bit different. Can we get output of "showmount -e <server-name>" please. And a full debug log from starting autofs until after the errors occurs. To get that set "logging = debug" in /etc/autofs.conf and post the output. You can get that output by using "journalctl | grep automount" although you might want to restrict it by using a date range. It's simple enough to ignore duplicates and I can make a test package that does that to see if that really is all it is here too. As I say I haven't applied that change yet because of the observed crash but we have several reports here that apparently don't see a crash ... even more puzzling ... Umm ... so I've been able to duplicate the broken behavior. Not sure if it's is actually the case with the customer. All I did was remove some of the exported directories. When I did that I got the same permission errors in autofs (but not the broken directory listing, don't know). I then found that both rev 48 and rev 74 succeed in mounting the exports tree provided the last exported directory exists. If it doesn't exist rev 74 will fail. I can fix that. Possibly the best next step is to fix this and see if that's enough to fix the customer problem, it is certainly a regression from rev 48 so it needs to be fixed anyway. Could we check to see if this package resolves the problem please: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=42032234 Sent the RPM to a customer who reported this. Will let you know when he responds. Could we check to see if this package scratch build resolves the problem please: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=42312016 This build includes Frank's underrun patch (from bug 2036277). (In reply to Ian Kent from comment #31) > Could we check to see if this package scratch build resolves the problem > please: > https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=42312016 > > This build includes Frank's underrun patch (from bug 2036277). I've passed build to user to test and let you know the outcome, probably that underrun is corrected i dont reproduce it either with that build I managed to reproduce a use-after-free seen by valgrind in rpc_get_exports:
==1595297== Thread 5:
==1595297== Invalid read of size 4
==1595297== at 0x506F231: clnt_dg_call (clnt_dg.c:475)
==1595297== by 0xA024BEE: ??? (in /usr/lib64/autofs/lookup_hosts.so)
==1595297== by 0xA025F45: rpc_get_exports (in /usr/lib64/autofs/lookup_hosts.so)
==1595297== by 0xA021C83: ??? (in /usr/lib64/autofs/lookup_hosts.so)
==1595297== by 0xA022665: lookup_mount (in /usr/lib64/autofs/lookup_hosts.so)
==1595297== by 0x127EA0: do_lookup_mount (lookup.c:831)
==1595297== by 0x128BB0: lookup_nss_mount (lookup.c:1216)
==1595297== by 0x118DC4: do_mount_indirect (indirect.c:750)
==1595297== by 0x4E4A2DD: start_thread (pthread_create.c:486)
==1595297== by 0x6823132: clone (clone.S:95)
==1595297== Address 0x94e7dd0 is 16 bytes inside a block of size 348 free'd
==1595297== at 0x4C34A93: free (vg_replace_malloc.c:872)
==1595297== by 0x506F228: clnt_dg_call (clnt_dg.c:473)
==1595297== by 0xA024BEE: ??? (in /usr/lib64/autofs/lookup_hosts.so)
==1595297== by 0xA025F45: rpc_get_exports (in /usr/lib64/autofs/lookup_hosts.so)
==1595297== by 0xA021C83: ??? (in /usr/lib64/autofs/lookup_hosts.so)
==1595297== by 0xA022665: lookup_mount (in /usr/lib64/autofs/lookup_hosts.so)
==1595297== by 0x127EA0: do_lookup_mount (lookup.c:831)
==1595297== by 0x128BB0: lookup_nss_mount (lookup.c:1216)
==1595297== by 0x118DC4: do_mount_indirect (indirect.c:750)
==1595297== by 0x4E4A2DD: start_thread (pthread_create.c:486)
==1595297== by 0x6823132: clone (clone.S:95)
==1595297== Block was alloc'd at
==1595297== at 0x4C36E4B: calloc (vg_replace_malloc.c:1328)
==1595297== by 0x506F018: clnt_dg_call (clnt_dg.c:441)
==1595297== by 0xA024BEE: ??? (in /usr/lib64/autofs/lookup_hosts.so)
==1595297== by 0xA025F45: rpc_get_exports (in /usr/lib64/autofs/lookup_hosts.so)
==1595297== by 0xA021C83: ??? (in /usr/lib64/autofs/lookup_hosts.so)
==1595297== by 0xA022665: lookup_mount (in /usr/lib64/autofs/lookup_hosts.so)
==1595297== by 0x127EA0: do_lookup_mount (lookup.c:831)
==1595297== by 0x128BB0: lookup_nss_mount (lookup.c:1216)
==1595297== by 0x118DC4: do_mount_indirect (indirect.c:750)
==1595297== by 0x4E4A2DD: start_thread (pthread_create.c:486)
==1595297== by 0x6823132: clone (clone.S:95)
on the nfs server:
# rpcinfo -p | grep -w mountd
100005 1 udp 20048 mountd
100005 1 tcp 20048 mountd
100005 2 udp 20048 mountd
100005 2 tcp 20048 mountd
100005 3 udp 20048 mountd
100005 3 tcp 20048 mountd
# iptables -I INPUT 1 -p udp --dport 20048 -s NFS_CLIENT_IP -j REJECT --reject-with icmp-host-prohibited
then on the client:
# ls -l /net/NFS_SERVER
1 0.000000000 192.168.122.99 → 192.168.122.83 Portmap 126 V4 GETADDR Call udp udp.port == 38439 udp.port == 111
2 0.003582143 192.168.122.83 → 192.168.122.99 Portmap 90 V4 GETADDR Reply (Call In 1) udp udp.port == 111 udp.port == 38439
3 0.035151921 192.168.122.99 → 192.168.122.83 MOUNT 134 V3 EXPORT Call udp udp.port == 43116 udp.port == 20048
4 0.036146581 192.168.122.83 → 192.168.122.99 ICMP 162 Destination unreachable (Host administratively prohibited) udp udp.port == 43116 udp.port == 20048
so this looks to be in libtirpc
Created attachment 1850664 [details]
patch to fix libtirpc use-after-free
I believe this patch fixes a use-after-free in libtirpc that autofs will hit when an attempt to get the exports from a system gets blocked with 'Host administratively prohibited'.
libtirpc is maintained on the linux-nfs list, right?
(In reply to Frank Sorenson from comment #40) > I managed to reproduce a use-after-free seen by valgrind in rpc_get_exports: > > > ==1595297== Thread 5: > ==1595297== Invalid read of size 4 > ==1595297== at 0x506F231: clnt_dg_call (clnt_dg.c:475) > ==1595297== by 0xA024BEE: ??? (in /usr/lib64/autofs/lookup_hosts.so) > ==1595297== by 0xA025F45: rpc_get_exports (in > /usr/lib64/autofs/lookup_hosts.so) > ==1595297== by 0xA021C83: ??? (in /usr/lib64/autofs/lookup_hosts.so) > ==1595297== by 0xA022665: lookup_mount (in > /usr/lib64/autofs/lookup_hosts.so) > ==1595297== by 0x127EA0: do_lookup_mount (lookup.c:831) > ==1595297== by 0x128BB0: lookup_nss_mount (lookup.c:1216) > ==1595297== by 0x118DC4: do_mount_indirect (indirect.c:750) > ==1595297== by 0x4E4A2DD: start_thread (pthread_create.c:486) > ==1595297== by 0x6823132: clone (clone.S:95) > ==1595297== Address 0x94e7dd0 is 16 bytes inside a block of size 348 free'd > ==1595297== at 0x4C34A93: free (vg_replace_malloc.c:872) > ==1595297== by 0x506F228: clnt_dg_call (clnt_dg.c:473) > ==1595297== by 0xA024BEE: ??? (in /usr/lib64/autofs/lookup_hosts.so) > ==1595297== by 0xA025F45: rpc_get_exports (in > /usr/lib64/autofs/lookup_hosts.so) > ==1595297== by 0xA021C83: ??? (in /usr/lib64/autofs/lookup_hosts.so) > ==1595297== by 0xA022665: lookup_mount (in > /usr/lib64/autofs/lookup_hosts.so) > ==1595297== by 0x127EA0: do_lookup_mount (lookup.c:831) > ==1595297== by 0x128BB0: lookup_nss_mount (lookup.c:1216) > ==1595297== by 0x118DC4: do_mount_indirect (indirect.c:750) > ==1595297== by 0x4E4A2DD: start_thread (pthread_create.c:486) > ==1595297== by 0x6823132: clone (clone.S:95) > ==1595297== Block was alloc'd at > ==1595297== at 0x4C36E4B: calloc (vg_replace_malloc.c:1328) > ==1595297== by 0x506F018: clnt_dg_call (clnt_dg.c:441) > ==1595297== by 0xA024BEE: ??? (in /usr/lib64/autofs/lookup_hosts.so) > ==1595297== by 0xA025F45: rpc_get_exports (in > /usr/lib64/autofs/lookup_hosts.so) > ==1595297== by 0xA021C83: ??? (in /usr/lib64/autofs/lookup_hosts.so) > ==1595297== by 0xA022665: lookup_mount (in > /usr/lib64/autofs/lookup_hosts.so) > ==1595297== by 0x127EA0: do_lookup_mount (lookup.c:831) > ==1595297== by 0x128BB0: lookup_nss_mount (lookup.c:1216) > ==1595297== by 0x118DC4: do_mount_indirect (indirect.c:750) > ==1595297== by 0x4E4A2DD: start_thread (pthread_create.c:486) > ==1595297== by 0x6823132: clone (clone.S:95) > > > on the nfs server: > # rpcinfo -p | grep -w mountd > 100005 1 udp 20048 mountd > 100005 1 tcp 20048 mountd > 100005 2 udp 20048 mountd > 100005 2 tcp 20048 mountd > 100005 3 udp 20048 mountd > 100005 3 tcp 20048 mountd > # iptables -I INPUT 1 -p udp --dport 20048 -s NFS_CLIENT_IP -j REJECT > --reject-with icmp-host-prohibited > > then on the client: > # ls -l /net/NFS_SERVER > > 1 0.000000000 192.168.122.99 → 192.168.122.83 Portmap 126 V4 GETADDR > Call udp udp.port == 38439 udp.port == 111 > 2 0.003582143 192.168.122.83 → 192.168.122.99 Portmap 90 V4 GETADDR > Reply (Call In 1) udp udp.port == 111 udp.port == 38439 > 3 0.035151921 192.168.122.99 → 192.168.122.83 MOUNT 134 V3 EXPORT Call > udp udp.port == 43116 udp.port == 20048 > 4 0.036146581 192.168.122.83 → 192.168.122.99 ICMP 162 Destination > unreachable (Host administratively prohibited) udp udp.port == 43116 > udp.port == 20048 > > so this looks to be in libtirpc But using the hack to read from a file reproduces the problem? That's not using libtirpc, right. I have been able to reproduce it using that but am having trouble tracking it down. (In reply to Ian Kent from comment #42) > (In reply to Frank Sorenson from comment #40) > > so this looks to be in libtirpc > > But using the hack to read from a file reproduces the problem? No, the hack to read from a file reproduces this use-after-free: ==1510991== Invalid read of size 1 ==1510991== at 0x4C39114: strcmp (vg_replace_strmem.c:924) ==1510991== by 0x12CAC2: cache_delete (cache.c:781) ==1510991== by 0x12DABE: tree_mapent_delete_offset_tree (mounts.c:1674) ==1510991== by 0x12DA5A: tree_mapent_delete_offset_tree (mounts.c:1652) ==1510991== by 0x12DA76: tree_mapent_delete_offset_tree (mounts.c:1658) ==1510991== by 0x12DA5A: tree_mapent_delete_offset_tree (mounts.c:1652) ==1510991== by 0x12DA76: tree_mapent_delete_offset_tree (mounts.c:1658) ==1510991== by 0x12DA76: tree_mapent_delete_offset_tree (mounts.c:1658) ==1510991== by 0x12DA5A: tree_mapent_delete_offset_tree (mounts.c:1652) ==1510991== by 0x12DA5A: tree_mapent_delete_offset_tree (mounts.c:1652) ==1510991== by 0x12DA5A: tree_mapent_delete_offset_tree (mounts.c:1652) ==1510991== by 0x12DA76: tree_mapent_delete_offset_tree (mounts.c:1658) ==1510991== Address 0x946b900 is 0 bytes inside a block of size 32 free'd ==1510991== at 0x4C34A93: free (vg_replace_malloc.c:872) ==1510991== by 0x12CB42: cache_delete (cache.c:762) ==1510991== by 0x12DABE: tree_mapent_delete_offset_tree (mounts.c:1674) ==1510991== by 0x12DA5A: tree_mapent_delete_offset_tree (mounts.c:1652) ==1510991== by 0x12DA76: tree_mapent_delete_offset_tree (mounts.c:1658) ==1510991== by 0x12DA5A: tree_mapent_delete_offset_tree (mounts.c:1652) ==1510991== by 0x12DA76: tree_mapent_delete_offset_tree (mounts.c:1658) ==1510991== by 0x12DA76: tree_mapent_delete_offset_tree (mounts.c:1658) ==1510991== by 0x12DA5A: tree_mapent_delete_offset_tree (mounts.c:1652) ==1510991== by 0x12DA5A: tree_mapent_delete_offset_tree (mounts.c:1652) ==1510991== by 0x12DA5A: tree_mapent_delete_offset_tree (mounts.c:1652) ==1510991== by 0x12DA76: tree_mapent_delete_offset_tree (mounts.c:1658) ==1510991== Block was alloc'd at ==1510991== at 0x4C32135: malloc (vg_replace_malloc.c:381) ==1510991== by 0x12C34C: cache_add (cache.c:533) ==1510991== by 0x12C639: cache_update (cache.c:709) ==1510991== by 0x12C814: cache_update_offset (cache.c:626) ==1510991== by 0xA24AFBF: update_offset_entry (parse_sun.c:863) ==1510991== by 0xA24AFBF: parse_mount (parse_sun.c:1520) ==1510991== by 0xA02353D: do_parse_mount (lookup_hosts.c:188) ==1510991== by 0xA02353D: lookup_mount (lookup_hosts.c:425) ==1510991== by 0x127EA0: do_lookup_mount (lookup.c:831) ==1510991== by 0x128BB0: lookup_nss_mount (lookup.c:1216) ==1510991== by 0x118DC4: do_mount_indirect (indirect.c:750) ==1510991== by 0x4E4A2DD: start_thread (pthread_create.c:486) ==1510991== by 0x6823132: clone (clone.S:95) > That's not using libtirpc, right. > > I have been able to reproduce it using that but am having trouble > tracking it down. The libtirpc use-after-free is reproduced by blocking the connection to mountd. The clnt_dg_call error handling reads the error number after it frees the memory that contains the error: for (cmsg = CMSG_FIRSTHDR (&msg); cmsg; cmsg = CMSG_NXTHDR (&msg, cmsg)) if (cmsg->cmsg_level == SOL_IP && cmsg->cmsg_type == IP_RECVERR) { mem_free(cbuf, (outlen + 256)); <<<<<<<< 'cbuf' freed e = (struct sock_extended_err *) CMSG_DATA(cmsg); cu->cu_error.re_errno = e->ee_errno; <<<<<<<< 'e' is a pointer to a location within 'cbuf' that was just freed release_fd_lock(cu->cu_fd_lock, mask); return (cu->cu_error.re_status = RPC_CANTRECV); } mem_free(cbuf, (outlen + 256)); (In reply to Ian Kent from comment #42) > (In reply to Frank Sorenson from comment #40) > I have been able to reproduce it using that but am having trouble > tracking it down. replication of this one requires a hash collision, and since the hash is dependent on the full path, hitting the bug has been inconsistent. I managed to pare down an exports list to 3 entries when accessing host 'nfs2' from path '/net' (I'll attach the ugly hack EXPORT hack file) the use-after-free in cache_delete appears to occur because tree_mapent_delete_offset_tree calls cache_delete() using the key that is at me->key, and 'me' itself is then located and deleted within cache_delete(); after freeing the string, cache_delete() continues to reference 'key' ret = cache_delete(me->mc, me->key); a complete hack that 'fixes' it: char *foo = strdup(me->key); ret = cache_delete(me->mc, foo); free(foo); (but don't do this at home) (In reply to Frank Sorenson from comment #44) > (In reply to Ian Kent from comment #42) > > (In reply to Frank Sorenson from comment #40) > > > I have been able to reproduce it using that but am having trouble > > tracking it down. > > replication of this one requires a hash collision, and since the hash is > dependent on the full path, hitting the bug has been inconsistent. > > I managed to pare down an exports list to 3 entries when accessing host > 'nfs2' from path '/net' (I'll attach the ugly hack EXPORT hack file) > > > the use-after-free in cache_delete appears to occur because > tree_mapent_delete_offset_tree calls cache_delete() using the key that is at > me->key, and 'me' itself is then located and deleted within cache_delete(); > after freeing the string, cache_delete() continues to reference 'key' Yes, I got the same impression and I have been trying to verify that is happening but I can't get any evidence that's what is really happening, it's very strange. Ian (In reply to Ian Kent from comment #45) > (In reply to Frank Sorenson from comment #44) > > (In reply to Ian Kent from comment #42) > > > (In reply to Frank Sorenson from comment #40) > > > > > I have been able to reproduce it using that but am having trouble > > > tracking it down. > > > > replication of this one requires a hash collision, and since the hash is > > dependent on the full path, hitting the bug has been inconsistent. > > > > I managed to pare down an exports list to 3 entries when accessing host > > 'nfs2' from path '/net' (I'll attach the ugly hack EXPORT hack file) > > > > > > the use-after-free in cache_delete appears to occur because > > tree_mapent_delete_offset_tree calls cache_delete() using the key that is at > > me->key, and 'me' itself is then located and deleted within cache_delete(); > > after freeing the string, cache_delete() continues to reference 'key' > > Yes, I got the same impression and I have been trying to verify that is > happening but I can't get any evidence that's what is really happening, > it's very strange. Oooh, wait! Your saying it's the passed in key parameter that's being used after free. Mmm ... that's kind off an obvious cause to miss ... oops. Could we check to see if this package scratch build resolves the problem please: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=42450990 Lets see if that fixes the crash. (In reply to Ian Kent from comment #47) > Could we check to see if this package scratch build resolves the problem > please: > https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=42450990 > > Lets see if that fixes the crash. This looks like it fixes it to me. We'll get the test package out to affected customers, and push for some feedback. Thanks! (In reply to Ian Kent from comment #47) > Could we check to see if this package scratch build resolves the problem > please: > https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=42450990 > > Lets see if that fixes the crash. I've passed it, keep you posted on feedback I have another test build: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=42539848 Fixes (should fix) a hang on direct/offset mount fail. Created attachment 1857475 [details]
fake mountd server for reproducer
this program will allow reproducing the automount segfault that occurs when mountd returns the export list in a different order when requested over tcp vs udp.
This will take the place of the local rpc.mountd program (just kill the process), and will register both tcp and udp port 20048 with rpcbind.
* only EXPORT calls receive a response
* returns an error ("remote can't support version") when called with program version 2
* returns fixed list of 2 paths, reversing the order over tcp vs udp
* has a fixed 900 hosts per path
* the response is 40 KiB, and therefore too large for udp
when autofs /net/hostname is accessed, autofs will request the export list over udp with program version 3, then 2, then 1. Versions 3 and 1 will be too large, and version 2 will also have an error, so autofs will then request the export list using tcp and program version 3. Because the list obtained over tcp is in the reverse order to the list over udp, the bug in autofs is exercised. and automount segfaults.
The recent patch "autofs-5.1.8 - fix memory leak in xdr_exports()" from Ian resolves this bug.
Do we have an ETA for the release of this fix at this point? Can we get a status on this bug? (In reply to Chuck Marshall from comment #82) > Can we get a status on this bug? You can read the status straight from the bug. The Fixed-in-Version field is set which means the needed changes have been made. The Bug itself has VERIFIED status which means the changes have been tested by QE and appear ok. At this point the bug remains with QE through to release. So what is it that you want to know? Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (autofs bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:2084 |