1411412 – 7.3 upgrade of rpcbind breaks IPA, systemd, dbus

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1411412 - 7.3 upgrade of rpcbind breaks IPA, systemd, dbus

Summary: 7.3 upgrade of rpcbind breaks IPA, systemd, dbus

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	rpcbind
Sub Component:
Version:	7.3
Hardware:	All
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	rc
Target Release:	---
Assignee:	Steve Dickson
QA Contact:	Yongcheng Yang
Docs Contact:
URL:
Whiteboard:
Depends On:	1425758
Blocks:	1298243 1385242 1420851 1469559
TreeView+	depends on / blocked

Reported:	2017-01-09 16:26 UTC by Amy Farley
Modified:	2021-06-10 11:48 UTC (History)
CC List:	15 users (show)
Fixed In Version:	rpcbind-0.2.0-44.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2018-03-12 03:02:28 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Bugzilla	1293430	0	unspecified	CLOSED	Add localhost:111 to rpcbind socket activation	2021-02-22 00:41:40 UTC

Internal Links: 1293430

Description Amy Farley 2017-01-09 16:26:52 UTC

Description of problem:

7.3 update caused race condition between systemd and dbus. (no services could authenticate) However, downgrading rpcbind allowed all to function correctly.

"When I uninstalled the ipa services from the broken 7.3 server it starting working properly. Maybe there's a race issue being created on startup where ipa is pushing in front of a critical service and not allowing it to start? The hotfix was to run 'kill -SIGUSR1 1' and then manually start all of the services. Once that was done the server was functional."

upgrade from rpcbind-0.2.0-32.el7.x86_64   to rpcbind-0.2.0-38.el7.x86_64 is what broke the server.

To further verify running 'yum downgrade rpcbind-0.2.0-32.el7.x86_64' then reboot the server and it's working again.

Comment 1 Steve Dickson 2017-01-17 15:10:41 UTC

Are there any error messages explaining what is going on?

Comment 2 martbhell 2017-01-18 11:29:50 UTC

I have similar issues since 7.3 which can be solved by downgrading rpcbind, but I have no ipa rpms installed or in use. Not sure if I should open a new one or piggyback on this?

These actions fixes the issue for me:
 - /usr/lib/polkit-1/polkitd & systemctl restart dbus
 - upgrading systemd to the one from facebook backport COPR
 - downgrading rpcbind to -33 
 - setting [socket] in rpcbind.socket to (with rpcbind.socket.d/override.conf:

[Socket]
ListenStream=/var/run/rpcbind.sock
#ListenStream=[::]:111
#ListenStream=0.0.0.0:111
#BindIPv6Only=ipv6-only

I can easily reproduce this by undoing the rpcbind fix and restarting the server.

### Some error collection:

selinux is disabled

rpcbind.service and rpcbind.socket both start.

from systemctl status dbus:

Jan 13 09:54:28 io4 dbus-daemon[2297]: dbus[2297]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out
Jan 13 09:54:39 io4 dbus[2297]: [system] Activating systemd to hand-off: service name='org.freedesktop.login1' unit='dbus-org.freedesktop.login1.service'

from journalctl -xeu systemd-logind:

systemd-logind[2328]: Failed to abandon session scope: Transport endpoint is not connected

systemd-logind gets continuously restarted - it is in "running" in "systemctl list-jobs"

## polkit has these

Jan 18 12:54:49 io4 systemd[1]: Starting Authorization Manager...
Jan 18 12:55:39 io4 polkitd[2305]: Started polkitd version 0.112
Jan 18 12:55:39 io4 polkitd[2305]: Loading rules from directory /etc/polkit-1/rules.d
Jan 18 12:55:39 io4 polkitd[2305]: Loading rules from directory /usr/share/polkit-1/rules.d
Jan 18 12:55:39 io4 polkitd[2305]: Finished loading, compiling and executing 2 rules
Jan 18 12:55:39 io4 polkitd[2305]: Acquired the name org.freedesktop.PolicyKit1 on the system bus
Jan 18 12:56:19 io4 systemd[1]: polkit.service start operation timed out. Terminating.

- before running "/usr/lib/polkit-1/polkitd & systemctl restart dbus"

# busctl --no-pager
NAME                                   PID PROCESS         USER             CONNECTION    UNIT                      SESSION    DESCRIPTION        
:1.28                                 5298 systemd-logind  root             :1.28         systemd-logind.service    -          -                  
:1.30                                 5301 busctl          root             :1.30         sshd.service              -          -                  
:1.7                                  3065 ypbind          root             :1.7          ypbind.service            -          -                  
fi.epitest.hostap.WPASupplicant          - -               -                (activatable) -                         -         
fi.w1.wpa_supplicant1                    - -               -                (activatable) -                         -         
org.freedesktop.DBus                     - -               -                -             -                         -          -                  
org.freedesktop.PolicyKit1               - -               -                (activatable) -                         -         
org.freedesktop.hostname1                - -               -                (activatable) -                         -         
org.freedesktop.import1                  - -               -                (activatable) -                         -         
org.freedesktop.locale1                  - -               -                (activatable) -                         -         
org.freedesktop.login1                   - -               -                (activatable) -                         -         
org.freedesktop.machine1                 - -               -                (activatable) -                         -         
org.freedesktop.systemd1                 - -               -                (activatable) -                         -         
org.freedesktop.timedate1                - -               -                (activatable) -                         -         
org.gnome.GConf.Defaults                 - -               -                (activatable) -                         -         

And after:

# busctl --no-pager
NAME                                   PID PROCESS         USER             CONNECTION    UNIT                      SESSION    DESCRIPTION        
:1.0                                     1 systemd         root             :1.0          -                         -          -                  
:1.1                                  5570 systemd-logind  root             :1.1          systemd-logind.service    -          -                  
:1.2                                  5566 polkitd         polkitd          :1.2          polkit.service            -          -                  
:1.3                                  5567 tuned           root             :1.3          tuned.service             -          -                  
:1.5                                  5641 busctl          root             :1.5          sshd.service              -          -                  
com.redhat.tuned                      5567 tuned           root             :1.3          tuned.service             -          -                  
fi.epitest.hostap.WPASupplicant          - -               -                (activatable) -                         -         
fi.w1.wpa_supplicant1                    - -               -                (activatable) -                         -         
org.freedesktop.DBus                     - -               -                -             -                         -          -                  
org.freedesktop.PolicyKit1            5566 polkitd         polkitd          :1.2          polkit.service            -          -                  
org.freedesktop.hostname1                - -               -                (activatable) -                         -         
org.freedesktop.import1                  - -               -                (activatable) -                         -         
org.freedesktop.locale1                  - -               -                (activatable) -                         -         
org.freedesktop.login1                5570 systemd-logind  root             :1.1          systemd-logind.service    -          -                  
org.freedesktop.machine1                 - -               -                (activatable) -                         -         
org.freedesktop.systemd1                 1 systemd         root             :1.0          -                         -          -                  
org.freedesktop.timedate1                - -               -                (activatable) -                         -         
org.gnome.GConf.Defaults                 - -               -                (activatable) -                         -

Comment 3 martbhell 2017-01-18 12:55:48 UTC

Tested the systemd 231 facebook backport COPR and that doesn't help either.
With that the starting polkit & restart dbus fix no longer works but changing the rpcbind.socket does - like this in /etc/systemd/system/rpcbind.socket/override.conf helps:

# /etc/systemd/system/rpcbind.socket.d/rpc_override.conf
[Socket]
ListenStream=
ListenStream=/var/run/rpcbind.sock
BindIPv6Only=default

Comment 4 Joe B 2017-01-19 15:47:33 UTC

This resolved the issue, which is how the old version looked prior to patching

vi /usr/lib/systemd/system/rpcbind.socket

[Socket]
ListenStream=/var/run/rpcbind.sock

<deleted the following lines>
ListenStream=[::]:111
ListenStream=0.0.0.0:111
BindIPv6Only=ipv6-only
<>

It also explains why issue wasn't being replicated by the engineers troubleshooting the issue at redhat. I'm going to assume they are using ipv6 while we are still using ipv4 here.

Thanks Mart. Wish I would have seen your post first. I actually stumbled into /usr/lib/systemd/system/  trying to resolve another issue that is causing autofs to start too soon on our ipa servers. It was suggested here ' http://serverfault.com/questions/707256/centos-7-autofs-mount-doesnt-work-right-after-reboot ' to add "Requires=network.target rpc-statd.service rpcbind.service" but it's still requiring a manual restart after fresh boot.

Anyhow, hopefully the rpcbind team can work in some kind of ipv4 check during rpcbind.

Comment 7 Steve Dickson 2017-01-30 20:11:17 UTC

My apologies... but I just don't understand what the problem

How is removing ipv6 support from rpcbind allow ipa, systemd and dbus 
to work???

Comment 17 Christopher Tubbs 2017-06-06 20:47:56 UTC

I just got hit by this bug on a recently upgraded 7.3. *Very* problematic. Firefox wouldn't work right, evolution broke, etc. Hope this gets fixed soon. I don't want to have to fix this for all my colleagues when they upgrade.

Comment 22 Steve Dickson 2017-10-25 15:26:47 UTC

(In reply to Amy Farley from comment #0)
> Description of problem:
> 
> 7.3 update caused race condition between systemd and dbus. (no services
> could authenticate) However, downgrading rpcbind allowed all to function
> correctly.
> 
> "When I uninstalled the ipa services from the broken 7.3 server it starting
> working properly. Maybe there's a race issue being created on startup where
> ipa is pushing in front of a critical service and not allowing it to start?

So this is happening during start... Things hanging when systemd
brings up service correct?

I know there is a loop between dbus, ypbind and rpcbind. 

dbus needs to look up its user id. That lookup is done 
via nis. So an rpcbind query is done to get ypbind's port.
but I have no idea why the ipv6 listening sockets is
causing the hang.

Ipv6 is enabled, correct? Meaning its not being disabled
via the boot line or sysctl.

Comment 23 Steve Dickson 2018-01-03 16:35:53 UTC

Here is the problem (from Comment 114 of bug 1425758):

When the listening sockets are created via the
rpcbind.socket file 

ListenStream=[::]:111
ListenStream=0.0.0.0:111
BindIPv6Only=ipv6-only

The deadlock occurs

When the listening sockets are not create via
rpcbind.socket file (rpcbind creates them when
it is started) the deadlock does not happen.

Now there is one difference between the rpcbind.socket
in upstream and RHEL7. In upstream both the 
udp and tcp sockets are created. In RHEL7
only the tcp sockets are created. But I don't think
that will help this problem (I'll create another bz)

In summary, systemd deadlocks systemd->dbus->nis->rpcbind 
when the listening sockets are created by systemd.

When they are created by rpcbind, after it's started, 
the deadlock does not happen.

Comment 26 Yongcheng Yang 2018-03-12 03:02:28 UTC

This issue has been fixed in version rpcbind-0.2.0-44.el7 now.

Just close this one according to comment #24.

Note You need to log in before you can comment on or make changes to this bug.