Bug 1402961

Summary: rpcbind.socket systemd unit fails to start when IPv6 is disabled.
Product: Red Hat Enterprise Linux 7 Reporter: Matthew LeSieur <matthew.lesieur>
Component: systemdAssignee: systemd-maint
Status: CLOSED NOTABUG QA Contact: qe-baseos-daemons
Severity: high Docs Contact:
Priority: high    
Version: 7.3CC: aparsons, arogozin, guillaume.pavese, iana, jakubr, jsynacek, kcleveng, kwalker, matthew.fansher, phil, pjmead, riehecky, rmetrich, rs, systemd-maint-list, tparsons, troels
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-07-07 08:24:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1420851    

Description Matthew LeSieur 2016-12-08 18:41:49 UTC
Description of problem:
The systemd unit rpcbind.socket fails to start when IPv6 is disabled on RHEL 7.3.  Previous version of RHEL 7 did not have this problem.  Kickstart installations failed after release of RHEL 7.3.

systemd[1]: rpcbind.socket failed to listen on sockets: Address family not supported by protocol

Version-Release number of selected component (if applicable):
systemd-219-30.el7_3.6.x86_64
rpcbind-0.2.0-38.el7.x86_64

-os-release-
NAME="Red Hat Enterprise Linux Server"
VERSION="7.3 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.3"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.3 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.3:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.3
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.3"

How reproducible:
Reproducible by installing a fresh RHEL 7.3 system, then disabling IPv6 using net.ipv6.conf.all.disable_ipv6 = 1.

Steps to Reproduce:
Install @core
# yum -y upgrade
# yum -y install @base
# systemctl status rpcbind.socket
● rpcbind.socket - RPCbind Server Activation Socket
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
   Active: active (listening) since Tue 2016-12-06 16:00:15 EST; 38s ago
   Listen: /var/run/rpcbind.sock (Stream)
           [::]:111 (Stream)
           0.0.0.0:111 (Stream)

Dec 06 16:00:15 rhel7.example.com systemd[1]: Listening on RPCbind Server Activation Socket.
Dec 06 16:00:15 rhel7.example.com systemd[1]: Starting RPCbind Server Activation Socket.
# echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.conf
# reboot

# systemctl status rpcbind.socket
● rpcbind.socket - RPCbind Server Activation Socket
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
   Active: failed (Result: resources)
   Listen: /var/run/rpcbind.sock (Stream)
           [::]:111 (Stream)
           0.0.0.0:111 (Stream)

Dec 06 16:02:51 rhel7.example.com systemd[1]: rpcbind.socket failed to listen on sockets: Address family not supported by protocol
Dec 06 16:02:51 rhel7.example.com systemd[1]: Failed to listen on RPCbind Server Activation Socket.
Dec 06 16:02:51 rhel7.example.com systemd[1]: Unit rpcbind.socket entered failed state.
Dec 06 16:02:51 rhel7.example.com systemd[1]: Starting RPCbind Server Activation Socket.
# mkdir /etc/systemd/system/rpcbind.socket.d/
# cat << __EOF__ > /etc/systemd/system/rpcbind.socket.d/override.conf
> [Socket]
> ListenStream=
> ListenStream=/var/run/rpcbind.sock
> ListenStream=0.0.0.0:111
> __EOF__
# reboot

# systemctl status rpcbind.socket
● rpcbind.socket - RPCbind Server Activation Socket
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/rpcbind.socket.d
           └─override.conf
   Active: active (listening) since Tue 2016-12-06 16:08:01 EST; 1min 11s ago
   Listen: /var/run/rpcbind.sock (Stream)
           0.0.0.0:111 (Stream)

Dec 06 16:08:02 rhel7.example.com systemd[1]: Listening on RPCbind Server Activation Socket.
Dec 06 16:08:02 rhel7.example.com systemd[1]: Starting RPCbind Server Activation Socket.
# rpm -q rpcbind
rpcbind-0.2.0-38.el7.x86_64
# rpm -q systemd
systemd-219-30.el7_3.6.x86_64

Actual results:
Systemd fails to listen for connections to rpcbind sockets.

Expected results:
Systemd should listen for connections to rpcbind sockets, and start rpcbind.service when a request is received on a socket.

Additional info:
This behavior is mentioned in Bug ID 1377055 and 1370241, but resolutions don't include actually addressing the behavior.

Comment 1 Lukáš Nykrýn 2016-12-09 07:22:05 UTC
The problem is that if you disable ipv6 n sysctl.conf, it is not disabled from the beginning of the boot, but actually quite late in the boot process. Meanwhile systemd found out that ipv6 is enabled and scheduled a start of a socket, that would listen on ipv6.

There are two solution to this problem. Either disable the ipv6 from kernel command line, or after you edit the sysctl.conf call dracut -f, so your change also gets into the initramfs.

Comment 3 Matthew LeSieur 2016-12-09 19:34:36 UTC
Your explanation lines-up with my experience.  Existing RHEL installations, before the change to rpcbind.socket, worked fine since the installation of new Kernels automatically executes dracut, so the sysctl setting to disable IPv6 was already in the initramfs image when rpcbind.socket was updated.

I modified our kickstart configuration to include the following lines so the sysctl setting to disable IPv6 makes it into the initramfs image on first-boot.

echo "net.ipv6.conf.all.disable_ipv6 = 1" >> /etc/sysctl.d/ipv6.conf
#raw
for kver in $(rpm --qf '%{VERSION}\-%{RELEASE}.%{ARCH}\n' -q kernel); do dracut -f --kver $kver; done
#endraw

Comment 4 Robert Story 2016-12-26 22:56:02 UTC
I've just been bitten by this as well, after updating to 7.3.

Comment 5 Pete Mead 2016-12-29 14:34:07 UTC
I also have had this problem at RHEL7.3 after removing IPV6 support!

I added the dracut -f and seemed to fix the issue:-

# Disable IPv6
cat <<EOF > /etc/sysctl.d/98-disable-ipv6.conf
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
EOF
sysctl -p /etc/sysctl.d/98-disable-ipv6.conf
dracut -f

Comment 6 Jakub Rączkowski 2017-02-15 16:05:13 UTC
This is also happens on the current version of RHEL Atomic Host with:
net.ipv6.conf.all.disable_ipv6 = 1

Modifying the initramfs seems troublesome with OSTree.

Comment 7 Aleksandr Rogozin 2017-03-24 13:36:36 UTC
Make sure ipv6 is disabled in sysctl.conf: 
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

Then: 

# sudo systemctl enable rpcbind.socket
# sudo systemctl restart rpcbind.socket
# sudo mount -a

Comment 9 Ian Allison 2017-06-15 00:50:41 UTC
This just bit us as well. We're making those updates inside the %post section of a kickstart so had to be a bit careful about calling dracut, because we want to update the installed kernel(s), not the running one. In the end we went with something like

KERNELS=$(rpm -q kernel --qf '%{version}-%{release}.%{arch}\n')
for kernel_version in ${KERNELS} ; do
  dracut -f /boot/initramfs-${kernel_version}.img ${kernel_version}
done

Comment 10 Jan Synacek 2017-07-07 08:24:37 UTC
As suggested in comment 1, this is a configuration problem.

Comment 11 Pat Riehecky 2017-09-26 15:17:50 UTC
I'm a bit surprised this is NOTABUG, as systemd-sysctl.service runs before sysinit.target.  I'd be inclined to believe any of the network socket bits should run their interface logic after sysinit.target and systemd-modules-load.service.

Comment 12 Phil Dumont 2019-09-05 19:21:29 UTC
I found that, initially, doing the dracut and rebooting did not help.  rpcbind still failed on boot.

When I did "lsinitrd", I found that /etc/sysctl.conf was not in there.  Since it's supposed to be the ipv6 setting in sysctl.conf that is what's supposed to fix the problem, I found this suspicious.

When I added "install_items+=/etc/sysctl.conf" to /etc/dracut.conf and then did another dracut and reboot, all was well.

I'm on RHEL 7.7

Comment 13 Renaud Métrich 2019-09-05 19:58:27 UTC
Hi Phil,

Probably your initramfs is a "generic" initramfs, not "host-specific".
Please verify that you don't have dracut-config-generic package installed, this package forces building a generic image.

Comment 14 Phil Dumont 2019-09-05 20:06:43 UTC
Renaud Métrich,

You are quite right.  I had dracut-config-generic installed.

I uninstalled it (no dependencies needed to be removed), removed my "/etc/sysctl.conf" addition from /etc/dracut.conf, invoked dracut, and lsinitrd showed that /etc/sysctl.conf was there.

I'm not sure how or why dracut-config-generic got installed.  I will look into it...

Thanks.

Comment 15 Renaud Métrich 2019-09-06 07:43:11 UTC
Hi Phil,

I updated the KCS we have (https://access.redhat.com/solutions/2798411) and linked it to this BZ.

Renaud.

Comment 16 Phil Dumont 2019-09-06 13:19:26 UTC
Thanks, Renaud.

I did run into that KCS when I was searching the web for a resolution to my problem.

I'd like to point out that Jacob Shivers' comment that "It is not required to use the -H flag to incorporate the changes into the initramfs" had me going for a while.  The comment is correct *if* you happened to notice the instruction earlier in the KCS to verify that dracut-config-generic was not installed.  I had not noticed that instruction.  It might have been nice if Jacob emphasized that point in his comment.  (But, then again, it might be nice if I read instructions more thoroughly.  :-)