Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Multiple dhclient instances for the same interface are spawned when network.service is started after another systemd service has performed ifup on the same interface
DescriptionAndrea Segalini
2022-07-28 13:03:39 UTC
In systems using network scripts (network.service), when generic systemd services with selinux context "system_u:system_r:unconfined_service_t" run /usr/sbin/ifup to bring up an interface set via dhcp, the dhclient that is spawned runs with selinux context "system_u:system_r:unconfined_service_t". That is different from the context that dhclient acquires if executed by /etc/init.d/network (system_u:system_r:dhcpc_t) or by the context dhclient acquires when spawn by a root user running /usr/sbin/ifup from the terminal:
# dhclient spwaned from generic systemd service via ifup:
$ ps auxZ | grep dhclient
system_u:system_r:unconfined_service_t:s0 root 26827 1 0 25751 3028 2 10:36 ? 00:00:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3
# dhclient spwaned from /etc/init.d/network:
$ ps auxZ | grep dhclient
unconfined_u:system_r:dhcpc_t:s0-s0:c0.c1023 root 29114 1 0 25751 3032 2 10:38 ? 00:00:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3
If we then run "/etc/init.d/network start" after the systemd service has performed /usr/sbin/ifup on a dhcp interface, /etc/init.d/network spwans additional dhclient instances for the same interface, which pile up in the host:
# After running a systemd service that does /usr/sbin/ifup on eth3, we run /etc/init.d/network start
$ ps auxZ | grep dhclient
system_u:system_r:unconfined_service_t:s0 root 17084 0.0 0.0 103004 2980 ? Ss 07:55 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3
system_u:system_r:dhcpc_t:s0 root 17464 0.0 0.0 103004 4160 ? Ss 07:55 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3
system_u:system_r:dhcpc_t:s0 root 18187 0.0 0.0 103004 4036 ? Ss 07:55 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 18265 0.0 0.0 112824 2248 pts/0 S+ 07:55 0:00 grep --color=auto dhclient
dhclient shall notice that the interface is up and already configured, and that another dhclient is already running.
---
By digging further, the root cause seems due to dhclient spawned by "/etc/init.d/network" being unable to replace the PID file in /var/run/dhclient-eth3.pid. This causes dhclient to fail to recognise that another dhclient instance is running, and ultimately causes several other instances to pile up. In particular, this seems due to the wrong selinux context that dhclient has when run from a generic systemd service, versus the selinux context it has when run from "/etc/init.d/network". Notice the error message in network.service logs when interface are brought up:
Jul 25 16:30:24 timberlake-4 network[18988]: Bringing up interface eth0: [ OK ]
Jul 25 16:30:24 timberlake-4 network[18988]: Bringing up interface eth1: [ OK ]
Jul 25 16:30:25 timberlake-4 network[18988]: Bringing up interface eth2: [ OK ]
Jul 25 16:30:25 timberlake-4 network[18988]: Bringing up interface eth3:
Jul 25 16:30:25 timberlake-4 dhclient[20024]: Can't create /var/run/dhclient-eth3.pid: Permission denied
Jul 25 16:30:25 timberlake-4 network[18988]: Determining IP information for eth3...Can't create /var/run/dhclient-eth3.pid: Permission denied
Jul 25 16:30:25 timberlake-4 dhclient[20024]: DHCPREQUEST on eth3 to 255.255.255.255 port 67 (xid=0x26dd3b44)
Jul 25 16:30:25 timberlake-4 dhclient[20024]: DHCPACK from 10.53.96.1 (xid=0x26dd3b44)
Jul 25 16:30:27 timberlake-4 dhclient[20024]: bound to 10.53.102.56 -- renewal in 1404 seconds.
Jul 25 16:30:27 timberlake-4 dhclient[20119]: Can't create /var/run/dhclient-eth3.pid: Permission denied
Jul 25 16:30:27 timberlake-4 network[18988]: done.
Jul 25 16:30:27 timberlake-4 network[18988]: [ OK ]
Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists
If we repeat the same experiment, but this time performing "restorecon -R /var/run/" after the systemd service runs /usr/sbin/ifup (to fix the labelling of the PID file /var/run/dhclient-eth3.pid), we notice that dhclient spawned from "/etc/init.d/network start" can tell that the interface is up and configured, and thus refrains from running again:
Determining IP information for eth3...dhclient(22807) is already running - exiting
Version-Release number of selected component (if applicable):
[root@localhost nutanix]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)
[root@localhost nutanix]# rpm -qa | grep selinux
selinux-policy-targeted-3.13.1-268.el7.noarch
libselinux-python-2.5-15.el7.x86_64
selinux-policy-3.13.1-268.el7.noarch
libselinux-2.5-15.el7.x86_64
libselinux-utils-2.5-15.el7.x86_64
How reproducible: Every time
Steps to Reproduce:
1. Configure an interface to get IP address via dhcp. E.g,: /etc/sysconfig/network-scripts/ifcfg-eth1:
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="dhcp"
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="eth1"
DEVICE="eth1"
ONBOOT="yes"
NM_CONTROLLED=no
2. Create a systemd service that brings up eth1 via ifup. E.g.:
[nutanix@localhost ~]$ cat /usr/example.py
#! /usr/bin/env python
import subprocess
def run(cmd_tokens):
subprocess.call(cmd_tokens)
if __name__ == '__main__':
# We first stop network.service, so we can bring up interface again.
run(["service", "network", "stop"])
# We manually bring up eth1.
run(["/sbin/ifdown", "eth1"])
run(["/sbin/ifup", "eth1"])
# We use network.service to bring-up the *already up* eth1
run(["service", "network", "start"])
[nutanix@localhost ~]$ systemctl cat example.service
# /etc/systemd/system/example.service
[Unit]
Description=Example
After=network.target
StartLimitIntervalSec=0
[Service]
Type=oneshot
ExecStart=/usr/example.py
[Install]
WantedBy=multi-user.target
[Install]
WantedBy=multi-user.target
3. Start the systemd service example.service
$ systemctl start example.service
4. Check "ps auxZ | grep dhclient" and "ls -lhZ /var/run" and notice the multiple instances of dhclient
Actual results:
$ ps auxZ | grep dhclient ; ls -lhZ /var/run/ | grep dhclient
system_u:system_r:unconfined_service_t:s0 root 17084 0.0 0.0 103004 2980 ? Ss 07:55 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3
system_u:system_r:dhcpc_t:s0 root 17464 0.0 0.0 103004 4160 ? Ss 07:55 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3
system_u:system_r:dhcpc_t:s0 root 18187 0.0 0.0 103004 4036 ? Ss 07:55 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 18265 0.0 0.0 112824 2248 pts/0 S+ 07:55 0:00 grep --color=auto dhclient
Expected results:
ps auxZ | grep dhcli ; ls -lhZ /var/run/ | grep dhcl ; systemctl status network
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 6941 0.0 0.0 112824 2280 pts/1 S+ 16:35 0:00 grep --color=auto dhcli
system_u:system_r:dhcpc_t:s0 root 20734 0.0 0.0 103004 2980 ? Ss 16:31 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H seabass-3 eth3
-rw-r--r--. root root system_u:object_r:dhcpc_var_run_t:s0 dhclient-eth3.pid
Switching the component for assessing the described scenario: Is it supported/recommended to start devices in 2 different ways on the same system?
Comment 5RHEL Program Management
2023-09-20 20:24:28 UTC
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.
Comment 6RHEL Program Management
2023-09-20 20:57:01 UTC
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.
Due to differences in account names between systems, some fields were not replicated. Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.
To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer. You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:
"Bugzilla Bug" = 1234567
In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.
In systems using network scripts (network.service), when generic systemd services with selinux context "system_u:system_r:unconfined_service_t" run /usr/sbin/ifup to bring up an interface set via dhcp, the dhclient that is spawned runs with selinux context "system_u:system_r:unconfined_service_t". That is different from the context that dhclient acquires if executed by /etc/init.d/network (system_u:system_r:dhcpc_t) or by the context dhclient acquires when spawn by a root user running /usr/sbin/ifup from the terminal: # dhclient spwaned from generic systemd service via ifup: $ ps auxZ | grep dhclient system_u:system_r:unconfined_service_t:s0 root 26827 1 0 25751 3028 2 10:36 ? 00:00:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3 # dhclient spwaned from /etc/init.d/network: $ ps auxZ | grep dhclient unconfined_u:system_r:dhcpc_t:s0-s0:c0.c1023 root 29114 1 0 25751 3032 2 10:38 ? 00:00:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3 If we then run "/etc/init.d/network start" after the systemd service has performed /usr/sbin/ifup on a dhcp interface, /etc/init.d/network spwans additional dhclient instances for the same interface, which pile up in the host: # After running a systemd service that does /usr/sbin/ifup on eth3, we run /etc/init.d/network start $ ps auxZ | grep dhclient system_u:system_r:unconfined_service_t:s0 root 17084 0.0 0.0 103004 2980 ? Ss 07:55 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3 system_u:system_r:dhcpc_t:s0 root 17464 0.0 0.0 103004 4160 ? Ss 07:55 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3 system_u:system_r:dhcpc_t:s0 root 18187 0.0 0.0 103004 4036 ? Ss 07:55 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3 unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 18265 0.0 0.0 112824 2248 pts/0 S+ 07:55 0:00 grep --color=auto dhclient dhclient shall notice that the interface is up and already configured, and that another dhclient is already running. --- By digging further, the root cause seems due to dhclient spawned by "/etc/init.d/network" being unable to replace the PID file in /var/run/dhclient-eth3.pid. This causes dhclient to fail to recognise that another dhclient instance is running, and ultimately causes several other instances to pile up. In particular, this seems due to the wrong selinux context that dhclient has when run from a generic systemd service, versus the selinux context it has when run from "/etc/init.d/network". Notice the error message in network.service logs when interface are brought up: Jul 25 16:30:24 timberlake-4 network[18988]: Bringing up interface eth0: [ OK ] Jul 25 16:30:24 timberlake-4 network[18988]: Bringing up interface eth1: [ OK ] Jul 25 16:30:25 timberlake-4 network[18988]: Bringing up interface eth2: [ OK ] Jul 25 16:30:25 timberlake-4 network[18988]: Bringing up interface eth3: Jul 25 16:30:25 timberlake-4 dhclient[20024]: Can't create /var/run/dhclient-eth3.pid: Permission denied Jul 25 16:30:25 timberlake-4 network[18988]: Determining IP information for eth3...Can't create /var/run/dhclient-eth3.pid: Permission denied Jul 25 16:30:25 timberlake-4 dhclient[20024]: DHCPREQUEST on eth3 to 255.255.255.255 port 67 (xid=0x26dd3b44) Jul 25 16:30:25 timberlake-4 dhclient[20024]: DHCPACK from 10.53.96.1 (xid=0x26dd3b44) Jul 25 16:30:27 timberlake-4 dhclient[20024]: bound to 10.53.102.56 -- renewal in 1404 seconds. Jul 25 16:30:27 timberlake-4 dhclient[20119]: Can't create /var/run/dhclient-eth3.pid: Permission denied Jul 25 16:30:27 timberlake-4 network[18988]: done. Jul 25 16:30:27 timberlake-4 network[18988]: [ OK ] Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists Jul 25 16:30:27 timberlake-4 network[18988]: RTNETLINK answers: File exists If we repeat the same experiment, but this time performing "restorecon -R /var/run/" after the systemd service runs /usr/sbin/ifup (to fix the labelling of the PID file /var/run/dhclient-eth3.pid), we notice that dhclient spawned from "/etc/init.d/network start" can tell that the interface is up and configured, and thus refrains from running again: Determining IP information for eth3...dhclient(22807) is already running - exiting Version-Release number of selected component (if applicable): [root@localhost nutanix]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.9 (Maipo) [root@localhost nutanix]# rpm -qa | grep selinux selinux-policy-targeted-3.13.1-268.el7.noarch libselinux-python-2.5-15.el7.x86_64 selinux-policy-3.13.1-268.el7.noarch libselinux-2.5-15.el7.x86_64 libselinux-utils-2.5-15.el7.x86_64 How reproducible: Every time Steps to Reproduce: 1. Configure an interface to get IP address via dhcp. E.g,: /etc/sysconfig/network-scripts/ifcfg-eth1: TYPE="Ethernet" PROXY_METHOD="none" BROWSER_ONLY="no" BOOTPROTO="dhcp" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" IPV6_ADDR_GEN_MODE="stable-privacy" NAME="eth1" DEVICE="eth1" ONBOOT="yes" NM_CONTROLLED=no 2. Create a systemd service that brings up eth1 via ifup. E.g.: [nutanix@localhost ~]$ cat /usr/example.py #! /usr/bin/env python import subprocess def run(cmd_tokens): subprocess.call(cmd_tokens) if __name__ == '__main__': # We first stop network.service, so we can bring up interface again. run(["service", "network", "stop"]) # We manually bring up eth1. run(["/sbin/ifdown", "eth1"]) run(["/sbin/ifup", "eth1"]) # We use network.service to bring-up the *already up* eth1 run(["service", "network", "start"]) [nutanix@localhost ~]$ systemctl cat example.service # /etc/systemd/system/example.service [Unit] Description=Example After=network.target StartLimitIntervalSec=0 [Service] Type=oneshot ExecStart=/usr/example.py [Install] WantedBy=multi-user.target [Install] WantedBy=multi-user.target 3. Start the systemd service example.service $ systemctl start example.service 4. Check "ps auxZ | grep dhclient" and "ls -lhZ /var/run" and notice the multiple instances of dhclient Actual results: $ ps auxZ | grep dhclient ; ls -lhZ /var/run/ | grep dhclient system_u:system_r:unconfined_service_t:s0 root 17084 0.0 0.0 103004 2980 ? Ss 07:55 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3 system_u:system_r:dhcpc_t:s0 root 17464 0.0 0.0 103004 4160 ? Ss 07:55 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3 system_u:system_r:dhcpc_t:s0 root 18187 0.0 0.0 103004 4036 ? Ss 07:55 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H pibb-4 eth3 unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 18265 0.0 0.0 112824 2248 pts/0 S+ 07:55 0:00 grep --color=auto dhclient Expected results: ps auxZ | grep dhcli ; ls -lhZ /var/run/ | grep dhcl ; systemctl status network unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 root 6941 0.0 0.0 112824 2280 pts/1 S+ 16:35 0:00 grep --color=auto dhcli system_u:system_r:dhcpc_t:s0 root 20734 0.0 0.0 103004 2980 ? Ss 16:31 0:00 /sbin/dhclient -1 -q -lf /var/lib/dhclient/dhclient--eth3.lease -pf /var/run/dhclient-eth3.pid -H seabass-3 eth3 -rw-r--r--. root root system_u:object_r:dhcpc_var_run_t:s0 dhclient-eth3.pid