Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1856263

Summary:	redis fails to bind an ip at system boot with "bind: Cannot assign requested address"
Product:	Red Hat Enterprise Linux 8	Reporter:	Muneaki Sugaya <msugaya>
Component:	NetworkManager	Assignee:	NetworkManager Development Team <nm-team>
Status:	CLOSED NOTABUG	QA Contact:	Desktop QE <desktop-qa-list>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	8.2	CC:	acardace, atragler, bgalvani, lrintel, rcollet, rkhan, sukulkar, thaller, till
Target Milestone:	rc	Flags:	pm-rhel: mirror+
Target Release:	8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	1858218 (view as bug list)		Environment:
Last Closed:	2020-07-17 09:24:44 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1858218

Description Muneaki Sugaya 2020-07-13 08:43:20 UTC

Description of problem:

redis fails to bind an ip at system boot with "bind: Cannot assign requested address" 

Version-Release number of selected component (if applicable):
RHEL8.2
  redis-5.0.3-2.module+el8.0.0.z+3657+acb471dc.x86_64
  NetworkManager-1.22.8-4.el8.x86_64
  systemd-239-29.el8.x86_64


How reproducible:
Always, however I think it depends on the system.


Steps to Reproduce:
1.
~~~
# yum install redis
# vi /etc/redis.conf
bind 127.0.0.1 XXX.XXX.XXX.XXX
~~~
XXX.XXX.XXX.XXX is an IP address of the system.


2.
~~~
# systemctl enable redis.service 
# reboot
~~~

3.
See the following log entry in /var/log/redis/redis.log. 

Could not create server TCP listening socket XXX.XXX.XXX.XXX:6379: bind: Cannot assign requested address


See the output of "netstat -antp".
~~~ Sample:
# netstat -antp | egrep "^Proto|redis"
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      860/redis-server 12 
~~~

There should be the following too.

tcp        0      0 XXX.XXX.XXX.XXX:6379    0.0.0.0:*               LISTEN      860/redis-server 12 


Actual results:
redis fails to bind an ip at system boot


Expected results:
redis binds an ip at system boot


Additional info:
"ExecStartPre=/bin/sleep 5" can be a workaround.

~~~ /etc/systemd/system/redis.service.d/override.conf
[Service]
ExecStartPre=/bin/sleep 5
~~~

Comment 1 Nathan Scott 2020-07-13 10:31:26 UTC

Could be the issue described here: https://serverfault.com/questions/941421/servers-fail-to-bind-to-addresses-at-boot

Is NetworkManager involved, and if so is NetworkManager-wait-online enabled?  AFAICT, the Redis service has a correct 'After' line for network-online.

Comment 4 Thomas Haller 2020-07-14 06:13:24 UTC

please provide a full level=TRACE log of a boot that shows the issue. See https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/contrib/fedora/rpm/NetworkManager.conf#n28 for how to get logs from NetworkManager.

Comment 6 Thomas Haller 2020-07-15 13:56:16 UTC

thanks for the log!!

You can see there:


```
NetworkManager[798]: <info>  [1594816384.6047] manager: startup complete
```
and only later
```
NetworkManager[798]: <debug> [1594816386.6750] platform: (eth1) signal: link   added: 3: eth1 <DOWN;broadcast,multicast> mtu 1500 arp 1 
...
NetworkManager[798]: <debug> [1594816386.7147] platform: (ens3) signal: link changed: 2: ens3 <DOWN;broadcast,multicast> mtu 1500 arp 1 ethernet? no
```

That means, the driver takes an unusually long time to detect the interface. NetworkManager doesn't know that the device will still show up, and at that point already declared "startup complete". Thus, "network-online.target" gets reached too early and Redis service gets start too early.


A possibly fix is to configure "connection.wait-device-timeout" to for example 5000 (milliseconds). See `man nm-settings`. Note that the manual tells you that this option only works if you also set "connection.interface-name", but luckily, your logs shows that you already do that.

Could you try with

    nmcli connection modify 75856aad-c9b9-41d7-b8fc-433c6372f521 connection.wait-device-timeout 5000

and reboot?



Check the logs for the "startup complete" line. That is what corresponds to completing NetworkManager-wait-online.service (and network-online.target). You don't want that to be reached too early.



I guess, a significantly more ugly workaround would be to start a systemd dummy service "Before=NetworkManager.service" that only does "sleep 5".

Comment 8 Thomas Haller 2020-07-16 09:49:26 UTC

> However I think it doesn't guarantee that the redis.service start after the processing for ens3 is completed. So there is a possibility that it might not work. Am I right?

Why do you think so? No, it's supposed to work now.

Comment 10 Thomas Haller 2020-07-17 09:24:44 UTC

Ah right. There is a possible small issue. I opened bug 1858218 for that.

However, it should still work mostly fine. Also, because you can already bind to an address, even if the device has no carrier yet.

The original issue should be avoided by configuring connection.wait-device-timeout.

I don't think there is anything left to do. Closing.

Please reopen, if something is missing. Thanks!!



(btw, the real fix is for Redis not being so fragile, and itself react dynamically when addresses come and go. network-online.target is a workaround for services that don't properly handle networking to change).