Bug 2214095

Description of problem:

When stopping a "IPsrcaddr" resource on an interface with multiple IP's on separate subnets configured, we always experience a stop failure. This additionally leads to fences due to the failed stop operation ( Below output replaces "#012" and "\n" characters with spaces " " ):

    rhel8-node1 $ cat /var/log/messages
    --------------------------->8-------------------------------
    Jun 11 11:53:22 rhel8-node1 pacemaker-controld[2325]: notice: Requesting local execution of stop operation for virtualsrc on rhel8-node1
    Jun 11 11:53:22 rhel8-node1 IPsrcaddr(virtualsrc)[43918]: ERROR: command 'ip route replace  192.168.120.218/24 dev enp1s0 proto kernel scope link src 192.168.122.218 192.168.120.218 ' failed
    Jun 11 11:53:22 rhel8-node1 pacemaker-controld[2325]: notice: Result of stop operation for virtualsrc on rhel8-node1: error (command 'ip route replace  192.168.120.218/24 dev enp1s0 proto kernel scope link src 192.168.122.218)


Version-Release number of selected component (if applicable):

# rpm -q resource-agents kernel
resource-agents-4.9.0-40.el8.x86_64
kernel-4.18.0-477.13.1.el8_8.x86_64

How reproducible:
100% reproducible

Steps to Reproduce:
1. Create a Virtual IP on the same network interface used by the system, or two virtual IP resources on the same network interface. The IP's must be on two different subnets, so they both show up as "primary" type networks:

    $ pcs resource create virtualip IPaddr2 ip=192.168.120.218 cidr_netmask=24 nic=enp1s0 --group testip

    $ ip  -4 -o addr show primary dev enp1s0
    2: enp1s0    inet 192.168.122.218/24 brd 192.168.122.255 scope global noprefixroute enp1s0\       valid_lft forever preferred_lft forever
    2: enp1s0    inet 192.168.120.218/24 scope global enp1s0\       valid_lft forever preferred_lft forever <--- 2nd virtual ip


2. Create a "IPsrcaddr" resource to change the source address of the new IP:

    $ pcs resource create virtualsrc IPsrcaddr destination=192.168.120.218/24 ipaddress=192.168.120.218 --group testip

    $ pcs status
    ---------------------------->8-------------------------
      * Resource Group: testip:
        * virtualip	(ocf::heartbeat:IPaddr2):	 Started rhel8-node1
        * virtualsrc	(ocf::heartbeat:IPsrcaddr):	 Started rhel8-node1

Actual results:

When the resource is stopped for any reason, this triggers a failure of the "ip route replace" command used by the "IPsrcaddr" resource:

    $ pcs resource disable testip

    $ pcs status
    ---------------------------->8-------------------------
      * Resource Group: testip (disabled):
        * virtualip	(ocf::heartbeat:IPaddr2):	 Started rhel8-node1 (disabled)
        * virtualsrc	(ocf::heartbeat:IPsrcaddr):	 FAILED rhel8-node1 (disabled)

    Failed Resource Actions:
      * virtualsrc_stop_0 on rhel8-node1 'error' (1): call=68, status='complete', exitreason='command 'ip route replace  192.168.120.218/24 dev enp1s0 proto kernel scope link src 192.168.122.218', last-rc-change='Sun Jun 11 11:53:22 2023', queued=0ms, exec=30ms


Expected results:
The IPsrcaddr resource stops without error.

Additional info:
Cause of error: During stop opeartions for the IPsrcaddr resource we collect information on the route to replace using the below command, and pass this into "ip route replace" command:

    $ ip  -4 -o addr show primary dev <dev> | awk '{split($4,a,"/");print a[1]}'

If there are 2 ip's from seperate networks attached to the same interface though, both are labeled as "primary". This results in 2 IP's returned from this command, but the resource expects just one IP:

    $ ip  -4 -o addr show primary dev enp1s0 | awk '{split($4,a,"/");print a[1]}'
    192.168.122.218
    192.168.120.218

This causes the "ip route replace" command to fail with the second ip being reported as garbage:

    $ pcs resource debug-stop --full virtualsrc
    ------------------------->8--------------------------
    ++ 12:02:57: srca_stop:289: ip -4 -o addr show dev enp1s0 primary
    ++ 12:02:57: srca_stop:289: awk '{split($4,a,"/");print a[1]}'
    + 12:02:57: srca_stop:289: PRIMARY_IP='192.168.122.218
    192.168.120.218' <--- 2 ip's assigned
    + 12:02:57: srca_stop:290: OPTS='proto kernel scope link src 192.168.122.218
    192.168.120.218'
    + 12:02:57: srca_stop:292: ip route replace 192.168.120.218/24 dev enp1s0 proto kernel scope link src 192.168.122.218 192.168.120.218
    Error: either "to" is duplicate, or "192.168.120.218" is a garbage. <---