Bug 2029796

+++ This bug was initially created as a clone of Bug #2012057 +++

Description of problem:

The Route RA's probe operation fails if the device doesn't have an address assigned.

Consider a situation where a Route resource's device is an interface that does not get any IPv4 address assigned automatically at boot. It will be assigned an address later by an IPaddr2 resource.

If the Route resource's probe runs before the IPaddr2 resource starts, then the probe will fail because `ip route show` fails in the route_status() function. Then in the ensuing recovery sequence, the stop operation also fails for the same reason.
~~~
route_status() {
    show_output="$(ip $addr_family route show $(create_route_spec) 2>/dev/null)"
    if [ $? -eq 0 ]; then
    ...
    else
        # "ip route show" returned an error code. Assume something
        # went wrong.
        return $OCF_ERR_GENERIC
    fi
}
~~~

~~~
route_stop() {
    route_status
    status=$?
    case $status in
        $OCF_SUCCESS)
        ...
        $OCF_NOT_RUNNING)
        ...
    esac
    return $OCF_ERR_GENERIC
}
~~~

Note that an order constraint (start IPaddr2 then start Route) will **not** prevent the Route resource's probe from running before the IPaddr2 resource starts. The Route probe can and should run before the IPaddr2 resource starts.

Therefore, I believe the probe should exit with OCF_NOT_RUNNING in this situation. It's reasonable to expect that the system may simply not be prepared for the Route resource at the time when the probe runs, so we should use OCF_NOT_RUNNING to give it a fair chance.
~~~
    else
        # "ip route show" returned an error code. Assume something
        # went wrong.
        if ocf_is_probe; then
            return $OCF_NOT_RUNNING
        else
            return $OCF_ERR_GENERIC
        fi
    fi
~~~

I'm currently unsure of what (if anything) we should do for the stop operation in this scenario. Currently, as mentioned above, the stop operation fails if the probe fails in this situation, because route_stop() wants to receive OCF_SUCCESS or OCF_NOT_RUNNING from route_status(). Three options:
  - Modify the route_stop() logic.
  - Modify route_status() so that it returns OCF_ERR_GENERIC **only** during a non-probe monitor operation (i.e., returns OCF_NOT_RUNNING during a probe **or** a stop operation).
  - Leave the stop operation alone and hope for the best. Returning OCF_NOT_RUNNING in route_status() during a probe will avoid this particular issue. But the stop operation is arguably brittle if we have it return an OCF_ERR_GENERIC failure when `ip route show` fails.

-----

Version-Release number of selected component (if applicable):

resource-agents-4.1.1-90.el8_4.6

-----

How reproducible:

Always

-----

Steps to Reproduce:

Start with an empty cluster configuration. I'm using a 2-node cluster (node1 and node2). My NIC with no address assigned at boot is ens3.

[root@fastvm-rhel-8-0-23 ~]# cat /tmp/setup3.sh 
#!/bin/bash
pcs cluster cib new_cfg

pcs -f new_cfg resource create vip ocf:heartbeat:IPaddr2 ip=192.168.23.61 cidr_netmask=24 nic=ens3
pcs -f new_cfg resource create route ocf:heartbeat:Route destination=0.0.0.0/0 gateway=192.168.23.1 device=ens3 table=1

pcs -f new_cfg constraint location vip prefers node1
pcs -f new_cfg constraint order vip then route
pcs -f new_cfg constraint colocation add route with vip

pcs cluster cib-push new_cfg --config


[root@fastvm-rhel-8-0-23 ~]# date && bash /tmp/setup.sh
Fri Oct  8 00:02:11 PDT 2021

-----

Actual results:

Route probe runs before IPaddr2 has started, and it fails with OCF_ERR_GENERIC because ens3 has no IP address (and thus the FIB table has not been initialized).

Oct  8 00:02:17 fastvm-rhel-8-0-24 pacemaker-schedulerd[1844]: notice:  * Start      vip     ( node1 )
Oct  8 00:02:17 fastvm-rhel-8-0-24 pacemaker-schedulerd[1844]: notice:  * Start      route   ( node1 )
Oct  8 00:02:17 fastvm-rhel-8-0-24 pacemaker-schedulerd[1844]: notice: Calculated transition 3, saving inputs in /var/lib/pacemaker/pengine/pe-input-523.bz2
...
Oct  8 00:02:17 fastvm-rhel-8-0-24 pacemaker-controld[1845]: notice: Initiating monitor operation vip_monitor_0 on node1
Oct  8 00:02:17 fastvm-rhel-8-0-24 pacemaker-controld[1845]: notice: Initiating monitor operation route_monitor_0 on node1
Oct  8 00:02:17 fastvm-rhel-8-0-24 pacemaker-controld[1845]: notice: Initiating start operation vip_start_0 on node1
Oct  8 00:02:17 fastvm-rhel-8-0-24 pacemaker-controld[1845]: notice: Transition 3 aborted by operation route_monitor_0 'modify' on node1: Event failed
Oct  8 00:02:17 fastvm-rhel-8-0-24 pacemaker-controld[1845]: notice: Transition 3 action 2 (route_monitor_0 on node1): expected 'not running' but got 'error'

-----

Expected results:

The Route probe on node1 exits with OCF_NOT_RUNNING.

-----

Additional info:

A workaround can be devised using an ocf:pacemaker:attribute resource and a location constraint rule with resource-discovery=never and score=-INFINITY. For example, here's a workaround I created for a customer, who has one VIP per node and wants the same route created on both nodes. Workaround works for me; pending customer's confirmation.
~~~
[root@fastvm-rhel-8-0-23 ~]# cat /tmp/setup2.sh 
#!/bin/bash
# # Save the CIB
pcs cluster cib ipv4_cfg

# # Create a group consisting of an IPaddr2 resource and an attribute that signifies that the IP is active
# # Configure the group to prefer node 1
pcs -f ipv4_cfg resource create ipv4_node1 ocf:heartbeat:IPaddr2 --group node1_grp ip=192.168.23.61 cidr_netmask=24 nic=ens3 op monitor interval=30s
pcs -f ipv4_cfg resource create attr_node1 ocf:pacemaker:attribute name=ipv4_node1_active --group node1_grp
pcs -f ipv4_cfg constraint location node1_grp prefers node1

# # Same for node 2
pcs -f ipv4_cfg resource create ipv4_node2 ocf:heartbeat:IPaddr2 --group node2_grp ip=192.168.23.62 cidr_netmask=24 nic=ens3 op monitor interval=30s
pcs -f ipv4_cfg resource create attr_node2 ocf:pacemaker:attribute name=ipv4_node2_active --group node2_grp
pcs -f ipv4_cfg constraint location node2_grp prefers node2

# # Create the Route resource and clone it
pcs -f ipv4_cfg resource create route4_haproxy ocf:heartbeat:Route destination=0.0.0.0/0 gateway=192.168.23.1 device=ens3 table=1 op monitor interval=30s
pcs -f ipv4_cfg resource clone route4_haproxy clone-max=2

# # Prevent route4_haproxy from running (score=-INFINITY) **and** prevent its probe from running (resource-discovery=never)
# # unless an appropriate attribute is set
pcs -f ipv4_cfg constraint location route4_haproxy-clone rule resource-discovery=never score=-INFINITY ipv4_node1_active ne 1 and ipv4_node2_active ne 1

# # Push the config
pcs cluster cib-push ipv4_cfg --config
~~~

--- Additional comment from Reid Wahl on 2021-10-12 10:52:07 CEST ---

Workaround listed in comment 0 did not prevent the issue.

Posted in support case:
~~~
The "workaround" does what it was intended to do (i.e., prevent the Route probe from running until after the IPaddr2 resource has started) -- but I misunderstood the requirement. I thought I had tested to be sure the workaround avoided the issue but I must have missed something the first time.


My understanding was that "if we create the IP address before we run `ip -4 route show to 0.0.0.0/0 dev ens10 via <addr> table 1`, then ipv4.method should get set to something other than 'disabled', the FIB table should get initialized, and the `ip -4 route show ...` command should succeed." Curtis clarified to me that that basically only happens if the IP address is configured to start at boot (perhaps also if it's started by nmcli later?). Since the IPaddr2 resource agent simply uses an `ip addr add` command to add the address to ens10, the FIB table still doesn't get initialized. This aligns with what you've reported -- i.e., that the Route resource fails even if you've started the IPaddr2 resource.


This leaves us with a few options until an official patch is released.

  Option 1: Add a NetworkManager dispatcher script on each node as Curtis suggested. This should cause the FIB table to get created at boot time so that pacemaker has no issues when running the Route probe operation later. This is the approach I would use if it works as intended.
 
        You may try adding a custom dispatcher script such as 30-do-route-tables to /etc/NetworkManager/dispatcher.d and setting it to executable:

        Here is a basic example:

          # cat 30-do-route-tables 
          #!/bin/bash

          logger "NM doing $0 $*" 
          if [ "$2" != "up" ] ; then
                logger "NM $0: no action"
        	exit 0
          fi
          if [ -f /etc/sysconfig/network-scripts/route-$1 ] ; then
        	logger "NM $0: ip route add `cat /etc/sysconfig/network-scripts/route-$1`"
        	ip route add `cat /etc/sysconfig/network-scripts/route-$1`
          fi
          exit 0


  Option 2: Disable the probe operation altogether for the route4_haproxy resource. You could do this by creating a location constraint rule that applies to all nodes (by using expression `defined #uname`, which should evaluate to true for every node) with `resource-discovery=never` to prevent the probe from ever running. In theory, the danger is that pacemaker will never check whether the route4_haproxy resource is running when it should not be running, since the probe would be disabled. In practice, this is unlikely to be a concern, since the resource is intended to run as a clone and doesn't manage anything that could easily lead to corruption if it runs outside cluster control. I believe the following command would accomplish this (ran on my cluster to check):

        # pcs constraint location route4_haproxy-clone rule resource-discovery=never defined \#uname


  Option 3: Create a new resource agent (ocf:custom:Route) using the modified version of the Route resource agent, which you confirmed worked. [This refers to the ocf_is_probe check added to route_status() in comment 0.] As we discussed, a custom resource agent is technically not supported by Red Hat. But in the event of an issue, point to this case and Bugzilla and I'll take a look.

        # mkdir /usr/lib/ocf/resource.d/custom
        # # Copy the modified (working) Route resource agent to /usr/lib/ocf/resource.d/custom/Route
        # # Create an ocf:custom:Route resource instead of an ocf:heartbeat:Route resource.
        # # Pacemaker will then use the version in /usr/lib/ocf/resource.d/custom
        # # The purpose of /usr/lib/ocf/resource.d/custom is twofold:
        # #   - To ensure that package updates will not overwrite the change.
        # #   - To make it clear that the resources are using a customized version of the resource agent.
~~~

--- Additional comment from Robert Scheck on 2021-10-12 11:12:05 CEST ---

Representing the before mentioned customer, I would like to note that we are fine to go with option 3 (unsupported from Red Hat perspective) if a final fix has been accepted by engineering for CentOS 8 Stream.

--- Additional comment from Oyvind Albrigtsen on 2021-12-02 15:02:14 CET ---

https://github.com/ClusterLabs/resource-agents/pull/1597

--- Additional comment from Robert Scheck on 2021-12-06 16:20:22 CET ---

(In reply to Oyvind Albrigtsen from comment #4)
> https://github.com/ClusterLabs/resource-agents/pull/1597

I am sorry, but that patch is NOT working for us. The difference is that the interface exists in our case, but the non-default IPv4 FIB table is still not yet initalized. And this is what fails here.

--- Additional comment from Oyvind Albrigtsen on 2021-12-07 10:20:00 CET ---

(In reply to Robert Scheck from comment #5)
> (In reply to Oyvind Albrigtsen from comment #4)
> > https://github.com/ClusterLabs/resource-agents/pull/1597
> 
> I am sorry, but that patch is NOT working for us. The difference is that the
> interface exists in our case, but the non-default IPv4 FIB table is still
> not yet initalized. And this is what fails here.

Thank you for your feedback.

I'll update it to Reid's suggested fix.