Bug 1962502

> step 4: the route named k8s-ingress-test-5s4gn is still there and admitted

In a way, OpenShift is actually working as designed.  The behavior is related to a decision regarding how we should handle ingresses that were made with previous versions of OpenShift:

    Finally, it is possible that a user could have created an Ingress with
    some nonempty value for spec.ingressClassName that did not match an
    OpenShift IngressClass object, and nevertheless intended for OpenShift
    to expose this Ingress.  Again, it is impossible to determine reliably
    what a user's intent was in such a scenario, but as OpenShift exposed
    such an Ingress before this enhancement, changing this behavior could
    break existing applications.
    
    To mitigate this last risk, the ingress-to-route controller does not
    remove Routes that earlier versions of OpenShift created for Ingresses
    that specify spec.ingressClassName.  Thus these Routes will continue to
    be in effect.  However, after this enhancement, OpenShift does not update
    such Routes and does not recreate them if the user deletes them.  As
    follow-up work to this enhancement, we are considering adding alerts in
    case any Routes existed in this state, so that the administrator would
    know that the Routes needed to be deleted, or the Ingress modified to
    specify an appropriate IngressClass so that OpenShift would once again
    reconcile the Routes.

https://github.com/openshift/enhancements/blob/master/enhancements/ingress/transition-ingress-from-beta-to-stable.md#risks-and-mitigations

    As follow-up work, we are considering modifying the ingress operator
    to list all Ingresses and Routes in the cluster and publish a metric
    for Routes that were created for Ingresses that OpenShift no longer
    manages.  This metric could be used in alerting rules.  The following
    alerting rules would be added to the ingress operator (see "Risks and
    Mitigations" for more context as to the purpose of these alerts):
    
    * An alert for Routes that were created from Ingresses that OpenShift
      is no longer managing.  

    * An alert for Ingresses older than 1 day that do not specify
      spec.ingressClassName.

https://github.com/openshift/enhancements/blob/master/enhancements/ingress/transition-ingress-from-beta-to-stable.md#implementation-details

The scenario that you describe is similar, in that it relates to what OpenShift should do with a route that was created for an ingress when the ingress specifies a nonexistent ingressclass.  

Granted, the scenario that you describe isn't quite the same as the scenario addressed in the enhancement proposal, and we *could* modify the ingress operator to remove routes for ingresses associated with an ingressclass when that ingressclass were deleted.  However, I'm still concerned that automatically performing such a destructive action would be risky.  It makes sense that deleting an ingress, route, or ingresscontroller disrupts traffic, but it seems less obvious that deleting an ingressclass would do the same.  

In my opinion, it thus makes sense to resolve this BZ by implementing the alert described in the enhancement proposal.  The result would be that after Step 4 in the BZ description, OpenShift would raise an alert indicating that an ingress that specified a nonexistent ingressclass existed.  Does this seem like a reasonable solution?

Thank you for your detailed explanation, Miciah. 

Yes, I think implementing the alert is a reasonable solution.

Verified with "4.12.0-0.ci.test-2022-09-06-043444-ci-ln-b6zz2d2-latest" image build with the patch via clusterbot. With this payload, the newly added "openshift_ingress_to_route_controller_ingress_without_class_name" and "openshift_ingress_to_route_controller_route_with_unmanaged_owner" are visible in the metric queries and they appear to get updated correctly for the respect state of ingresses having the ingressclasses uset or unset with mismatching values [Reference images attached]

With the inclusion of PR#823, it is observed that the "openshift_ingress_to_route_controller_ingress_without_class_name" increments and decrements properly as expected

Verified with 4.12.0-0.nightly-2022-11-07-181244 and passed.

we can see the alerts message on web UI: 
"Route k8s-ingress-test-mns6d is owned by an unmanaged Ingress."

and the ingress can be queried by metrics "openshift_ingress_to_route_controller_ingress_without_class_name"

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7399