Bug 1995953

Description of problem:
Ingresscontroller change the replicas from default 2 to 3 for scaling up first time, all the ingress pods will be rolling update, it makes an unwanted rolling update for the current using ingress pods.

However, from the second time the replicas scaling up will not be rolling update for all the ingress pods, it will only scale up the new replicas pods.

It found the reason is the first time deploying with the initial deploymenmt doesn't 
sort the ENV in 'spec.template.spec.containers.env' fields.
      - env:
        - name: STATS_PORT
          value: "1936"
        - name: ROUTER_SERVICE_NAMESPACE
          value: openshift-ingress
        - name: DEFAULT_CERTIFICATE_DIR
          value: /etc/pki/tls/private
        - name: DEFAULT_DESTINATION_CA_PATH
          value: /var/run/configmaps/service-ca/service-ca.crt
        - name: ROUTER_SERVICE_NAME
          value: default
        - name: STATS_USERNAME
          valueFrom:
            secretKeyRef:
              key: statsUsername
              name: router-stats-default
        - name: STATS_PASSWORD
          valueFrom:
            secretKeyRef:
              key: statsPassword
              name: router-stats-default
        - name: ROUTER_METRICS_TYPE
          value: haproxy
        - name: ROUTER_METRICS_TLS_CERT_FILE
          value: /etc/pki/tls/metrics-certs/tls.crt
        - name: ROUTER_METRICS_TLS_KEY_FILE
          value: /etc/pki/tls/metrics-certs/tls.key
        - name: ROUTER_CANONICAL_HOSTNAME
          value: apps.ocp4.example.com
        - name: ROUTER_THREADS
          value: "4"
        - name: ROUTER_CIPHERS
          value: TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        - name: SSL_MIN_VERSION
          value: TLSv1.2
        - name: ROUTER_DISABLE_NAMESPACE_OWNERSHIP_CHECK
          value: "false"
        - name: ROUTER_ALLOW_WILDCARD_ROUTES
          value: "false"
        - name: ROUTER_SET_FORWARDED_HEADERS
          value: append
        - name: ROUTER_DISABLE_HTTP2
          value: "true"

Once the ingress controller has been change the replicas the default 2 to 3, the sorting of ENV has been changed, it will trigger the rolling update for all the ingress pods, will make the unwanted rolling update for all the ingress pods.

      - env:
        - name: DEFAULT_CERTIFICATE_DIR
          value: /etc/pki/tls/private
        - name: DEFAULT_DESTINATION_CA_PATH
          value: /var/run/configmaps/service-ca/service-ca.crt
        - name: ROUTER_ALLOW_WILDCARD_ROUTES
          value: "false"
        - name: ROUTER_CANONICAL_HOSTNAME
          value: apps.ocp4.example.com
        - name: ROUTER_CIPHERS
          value: TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        - name: ROUTER_DISABLE_HTTP2
          value: "true"
        - name: ROUTER_DISABLE_NAMESPACE_OWNERSHIP_CHECK
          value: "false"
        - name: ROUTER_METRICS_TLS_CERT_FILE
          value: /etc/pki/tls/metrics-certs/tls.crt
        - name: ROUTER_METRICS_TLS_KEY_FILE
          value: /etc/pki/tls/metrics-certs/tls.key
        - name: ROUTER_METRICS_TYPE
          value: haproxy
        - name: ROUTER_SERVICE_NAME
          value: default
        - name: ROUTER_SERVICE_NAMESPACE
          value: openshift-ingress
        - name: ROUTER_SET_FORWARDED_HEADERS
          value: append
        - name: ROUTER_THREADS
          value: "4"
        - name: SSL_MIN_VERSION
          value: TLSv1.2
        - name: STATS_PASSWORD
          valueFrom:
            secretKeyRef:
              key: statsPassword
              name: router-stats-default
        - name: STATS_PORT
          value: "1936"
        - name: STATS_USERNAME
          valueFrom:
            secretKeyRef:
              key: statsUsername
              name: router-stats-default

The user's environment is OCP4.7.13.
I verified on OCP4.6.40 is the same, but verified on OCP4.8.2 doesn't happen and the sorting of ENV is correct.

OpenShift release version:
OCP 4.7.13

Cluster Platform:
UPI

How reproducible:
Please refer to the details.

Steps to Reproduce (in detail):
1. Simulated cluster installation has been completed, change the replicas to 2 and delete the deployment for recreating.

# oc patch -n openshift-ingress-operator ingresscontroller/default --patch='{"spec": {"replicas": 2}}' --type=merge

# oc delete -n openshift-ingress deployment/router-default
deployment.apps "router-default" deleted

# oc -n openshift-ingress get pods
NAME                              READY   STATUS        RESTARTS   AGE
router-default-76fc4669d7-9qqct   0/1     Running       0          7s
router-default-76fc4669d7-d8vq8   0/1     Running       0          7s

# oc -n openshift-ingress get deployment/router-default -o yaml > router-default_initial_replicas.yaml

2. Change the replicas from 2 to 3 for scaling up the ingress pods.
It found the current pods for 
'router-default-76fc4669d7-d8vq8' and 'router-default-76fc4669d7-zpm2t' 
will be triggering the rolling update.

# oc patch -n openshift-ingress-operator  ingresscontroller/default --patch='{"spec": {"replicas": 3}}' --type=merge

# oc -n openshift-ingress get pods
NAME                              READY   STATUS        RESTARTS   AGE
router-default-76fc4669d7-9qqct   1/1     Running       0          106s
router-default-76fc4669d7-d8vq8   1/1     Terminating   0          106s
router-default-76fc4669d7-zpm2t   1/1     Terminating   0          26s
router-default-77b7b74855-tqnht   0/1     Pending       0          3s
router-default-77b7b74855-xwmwh   1/1     Running       0          59s

# oc -n openshift-ingress get deployment/router-default -o yaml > router-default_change_replicas.yaml

3.Comparing the files of 'router-default_initial_replicas.yaml' and 'router-default_change_replicas.yaml', 
not only replicas fields will be changed, 
it found the sorting of ENV items also will be changed.

Actual results:
Ingresscontroller change the replicas from default 2 to 3 for scaling up first time, all the ingress pods will be rolling update.

Expected results:
Ingresscontroller change the replicas from default 2 to 3 for scaling up first time will NOT be rolling update for all the ingress pods. Only scale up the new replicas pods.

Impact of the problem:
Ingresscontroller change the replicas from default 2 to 3, all the ingress pods will be rolling update that it makes an unwanted rolling update for the current using ingress pods and losing the trascation traffic.

Additional info:
None.

** Please do not disregard the report template; filling the template out as much as possible will allow us to help you. Please consider attaching a must-gather archive (via `oc adm must-gather`). Please review must-gather contents for sensitive information before attaching any must-gathers to a bugzilla report.  You may also mark the bug private if you wish.