Bug 1988291 - 4.7 -> 4.8 upgrade, node-exporter can't rollout
Summary: 4.7 -> 4.8 upgrade, node-exporter can't rollout
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.9.0
Assignee: Jayapriya Pai
QA Contact: Junqi Zhao
URL:
Whiteboard:
Depends On:
Blocks: 1988991
TreeView+ depends on / blocked
 
Reported: 2021-07-30 09:37 UTC by Vadim Rutkovsky
Modified: 2021-10-18 17:43 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1988991 (view as bug list)
Environment:
Last Closed: 2021-10-18 17:43:22 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-monitoring-operator pull 1307 0 None open Bug 1988291: pkg/client/client.go: Add retry logic for daemonset create 2021-07-30 10:11:01 UTC
Red Hat Product Errata RHSA-2021:3759 0 None None None 2021-10-18 17:43:43 UTC

Description Vadim Rutkovsky 2021-07-30 09:37:50 UTC
Description of problem:
All node-exporter pods are pending:
```
$ oc get pods -l app.kubernetes.io/component=exporter -o wide   
NAME                                  READY   STATUS    RESTARTS   AGE     IP            NODE                                        NOMINATED NODE   READINESS GATES
kube-state-metrics-669dbd4d69-4wrtw   3/3     Running   0          53m     10.131.8.27   ip-10-0-53-237.us-west-2.compute.internal   <none>           <none>
node-exporter-4qjv9                   0/2     Pending   0          18h     <none>        <none>                                      <none>           <none>
node-exporter-5cvxk                   0/2     Pending   0          18h     <none>        <none>                                      <none>           <none>
```

with 
```
  Warning  FailedScheduling   3m11s                  default-scheduler   0/12 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 11 node(s) didn't match Pod's node affinity/selector.
```
events. Happened during 4.7.21 -> 4.8.2 upgrade on vSphere build cluster

Comment 1 Damien Grisonnet 2021-07-30 09:43:00 UTC
This is also related to https://github.com/openshift/cluster-monitoring-operator/pull/1257#pullrequestreview-700719930

Setting severity/priority to high since it is preventing a CI cluster upgrade.

Comment 4 Junqi Zhao 2021-08-05 03:30:10 UTC
checked with recent 4.8 -> 4.9 nightly payload upgrade with azure cluster, did not see the reported error,
and search from CI upgrade jobs, no such error for node-exporter
https://search.ci.openshift.org/?search=didn%27t+have+free+ports+for+the+requested+pod+ports&maxAge=168h&context=1&type=all&name=periodic-ci-openshift-release-master-nightly-4.9-e2e-.*-upgrade&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

Comment 11 errata-xmlrpc 2021-10-18 17:43:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.9.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:3759


Note You need to log in before you can comment on or make changes to this bug.