Bug 1917579 - DNS daemonset rollout is taking 5.27 hours on a 250 node cluster during the upgrades
Summary: DNS daemonset rollout is taking 5.27 hours on a 250 node cluster during the u...
Keywords:
Status: CLOSED DUPLICATE of bug 1903887
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Networking
Version: 4.7
Hardware: Unspecified
OS: Linux
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: aos-network-edge-staff
QA Contact: Hongan Li
URL:
Whiteboard: aos-scalability-47
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-01-18 20:00 UTC by Naga Ravi Chaitanya Elluri
Modified: 2022-08-04 22:39 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-01-19 17:53:12 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Naga Ravi Chaitanya Elluri 2021-01-18 20:00:04 UTC
Description of problem:
DNS cluster operator is taking ~5 hours for the daemonsets to roll out on a 250 node cluster during the upgrade from 4.6.9 -> 4.7.0-fc.2 even when the max unavilable replicas is set to 10% instead of 1 after patching https://bugzilla.redhat.com/show_bug.cgi?id=1880148:

  updateStrategy:
    rollingUpdate:
      maxUnavailable: 10%
    type: RollingUpdate

The replicas seem to still rollout one at a time: https://snapshot.raintank.io/dashboard/snapshot/NOB8VPaH1UUNQIdl2gl5CwREdiPiWYwX.

Logs: http://dell-r510-01.perf.lab.eng.rdu2.redhat.com/large-scale/4.7-sdn-kube-1.20/bugs/dns-slow-rollout-upgrades/
 
Version-Release number of selected component (if applicable):
4.7.0-fc.2

How reproducible:
We just ran the upgrade once for this version.

Steps to Reproduce:
1. Install a large scale cluster using 4.6.9 bits.
2. Upgrade to 4.7.0-fc.2
3. Monitor the upgrade timing of the dns cluster operator

Actual results:
1 replica of DNS daemonset is rolled out at a time.

Expected results:
10% of the replicas are rolled out at a time.

Additional info:

Comment 1 Andrew McDermott 2021-01-19 17:53:12 UTC

*** This bug has been marked as a duplicate of bug 1903887 ***


Note You need to log in before you can comment on or make changes to this bug.