Bug 1365146
| Summary: | [preview] deployments and scale up/down are very, very slow | |||
|---|---|---|---|---|
| Product: | OpenShift Online | Reporter: | Steve Speicher <sspeiche> | |
| Component: | Deployments | Assignee: | Dan Mace <dmace> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Mike Fiedler <mifiedle> | |
| Severity: | urgent | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 3.x | CC: | aos-bugs, mfojtik, mifiedle, sspeiche, xtian, yasun | |
| Target Milestone: | --- | |||
| Target Release: | --- | |||
| Hardware: | All | |||
| OS: | All | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1366381 (view as bug list) | Environment: | ||
| Last Closed: | 2016-10-04 13:08:04 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1366381 | |||
|
Description
Steve Speicher
2016-08-08 13:20:42 UTC
I think this should be fixed in >3.3, but will let Michalis prove me wrong. I think DC will retry after the conflict and that retry period seems long. @Michal but this is an urgent outage on prod preview close now. I only see the conflict some times and thing it is due to the deployment / scale performance slowness. So if the slowness is fixed, the conflict won't surface, right? (In reply to Steve Speicher from comment #2) > @Michal but this is an urgent outage on prod preview close now. I only see > the conflict some times and thing it is due to the deployment / scale > performance slowness. So if the slowness is fixed, the conflict won't > surface, right? Yeah, I think Michalis told me once that the conflict is not critical and it is retried. I'm not sure if retrying is causing the slowness or not. I'm going to investigate this tomorrow as this might be related to other deployment flakes we are seeing nowadays. Dan: It seems you will be the right person to look at this (it does not seem like a DC problem, but more as RC problem). Also Andy told me you were chasing something similar yesterday. The current issue I'm chasing is a minutes-long delay between DC and initial RC creation. This *could* be related since RC scaling via DC is also handled by the controllers: if processing is somehow slowed down for creating the RCs, it could also be slow for scaling the RCs in response to DC scale changes. My current theory is that deployment config controller processing time is now exceeding the reflector resync period. Each resync interval, the entire work queue is replaced and randomized, which would account for the wild variance. I'm going to do some load testing to verify. If the resync interval is indeed the problem, we'll need to introduce configuration to allow the resync interval to be modified beyond its current hard-coded 2 minute default. There any workaround or fix planned for preview (prod)? I'm still observing the same behavior since Thursday (Aug 4th) https://github.com/openshift/ose/pull/336 is our first attempt at a fix. Verified against atomic-openshift-3.2.1.13-1.git.4.8365dd3.el7.x86_64 With the release of atomic-openshift-3.2.1.13-1.git.4.8365dd3.el7.x86_64, the following root-level configuration key( deploymentControllerResyncMinutes: 15) added to the master-config.yaml. Hi Mike, For this bug is a performance bug and needs test case, has SVT team cover it? If yes, please help add the Polarion case number. Thanks. Thanks, Mike. |