Bug 1776079 - Cloud Credential Operator tries to create existing Azure Role Assignments every 10 hours triggering alerts in Azure
Summary: Cloud Credential Operator tries to create existing Azure Role Assignments eve...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cloud Credential Operator
Version: 4.2.z
Hardware: All
OS: Linux
unspecified
low
Target Milestone: ---
: 4.4.0
Assignee: Joel Diaz
QA Contact: Xiaoli Tian
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-11-25 05:00 UTC by Joel Pearson
Modified: 2020-05-04 11:17 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Reconciling a CredentialsRequest would attempt to create a role assignment that already exists. Consequence: Azure logs would show "create role assignment" errors. Fix: Check for existing role assignment to avoid creating one that already exists. Result: Less messages in Azure logs.
Clone Of:
Environment:
Last Closed: 2020-05-04 11:16:28 UTC
Target Upstream Version:


Attachments (Terms of Use)
Cloud credential operator logs (33.15 KB, text/plain)
2019-11-25 05:00 UTC, Joel Pearson
no flags Details
Azure error json (3.39 KB, text/plain)
2019-11-25 05:31 UTC, Joel Pearson
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift cloud-credential-operator pull 137 0 'None' closed Bug 1776079: Azure: check existing role assignments before creating a new one 2020-09-29 14:25:57 UTC
Red Hat Product Errata RHBA-2020:0581 0 None None None 2020-05-04 11:17:26 UTC

Description Joel Pearson 2019-11-25 05:00:42 UTC
Created attachment 1639342 [details]
Cloud credential operator logs

Description of problem:
Cloud Credential Operator (CCO) tries to create existing Azure Role Assignments every 10 hours triggering alerts in Azure.


Version-Release number of selected component (if applicable):
4.2.7


How reproducible:
It always happens every 10 hours


Steps to Reproduce:
1. Install an Azure cluster
2. Wait until it tries to renew the azure credentials (every 10 hours it looks like)
3. CCO won't report any errors
4. Go into the Azure console, and open the resource group that openshift is installed in, open the activity log, and change the event severity to exclude informational events. Look for any "Create role assignment" errors.

Actual results:
There are 3 "Create role assignment" errors, presumably one for each of the 3 tracked credential requests.


Expected results:
No errors, I would expect that the CCO wouldn't try to create credentials that already exist, which trigger Azure errors.


Additional info:

Comment 1 Joel Pearson 2019-11-25 05:31:44 UTC
Created attachment 1639343 [details]
Azure error json

Comment 3 Oleg Nesterov 2019-12-09 19:48:56 UTC
Verified on 4.4.0-0.nightly-2019-12-09-012357

Comment 6 errata-xmlrpc 2020-05-04 11:16:28 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.