Bug 2095623 - [rebase v1.24] [sig-storage] In-tree Volumes [Driver: azure-file] tests fail
Summary: [rebase v1.24] [sig-storage] In-tree Volumes [Driver: azure-file] tests fail
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.11
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.12.0
Assignee: Fabio Bertinatto
QA Contact: Wei Duan
URL:
Whiteboard:
: 2029841 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-06-10 03:47 UTC by Abu Kashem
Modified: 2023-01-17 19:50 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-01-17 19:49:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 27259 0 None open Bug 2095623: Revert "Disable azure-file tests" 2022-06-21 13:12:13 UTC
Github openshift release pull 29274 0 None Merged Bug 2095623: Add secrets permissions for Azure File in-tree plugin 2022-06-21 13:12:10 UTC
Github openshift release pull 29477 0 None Merged Bug 2095623: Add secrets permissions in all Azure "pre" workflows 2022-06-21 13:12:08 UTC
Github openshift release pull 29643 0 None Merged Bug 2095623: Add secrets permissions in all all Azure ipi workflows 2022-06-21 13:12:06 UTC
Github openshift release pull 29698 0 None open Bug 2095623: Add secrets permissions to OVN upgrade workflow 2022-06-21 21:58:44 UTC
Red Hat Product Errata RHSA-2022:7399 0 None None None 2023-01-17 19:50:17 UTC

Description Abu Kashem 2022-06-10 03:47:35 UTC
new dynamic PV tests are failing: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.11-informing#periodic-ci-openshift-release-master-ci-4.11-upgrade-from-stable-4.10-e2e-azure-upgrade&show-stale-tests=

These tests came in #27181

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.11-upgrade-from-stable-4.10-e2e-azure-upgrade/1534420538045239296 is an example run which shows multiple failures.
expanding [sig-storage] In-tree Volumes [Driver: azure-file] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path and clicking stdout takes you to a spot that shows the events in the namespace.
like:

Jun 8 08:56:01.547: INFO: At 2022-06-08 08:51:01 +0000 UTC - event for azure-filekq7mn: {persistentvolume-controller } ProvisioningFailed: Failed to provision volume with StorageClass "e2e-volume-1958kblnl": couldn't create secret secrets is forbidden: User "system:serviceaccount:kube-system:persistent-volume-binder" cannot create resource "secrets" in API group "" in the namespace "e2e-volume-1958"

This is likely a product problem with permissions (though creating secrets seems odd) that is exposed via a new test in 1.24. To get back to a working state, the original PR should be reverted. It's unlikely that a permissions error is environmental in nature.

See https://github.com/openshift/origin/pull/27233

Comment 1 Abu Kashem 2022-06-10 04:01:07 UTC
origin pr is skipping these tests until June 18 so we can unblock folks - https://github.com/openshift/origin/pull/27235/commits

Comment 2 Fabio Bertinatto 2022-06-10 12:55:38 UTC
This PR adds the required permissions in our CI: https://github.com/openshift/release/pull/29274

Comment 4 Fabio Bertinatto 2022-06-13 14:38:49 UTC
We'ĺl be able to verify this once https://github.com/openshift/origin/pull/27244 merges. That PR brings in the Azure File tests that have been failing because of the missing RBAC rule in our CI.

Comment 5 Fabio Bertinatto 2022-06-14 11:02:52 UTC
Actually, I created a draft PR [1] that doesn't skip Azure File tests and launched an azure-e2e job there. 

As we can see in [2], Azure File tests ran successfully. Moving status to MODIFIED.

[1] https://github.com/openshift/origin/pull/27248
[2] https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/27248/pull-ci-openshift-origin-master-e2e-azure/1536360637641265152

Comment 7 ravig 2022-06-15 16:13:53 UTC
Not yet fixed.

Comment 8 Fabio Bertinatto 2022-06-15 18:31:44 UTC
The PR [1] fixed CSI jobs that use the "ipi-azure-pre" chain. This fixed regular jobs like "e2e-azure".

However, some jobs use different "pre" chain to configure clusters before tests run. For example, the "periodic-ci-openshift-release-master-ci-4.11-upgrade-from-stable-4.10-e2e-azure-upgrade" job [2] uses the "ipi-azure-pre-stableinitial" chain instead.

The PR [3] extended the fix done in [1] to all the other "ipi-azure-pre-*" chains.

[1] https://github.com/openshift/release/pull/29274
[2] https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/aggregator-periodic-ci-openshift-release-master-ci-4.11-upgrade-from-stable-4.10-e2e-azure-upgrade/1536690381557600256
[3] https://github.com/openshift/release/pull/29477

Comment 9 Fabio Bertinatto 2022-06-15 20:15:15 UTC
Next steps for this ticket:

1. Wait for https://github.com/openshift/origin/pull/27244 to merge
2. Create a PR against openshift/origin to revert this commit: https://github.com/openshift/origin/pull/27244/commits/40a803314046e70f1ad9a8db6eaf4e9fece176f3

Comment 14 Fabio Bertinatto 2022-08-09 16:26:33 UTC
*** Bug 2029841 has been marked as a duplicate of this bug. ***

Comment 17 Wei Duan 2022-10-14 10:13:41 UTC
The azure-file cases are added back and get passed like: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.12-informing#periodic-ci-openshift-release-master-ci-4.12-e2e-azure-ovn&show-stale-tests&show-stale-tests=

Marked as VERIFIED.

Comment 20 errata-xmlrpc 2023-01-17 19:49:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.12.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7399


Note You need to log in before you can comment on or make changes to this bug.