Bug 1934978

Summary: ARO cluster accessing NFS via CSI driver throws error - "The Resource Microsoft.Network/virtualNetworks/subnets under resource group xx was not found”
Product: OpenShift Container Platform Reporter: Amit Kumar Das <amdas>
Component: StorageAssignee: aos-storage-staff <aos-storage-staff>
Storage sub component: Kubernetes External Components QA Contact: Qin Ping <piqin>
Status: CLOSED CANTFIX Docs Contact:
Severity: medium    
Priority: unspecified CC: aos-bugs, jsafrane
Version: 4.5   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-03-04 08:51:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Amit Kumar Das 2021-03-04 06:06:46 UTC
Description of problem:

ARO cluster accessing NFS via CSI driver throws error - "The Resource Microsoft.Network/virtualNetworks/subnets under resource group xx was not found”


Version-Release number of selected component (if applicable):

CSI Driver version: Driver Version: v1.1.0
OC Server Version: 4.5.31
Kubernetes Version: v1.18.3+e574db2
Red Hat Enterprise Linux CoreOS 45.82.202101291733-0 (Ootpa)
Kernel (e.g. uname -a): 4.18.0-193.41.1.el8_2.x86_64 
cri-o://1.18.4-6.rhaos4.5.gitbe796d1.el8
Install tools: kubectl
Others: Azure OpenShift (ARO) with exiting Network


How reproducible:

While requesting NFS PVC through CSI driver which has existing network gives error.

Error:
The Resource 'Microsoft.Network/virtualNetworks/subnets' under resource group 'CL-EXT-OCP-NPRD-RG' was not found


However, it works for SMB .

Actual results: Error
Expected results: Get NFS PVC

Master Log: nil
Node Log (of failed PODs): nil
PV Dump: nil
PVC Dump: nil
StorageClass Dump (if StorageClass used by PV/PVC): nil

Additional info:

I0226 19:08:19.117813       1 azure_subnetclient.go:131] Received error in subnet.get.request: resourceID: /subscriptions/e6db90b4-zzzz-zzzz-zzzz-4e81412fef0b/resourceGroups/CL-EXT-OCP-NPRD-RG/providers/Microsoft.Network/virtualNetworks//subnets/, error: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Network/virtualNetworks/subnets' under resource group 'CL-EXT-OCP-NPRD-RG' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}
E0226 19:08:19.117926       1 utils.go:123] GRPC error: rpc error: code = Internal desc = failed to update the service endpoints: failed to get the subnet  under vnet : &{false 404 0001-01-01 00:00:00 +0000 UTC Retriable: false, RetryAfter: 0s, HTTPStatusCode: 404, RawError: {"error":{"code":"ResourceNotFound","message":"The Resource 'Microsoft.Network/virtualNetworks/subnets' under resource group 'CL-EXT-OCP-NPRD-RG' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix"}}}


Upstream bz is raised - https://github.com/kubernetes-sigs/azurefile-csi-driver/issues/571

Comment 2 Jan Safranek 2021-03-04 08:51:57 UTC
We do not ship nor support kubernetes-sigs/azurefile-csi-driver. We only support the CSI interface on Kubernetes/OCP side and it appears that correct CSI call (NodeStage / NodePublish) has reached the CSI driver correctly. Everything else is up to the CSI driver. That's a risk of running community drivers. Still, community support in the github issue you linked has been great so far and I'd suggest the customer to continue there (and listen to Andy Zhang, he's good!)