Bug 1669993 - Could not mount efs volume on openshift worker server
Summary: Could not mount efs volume on openshift worker server
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.1.0
Assignee: Bradley Childs
QA Contact: Chao Yang
Depends On:
TreeView+ depends on / blocked
Reported: 2019-01-28 10:16 UTC by Chao Yang
Modified: 2019-06-04 10:42 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2019-06-04 10:42:19 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 0 None None None 2019-06-04 10:42:25 UTC

Description Chao Yang 2019-01-28 10:16:06 UTC
Description of problem:
Could not mount efs volume from openshift worker server

Version-Release number of selected component (if applicable):
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-0.nightly-2019-01-25-205123   True        False         4h        Cluster version is 4.0.0-0.nightly-2019-01-25-205123

How reproducible:

Steps to Reproduce:
1.Install openshift env
2.Create efs volume from https://us-east-2.console.aws.amazon.com/efs/home?region=us-east-2#/filesystems
3.Opeshift worker security group:
Group ID: sg-0a9f31bc392ec8265
Group Name: terraform-20190128045437340200000006
VPC ID: vpc-0d54a4e6df8542d08
OWNER: 301721915996
Description: Managed by Terraform

Openshift worker inbound rules are as below:

Security Groups associated with i-0391b31f00484f990
Ports	Protocol	Source	terraform-20190128045437340200000006
80	tcp	✔
30000-32767	tcp	sg-091b999268ff1680e, sg-0a9f31bc392ec8265	✔
0	icmp	✔
All	All	sg-032caf68a9ba6cb30	✔
22	tcp	✔
10255	tcp	sg-0081e88f123f3fb7e, sg-0a9f31bc392ec8265	✔
10250	tcp	sg-0081e88f123f3fb7e, sg-0a9f31bc392ec8265	✔
4789	udp	sg-0081e88f123f3fb7e, sg-0a9f31bc392ec8265, sg-0fc3f2a3b2302eb23	✔
4194	tcp	sg-0081e88f123f3fb7e, sg-0a9f31bc392ec8265	✔
443	tcp	✔
9100	tcp	sg-0081e88f123f3fb7e, sg-0a9f31bc392ec8265	✔

VPC: vpc-0d54a4e6df8542d08 - qe-piqin2.qe.devcluster.openshift.com	
Availability Zone: us-east-2a	
Subnet: subnet-001829760f2af164b - qe-piqin2-worker-us-east-2a	
IP address:	
Mount target ID: fsmt-bd3f90c4	
Network interface ID: eni-05ac20712bd213406	
Security groups: sg-0a9f31bc392ec8265 - terraform-20190128045437340200000006
                 sg-032caf68a9ba6cb30 - k8s-elb-a24ac1c3122bb11e98079021de76574f
Mount target state: Available
4.Mount efs volume from openshift worker server
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport fs-0d22bf74.efs.us-east-2.amazonaws.com:/ efs
mount.nfs4: Connection timed out
Actual results:
Efs volume failed to mount

Expected results:
Should mount efs volume successfully
Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 1 Matthew Wong 2019-01-28 14:13:32 UTC
What if NFS (2049) inbound access is added to the EFS security group?

Comment 2 Matthew Wong 2019-01-30 13:45:43 UTC
please check if 2049 is open as shown here https://docs.aws.amazon.com/efs/latest/ug/accessing-fs-create-security-groups.html

Comment 3 Chao Yang 2019-01-31 08:42:44 UTC
After I add the 2049 port into the security group, EFS volume can be mount
80	tcp	✔
30000-32767	tcp	sg-02b3f6605a435d19f, sg-0af3df3a0948c9ef0	✔
0	icmp	✔
All	All	sg-0bd95f490853b5972	✔
22	tcp	✔
10255	tcp	sg-02b3f6605a435d19f, sg-0cadd781d9319a54a	✔
10250	tcp	sg-02b3f6605a435d19f, sg-0cadd781d9319a54a	✔
4789	udp	sg-02b3f6605a435d19f, sg-07d987ba7a62adf53, sg-0cadd781d9319a54a	✔
4194	tcp	sg-02b3f6605a435d19f, sg-0cadd781d9319a54a	✔
443	tcp	✔
9100	tcp	sg-02b3f6605a435d19f, sg-0cadd781d9319a54a	✔
2049	tcp	sg-02b3f6605a435d19f	✔

Comment 4 Chao Yang 2019-01-31 08:46:51 UTC
mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 efs

[root@ip-10-0-130-175 efs]# mount | grep efs
rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) on /var/roothome/efs type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=,local_lock=none,addr=

Comment 9 Chao Yang 2019-04-01 05:32:03 UTC
Marked as verified due to https://bugzilla.redhat.com/show_bug.cgi?id=1669993#c7

Comment 12 errata-xmlrpc 2019-06-04 10:42:19 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.