Bug 1503906 - fsGroup (chown+chmod) causes time out if storage contains lots of files
Summary: fsGroup (chown+chmod) causes time out if storage contains lots of files
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RFE
Version: 3.3.1
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Eric Paris
QA Contact: Xiaoli Tian
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-10-19 03:12 UTC by Kenjiro Nakayama
Modified: 2023-09-07 18:58 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-06-12 12:49:24 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1459106 0 unspecified CLOSED Deployments timing out if storage contains lots of small files 2023-09-15 00:02:28 UTC
Red Hat Knowledge Base (Solution) 3200662 0 None None None 2019-06-12 12:53:33 UTC

Internal Links: 1459106

Description Kenjiro Nakayama 2017-10-19 03:12:46 UTC
Description of problem:

- Related to bz#1459106, when containers has a lot of files, fsGroup(chmod+chown) causes time out.
- The podAttachAndMountTimeout has fixed value (2min)[1], but it is not configurable.

Version-Release number of selected component (if applicable):
- OCP 3.x (we experienced 3.3)

Steps to Reproduce:
1. Please refer to the description of bz#1459106. However, it might need small file for the fsGroup issue.

Actual results:
- Caused timeout after 2m.

  FirstSeen     LastSeen        Count   From                                                            SubobjectPath   Type            Reason          Message
  ---------     --------        -----   ----                                                            -------------   --------        ------          -------
  2m            2m              1       {default-scheduler }                                                            Normal          Scheduled       Successfully assigned <NODE_NAME>
  14s           14s             1       {kubelet <NODE_NAME>}                     Warning         FailedMount     Unable to mount volumes for pod "jenkins-xxx(xxx)": timeout expired waiting for volumes to attach/mount for pod "jenkins-xxx"/"xxx-xx". list of unattached/unmounted volumes=[jenkins-xxx]

  14s           14s             1       {kubelet <NODE_NAME>}                     Warning         FailedSync      Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "jenkins-xxx"/"xxx-xx". list of unattached/unmounted volumes=[jenkins-data]

-  In the OpenShift Node log, there are tons of chown/chmod logs.

Expected results:
- We wanted to expand podAttachAndMountTimeout.

Additional info:
- Workaround is to remove fsGroup from DC[2].


[1] https://github.com/openshift/origin/blob/release-1.3/vendor/k8s.io/kubernetes/pkg/kubelet/volumemanager/volume_manager.go#L70
[2] https://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/pkg/volume/volume_linux.go#L35-L38

Comment 2 Kenjiro Nakayama 2017-12-28 00:41:46 UTC
I don't think that it is a RFE ticket. Without configurable timeout value, if the users have a lot of files, fsGroup(chown+chgroup) causes timeout and the pod cannot start.

Comment 7 Kirsten Newcomer 2019-06-12 11:59:07 UTC
With the introduction of OpenShift 4, Red Hat has delivered or roadmapped a substantial number of features based on feedback by our customers.  Many of the enhancements encompass specific RFEs which have been requested, or deliver a comparable solution to a customer problem, rendering an RFE redundant.

This bz (RFE) has been identified as a feature request not yet planned or scheduled for an OpenShift release and is being closed. 

If this feature is still an active request that needs to be tracked, Red Hat Support can assist in filing a request in the new JIRA RFE system, as well as provide you with updates as the RFE progress within our planning processes. Please open a new support case: https://access.redhat.com/support/cases/#/case/new 

Opening a New Support Case: https://access.redhat.com/support/cases/#/case/new 

As the new Jira RFE system is not yet public, Red Hat Support can help answer your questions about your RFEs via the same support case system.


Note You need to log in before you can comment on or make changes to this bug.