Bug 1349727 - Pod will not be running if using ebs volume second time with different fsType
Summary: Pod will not be running if using ebs volume second time with different fsType
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Documentation
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: brice
QA Contact: Chao Yang
Vikram Goyal
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-06-24 06:20 UTC by Chao Yang
Modified: 2016-10-24 11:54 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-07-28 00:07:56 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Chao Yang 2016-06-24 06:20:37 UTC
Document URL: 
https://docs.openshift.org/latest/install_config/persistent_storage/persistent_storage_aws.html

Section Number and Name: 
Volume Format
Describe the issue: 
In the openshift, pod will not be running if volume changed fsType which already formatted as other fstype
1.Create a volume
2.Create a pv using this volume id
apiVersion: "v1"
kind: "PersistentVolume"
metadata:
  name: "pv0001"
spec:
  capacity:
    storage: "1Gi"
  accessModes:
    - "ReadWriteOnce"
  awsElasticBlockStore:
    fsType: "exgt4"
    volumeID: "vol-61c723c5"
3. Create a pvc and pod
4. Pod is running
5. Delete pod, pvc and pv
6. From aws web console, this volume is available
7. Create pv using this volume, set fsType is xfs in the pv yaml file
8. Create pvc and pod
9. Pod will be failed
Error message like 
Mounting arguments: /dev/xvdba /var/lib/origin/openshift.local.volumes/plugins/kubernetes.io/aws-ebs/mounts/vol-61c723c5 xfs [defaults]
Output: mount: wrong fs type, bad option, bad superblock on /dev/xvdba,
       missing codepage or helper program, or other error

But doc said "However, it can result in data loss if value of the fsType parameter is changed after the volume was already formatted and provisioned with data."
Suggestions for improvement: 

Additional information:

Comment 5 Jan Safranek 2016-07-04 08:06:20 UTC
> But format volume from ext3 to ext4 and mount volume is OK on the openshift.

ext4 is backward compatible with ext3, i.e. devices formatted with ext3 can be mounted by ext4 kernel driver.

As Brice pointed out, changing fsType is a dangerous operation and IMO we should *not* blindly reformat user data with another filesystem. There may be precious data on the device!

Comment 6 brice 2016-07-06 03:27:10 UTC
OK. I think I'm understanding this correctly. 

I've submitted a PR for this:

https://github.com/openshift/openshift-docs/pull/2469

Chao Yang, do you think this is enough to fulfil this BZ, or do you think extra is needed?

Thanks for the extra information, all!

Comment 7 Chao Yang 2016-07-07 09:24:34 UTC
I don't think pod should fail when re-formated volume according to https://bugzilla.redhat.com/show_bug.cgi?id=1349727#c5
so I will assign this bug to storage team

Comment 8 Jan Safranek 2016-07-07 11:39:20 UTC
Chao, Kubernetes/OpenShift *must* fail here otherwise it will overwrite user data. It could fail with a better message, but it should not reformat the volume.

Comment 9 brice 2016-07-08 01:02:37 UTC
OK. It sounds like what I've documented is still correct, but I'll not merge yet if we're still thinking about the direction. If this is indeed a product BZ and not a docs BZ, it might be worthwhile to merge it the PR.

Please let me know when we come to a decision.

Comment 10 Chao Yang 2016-07-11 05:44:44 UTC
Hi Jan, I will re-assign this bug to Doc team and open a new enhanced bug for storage team to update to a better message when reformat the volume, what do you think?

Comment 11 Chao Yang 2016-07-14 07:32:03 UTC
Hi brice, I will assigh this to you according to the https://bugzilla.redhat.com/show_bug.cgi?id=1349727#c8

Comment 12 brice 2016-07-15 00:28:49 UTC
Chao, OK, I'll go ahead with this. I agree creating another BZ for the product is a good idea.

Thanks for the information.

Comment 13 openshift-github-bot 2016-07-18 02:20:21 UTC
Commit pushed to master at https://github.com/openshift/openshift-docs

https://github.com/openshift/openshift-docs/commit/1e662b262da881392fc83fae1b0015b1712af4ff
Merge pull request #2469 from bfallonf/bz1349727

Bug 1349727 added important box about changing fstype field


Note You need to log in before you can comment on or make changes to this bug.