Bug 1443623

Description of problem:
During the upgrade of a multi-node cluster running OpenShift 3.2 to 3.3 the PV folders on the nfs server were deleted. Before the upgrade projects were existing on the cluster with data stored.

Output of `oc get pv` lists the pvs after the upgrade was as before the upgrade including pvcs for the existing projects.

eg. Bound  <project-name>/<pod>   were still intact after the upgrade

Error on pods after the upgrade:
Unable to mount volumes for pod ..... timeout expired waiting for volumes to attach/mount for pod

Also output for journalctl
Apr 19 10:37:47 ip-<masked>.us-west-1.compute.internal atomic-openshift-node[8853]: I0419 10:37:47.654818    8853 reconciler.go:294] MountVolume operation started for volume "kubernetes.io/secret/3bd31a6a-24ef-11e7-b704-02f860c51e92-router-token-yf68v" (spec.Name: "router-token-yf68v") to pod "3bd31a6a-24ef-11e7-b704-02f860c51e92" (UID: "3bd31a6a-24ef-11e7-b704-02f860c51e92"). Volume is already mounted to pod, but remount was requested.




Version-Release number of selected component (if applicable):
After the upgrade
oc v3.3.1.17
kubernetes v1.3.0+52492b4
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-<masked>.us-west-1.compute.internal:8443
openshift v3.3.1.17
kubernetes v1.3.0+52492b4



How reproducible:
Upgrade a multi-node cluster using nfs server and running Openshift 3.2 to 3.3 using the guide here:
https://docs.openshift.com/container-platform/3.3/install_config/upgrading/automated_upgrades.html

Steps to Workaround:
for i in {51..100}; do sudo mkdir pv${i} ; sudo chown nfsnobody:nfsnobody pv${i}; sudo chmod 777 pv${i}; done
sudo mkdir registry ; sudo chown nfsnobody:nfsnobody registry; sudo chmod 777 registry
sudo service nfs restart

These steps allowed the pods to come back up but with all data still removed.


Actual results:
pv folders on nfs server deleted after upgrade

Expected results:
Upgrade to complete and data remain in the pvs

Additional info:
used the automated upgrade using ansible using the ~/.config/openshift/installer.cfg.yml file