Bug 1384457

Description of problem:
"volume_quota: 3" was set when creating the heat stack, then scaling up a node after the stack is completed. At last the stack was failed to update due to atomic-openshift-node on the new node can't start.

Version-Release number of selected component (if applicable):
openshift-on-openstack v0.9.2

How reproducible:
100%

Steps to Reproduce:
1.Create a heat stack with "volume_quota: 3"
<--snip-->
  node_count: 1
  master_count: 3
  infra_count: 2
  volume_quota: 3
  container_quota: 3
  deploy_registry: true
<--snip-->
2. Update the stack with "node_count: 2"
3.

Actual results:
Heat stack can't be updated due to openshift-ansible failed.

On the bastion node:

[root@ghuang-verify-bugs-bastion ~]# cat /var/lib/ansible/playbooks/main.yml 
- include: /var/lib/ansible/playbooks/dns.yml


- include: /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml
  vars:
    openshift_infra_nodes: "{{ groups.infra | default([]) }}"


- hosts: masters[0]
  sudo: yes
  tasks:
  - name: Fetch cert file
    fetch:
      src=/etc/origin/master/ca.crt
      dest=/var/run/heat-config/heat-config-script/01637a88-ce4e-492e-99c3-c65183c3533c.ca_cert
      flat=yes
<--snip-->

On the failed node:

Oct 13 06:15:53 ghuang-verify-bugs-openshift-node-b23nh2b8 systemd: Failed to start Atomic OpenShift Node.
Oct 13 06:15:53 ghuang-verify-bugs-openshift-node-b23nh2b8 systemd: Unit atomic-openshift-node.service entered failed state.
Oct 13 06:15:53 ghuang-verify-bugs-openshift-node-b23nh2b8 systemd: atomic-openshift-node.service failed.
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 systemd: atomic-openshift-node.service holdoff time over, scheduling restart.
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 systemd: Starting Atomic OpenShift Node...
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 atomic-openshift-node: I1013 06:15:58.307417   55005 docker.go:329] Start docker client with request timeout=2m0s
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 atomic-openshift-node: I1013 06:15:58.364409   55005 openstack.go:187] Got instance id from /var/lib/cloud/data/instance-id: 039f64f6-7ae4-4948-bac4-8205cefaff98
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 atomic-openshift-node: I1013 06:15:58.364432   55005 node_config.go:274] Successfully initialized cloud provider: "openstack" from the config file: "/etc/origin/cloudprovider/openstack.conf"
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 atomic-openshift-node: I1013 06:15:58.364444   55005 node.go:42] Initializing SDN node of type "redhat/openshift-ovs-subnet" with configured hostname "ghuang-verify-bugs-openshift-node-b23nh2b8.example.com" (IP ""), iptables sync period "30s"
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 atomic-openshift-node: I1013 06:15:58.365228   55005 start_node.go:298] Starting node ghuang-verify-bugs-openshift-node-b23nh2b8.example.com (v3.3.0.35)
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 atomic-openshift-node: I1013 06:15:58.373716   55005 start_node.go:307] Connecting to API server https://ghuang-verify-bugs-lb.example.com:8443
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 atomic-openshift-node: I1013 06:15:58.375179   55005 docker.go:309] Connecting to docker on unix:///var/run/docker.sock
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 atomic-openshift-node: I1013 06:15:58.375198   55005 docker.go:329] Start docker client with request timeout=0
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 docker-current: time="2016-10-13T06:15:58.378683379-04:00" level=info msg="{Action=_ping, LoginUID=4294967295, PID=55005}"
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 atomic-openshift-node: I1013 06:15:58.379212   55005 node.go:138] Connecting to Docker at unix:///var/run/docker.sock
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 atomic-openshift-node: I1013 06:15:58.402608   55005 node.go:217] Replacing empty-dir volume plugin with quota wrapper
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 atomic-openshift-node: F1013 06:15:58.407136   55005 node.go:222] Could not set up local quota, /var/lib/origin/openshift.local.volumes is not on a filesystem mounted with the grpquota option
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 systemd: atomic-openshift-node.service: main process exited, code=exited, status=255/n/a
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 systemd: Failed to start Atomic OpenShift Node.
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 systemd: Unit atomic-openshift-node.service entered failed state.
Oct 13 06:15:58 ghuang-verify-bugs-openshift-node-b23nh2b8 systemd: atomic-openshift-node.service failed.


Expected results:
Scaling up successfully.

Additional info: