Bug 1294679

Summary: Compute nodes don't mount the NFS shares
Product: Red Hat OpenStack Reporter: Udi Kalifon <ukalifon>
Component: openstack-tripleo-heat-templatesAssignee: Jay Dobies <jason.dobies>
Status: CLOSED NOTABUG QA Contact: Udi Kalifon <ukalifon>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 8.0 (Liberty)CC: calfonso, jcoufal, mburns, rhel-osp-director-maint, sasha, ukalifon
Target Milestone: gaKeywords: Regression
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-01-04 19:47:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Udi Kalifon 2015-12-29 15:44:29 UTC
Description of problem:
I configured cinder to use NFS:

  CinderEnableIscsiBackend: false
  CinderEnableNfsBackend: true
  CinderNfsMountOptions: 'retry=1'
  CinderNfsMountOptions: 'rw,soft,intr,rsize=8192,wsize=8192'
  CinderNfsServers: 'puma49.scl.lab.tlv.redhat.com:/tmp/nfs_export'

The controller nodes mount the NFS share and the created volumes are correctly created on the NFS server. However on the compute nodes, no NFS mounts exist (run the mount command to see all the mounts).

As a result, when you try to attach the volume to an instance - you get no errors but the volume stays unattached.

I looked at the templates and saw that the compute nodes don't accept the same parameters that the controller nodes do. I edited the templates to mimic the controllers configuration on the computes as well, but it didn't fix the problem for me.


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-0.8.7-2.el7ost.noarch


How reproducible:
100%


Steps to Reproduce:
1. Add the described parameters to your customization file
2. Deploy
3. Create an external network and subnet, an internal network and subnet, a router between them and an instance that's attached to the internal network.
4. Create a cinder volume
5. Attach the cinder volume to the instance
6. Try to use the volume in the instance


Actual results:
Only when you try to access the volume from within the instance do you see that it is not attached. Running the command "cinder list" also confirms that the volume is not attached.

Comment 2 Alexander Chuzhoy 2015-12-30 15:52:55 UTC
Worked for me:

Environment:
openstack-tripleo-0.0.7-1.el7ost.noarch
openstack-tripleo-common-0.0.2-4.el7ost.noarch
openstack-tripleo-puppet-elements-0.0.2-1.el7ost.noarch
instack-undercloud-2.2.0-1.el7ost.noarch
openstack-tripleo-heat-templates-0.8.7-2.el7ost.noarch
openstack-tripleo-image-elements-0.9.7-1.el7ost.noarch





[stack@undercloud ~]$ . overcloudrc                                              
[stack@undercloud ~]$ ssh x.x.x.x -l cirros "sudo /sbin/fdisk -l|grep ^/dev"
/dev/vda1   *       16065     2088449     1036192+  83  Linux                    
[stack@undercloud ~]$ cinder list
+----+--------+------------------+------+------+-------------+----------+-------------+-------------+
| ID | Status | Migration Status | Name | Size | Volume Type | Bootable | Multiattach | Attached to |
+----+--------+------------------+------+------+-------------+----------+-------------+-------------+
+----+--------+------------------+------+------+-------------+----------+-------------+-------------+
[stack@undercloud ~]$ cinder create 4                                                                
+---------------------------------------+--------------------------------------+                     
|                Property               |                Value                 |                     
+---------------------------------------+--------------------------------------+                     
|              attachments              |                  []                  |                     
|           availability_zone           |                 nova                 |                     
|                bootable               |                false                 |                     
|          consistencygroup_id          |                 None                 |                     
|               created_at              |      2015-12-30T15:27:21.000000      |                     
|              description              |                 None                 |                     
|               encrypted               |                False                 |                     
|                   id                  | c20b5582-7ecf-4840-9f0a-c3a1e8568be6 |                     
|                metadata               |                  {}                  |                     
|            migration_status           |                 None                 |                     
|              multiattach              |                False                 |                     
|                  name                 |                 None                 |                     
|         os-vol-host-attr:host         |                 None                 |                     
|     os-vol-mig-status-attr:migstat    |                 None                 |                     
|     os-vol-mig-status-attr:name_id    |                 None                 |                     
|      os-vol-tenant-attr:tenant_id     |   83cf8ab57e1a42948cc472e5126792a5   |                     
|   os-volume-replication:driver_data   |                 None                 |                     
| os-volume-replication:extended_status |                 None                 |                     
|           replication_status          |               disabled               |                     
|                  size                 |                  4                   |                     
|              snapshot_id              |                 None                 |                     
|              source_volid             |                 None                 |                     
|                 status                |               creating               |                     
|                user_id                |   61e783edeab647bcb1d9de2d72f126df   |                     
|              volume_type              |                 None                 |                     
+---------------------------------------+--------------------------------------+                     
[stack@undercloud ~]$ cinder list                                                                
+--------------------------------------+-----------+------------------+------+------+-------------+----------+-------------+-------------+
|                  ID                  |   Status  | Migration Status | Name | Size | Volume Type | Bootable | Multiattach | Attached to |
+--------------------------------------+-----------+------------------+------+------+-------------+----------+-------------+-------------+
| c20b5582-7ecf-4840-9f0a-c3a1e8568be6 | available |        -         |  -   |  4   |      -      |  false   |    False    |             |
+--------------------------------------+-----------+------------------+------+------+-------------+----------+-------------+-------------+
[stack@undercloud ~]$ nova list                                                                                                           
nova+--------------------------------------+--------+--------+------------+-------------+-----------------------------------+             
| ID                                   | Name   | Status | Task State | Power State | Networks                          |                 
+--------------------------------------+--------+--------+------------+-------------+-----------------------------------+                 
| 0436fe0c-9cdd-4d10-aafc-10115ef53d37 | nisim1 | ACTIVE | -          | Running     | tenant=192.168.32.5, x.x.x.x |                 
+--------------------------------------+--------+--------+------------+-------------+-----------------------------------+                 
 [stack@undercloud ~]$ nova volume-attach nisim1 c20b5582-7ecf-4840-9f0a-c3a1e8568be6
+----------+--------------------------------------+                                  
| Property | Value                                |                                  
+----------+--------------------------------------+                                  
| device   | /dev/vdb                             |                                  
| id       | c20b5582-7ecf-4840-9f0a-c3a1e8568be6 |                                  
| serverId | 0436fe0c-9cdd-4d10-aafc-10115ef53d37 |                                  
| volumeId | c20b5582-7ecf-4840-9f0a-c3a1e8568be6 |                                  
+----------+--------------------------------------+                             

[stack@undercloud ~]$ ssh x.x.x.x -l cirros "sudo /sbin/fdisk -l|grep ^/dev"
/dev/vda1   *       16065     2088449     1036192+  83  Linux                    
Disk /dev/vdb doesn't contain a valid partition table                            
[stack@undercloud ~]$ ssh x.x.x.x -l cirros -t sudo -i
# fdisk -l                                                 

Disk /dev/vda: 1073 MB, 1073741824 bytes
255 heads, 63 sectors/track, 130 cylinders, total 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes                           
Sector size (logical/physical): 512 bytes / 512 bytes            
I/O size (minimum/optimal): 512 bytes / 512 bytes                
Disk identifier: 0x00000000                                      

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *       16065     2088449     1036192+  83  Linux 

Disk /dev/vdb: 4294 MB, 4294967296 bytes
16 heads, 63 sectors/track, 8322 cylinders, total 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes                           
Sector size (logical/physical): 512 bytes / 512 bytes            
I/O size (minimum/optimal): 512 bytes / 512 bytes                
Disk identifier: 0x00000000                                      

Disk /dev/vdb doesn't contain a valid partition table
# fdisk /dev/vdb                                     
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xfd7e1012.                     
Changes will remain in memory only, until you decide to write them.               
After that, of course, the previous content won't be recoverable.                 

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): p

Disk /dev/vdb: 4294 MB, 4294967296 bytes
16 heads, 63 sectors/track, 8322 cylinders, total 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes                           
Sector size (logical/physical): 512 bytes / 512 bytes            
I/O size (minimum/optimal): 512 bytes / 512 bytes                
Disk identifier: 0xfd7e1012                                      

   Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Partition type:        
   p   primary (0 primary, 0 extended, 4 free)
   e   extended                               
Select (default p): p                         
Partition number (1-4, default 1):            
Using default value 1                         
First sector (2048-8388607, default 2048):    
Using default value 2048                      
Last sector, +sectors or +size{K,M,G} (2048-8388607, default 8388607): 
Using default value 8388607                                            

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.                             
# p                                        
-sh: p: not found                          
# fdisk /dev/vdb

Command (m for help): q

# Connection to x.x.x.x closed.
[stack@undercloud ~]$ ssh x.x.x.x -l cirros "sudo /sbin/fdisk -l|grep ^/dev"
/dev/vda1   *       16065     2088449     1036192+  83  Linux                    
/dev/vdb1            2048     8388607     4193280   83  Linux




[stack@undercloud ~]$ cinder list   
cinder show +--------------------------------------+--------+------------------+------+------+-------------+----------+-------------+--------------------------------------+
|                  ID                  | Status | Migration Status | Name | Size | Volume Type | Bootable | Multiattach |             Attached to              |            
+--------------------------------------+--------+------------------+------+------+-------------+----------+-------------+--------------------------------------+            
| c20b5582-7ecf-4840-9f0a-c3a1e8568be6 | in-use |        -         |  -   |  4   |      -      |  false   |    False    | 0436fe0c-9cdd-4d10-aafc-10115ef53d37 |            
+--------------------------------------+--------+------------------+------+------+-------------+----------+-------------+--------------------------------------+            
[stack@undercloud ~]$ cinder show c20b5582-7ecf-4840-9f0a-c3a1e8568be6                                                                                                      
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+                                                                                                                       
|                Property               |                                                                                                                                  Value                                                                                                                                  |                                                                                                                       
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+                                                                                                                       
|              attachments              | [{u'server_id': u'0436fe0c-9cdd-4d10-aafc-10115ef53d37', u'attachment_id': u'3325be4d-a91f-4259-90a2-0d83ac34800c', u'host_name': None, u'volume_id': u'c20b5582-7ecf-4840-9f0a-c3a1e8568be6', u'device': u'/dev/vdb', u'id': u'c20b5582-7ecf-4840-9f0a-c3a1e8568be6'}] |                                                                                                                       
|           availability_zone           |                                                                                                                                   nova                                                                                                                                  |                                                                                                                       
|                bootable               |                                                                                                                                  false                                                                                                                                  |                                                                                                                       
|          consistencygroup_id          |                                                                                                                                   None                                                                                                                                  |                                                                                                                       
|               created_at              |                                                                                                                        2015-12-30T15:27:21.000000                                                                                                                       |                                                                                                                       
|              description              |                                                                                                                                   None                                                                                                                                  |                                                                                                                       
|               encrypted               |                                                                                                                                  False                                                                                                                                  |                                                                                                                       
|                   id                  |                                                                                                                   c20b5582-7ecf-4840-9f0a-c3a1e8568be6                                                                                                                  |                                                                                                                       
|                metadata               |                                                                                                             {u'readonly': u'False', u'attached_mode': u'rw'}                                                                                                            |                                                                                                                       
|            migration_status           |                                                                                                                                   None                                                                                                                                  |                                                                                                                       
|              multiattach              |                                                                                                                                  False                                                                                                                                  |                                                                                                                       
|                  name                 |                                                                                                                                   None                                                                                                                                  |                                                                                                                       
|         os-vol-host-attr:host         |                                                                                                        overcloud-controller-0.localdomain@tripleo_nfs#tripleo_nfs                                                                                                       |                                                                                                                       
|     os-vol-mig-status-attr:migstat    |                                                                                                                                   None                                                                                                                                  |
|     os-vol-mig-status-attr:name_id    |                                                                                                                                   None                                                                                                                                  |
|      os-vol-tenant-attr:tenant_id     |                                                                                                                     83cf8ab57e1a42948cc472e5126792a5                                                                                                                    |
|   os-volume-replication:driver_data   |                                                                                                                                   None                                                                                                                                  |
| os-volume-replication:extended_status |                                                                                                                                   None                                                                                                                                  |
|           replication_status          |                                                                                                                                 disabled                                                                                                                                |
|                  size                 |                                                                                                                                    4                                                                                                                                    |
|              snapshot_id              |                                                                                                                                   None                                                                                                                                  |
|              source_volid             |                                                                                                                                   None                                                                                                                                  |
|                 status                |                                                                                                                                  in-use                                                                                                                                 |
|                user_id                |                                                                                                                     61e783edeab647bcb1d9de2d72f126df                                                                                                                    |
|              volume_type              |                                                                                                                                   None                                                                                                                                  |
+---------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+




[root@overcloud-novacompute-0 ~]# mount|grep :
x.x.x.x:/shared/cinder on /var/lib/nova/mnt/7f9a58221c3a8ec0aecefe17b87c6a22 type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.20,local_lock=none,addr=x.x.x.x)

Comment 3 chris alfonso 2016-01-04 17:37:21 UTC
Udi, please recheck this, as it's working for Sasha.

Comment 4 Alexander Chuzhoy 2016-01-04 19:44:39 UTC
This was due to missing perms on the NFS server.

Once I added "no_root_squash" to the export pamams - the issue was gone.

Comment 5 Mike Burns 2016-01-04 19:47:12 UTC
closing per comment 4

Comment 6 Udi Kalifon 2016-01-18 12:15:07 UTC
Closing the needinfo.