Bug 1376064

Summary: rhel-osp-director: Unable to launch instance, Block format 'raw' does not support the option 'password-secret'
Product: Red Hat OpenStack Reporter: Alexander Chuzhoy <sasha>
Component: rhosp-directorAssignee: Angus Thomas <athomas>
Status: CLOSED DUPLICATE QA Contact: Omri Hochman <ohochman>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: berrange, dbecker, mburns, mcornea, morazi, rhel-osp-director-maint
Target Milestone: ga   
Target Release: 10.0 (Newton)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-09-15 15:33:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Alexander Chuzhoy 2016-09-14 14:56:33 UTC
2016-09-14 14:44:28.447 29392 ERROR nova.virt.libvirt.guest [req-ee39700f-b5a6-4aed-9bb6-37ab09256b24 c7e08bf99b1f46f587f849c8e7603b8e cd166989ce974c6181b5b456ecadde65 - - -] Error launching a defined domain with XML: <domain type='kvm'>                                                                                                                                                                                             
  <name>instance-0000000b</name>                                                                                                                                                                                     
  <uuid>8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6</uuid>                                                                                                                                                                  
  <metadata>                                                                                                                                                                                                         
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">                                                                                                                                         
      <nova:package version="14.0.0-0.20160823074012.21312bf.el7ost"/>                                                                                                                                               
      <nova:name>vm1</nova:name>                                                                                                                                                                                     
      <nova:creationTime>2016-09-14 14:44:26</nova:creationTime>                                                                                                                                                     
      <nova:flavor name="m1.tiny">                                                                                                                                                                                   
        <nova:memory>512</nova:memory>                                                                                                                                                                               
        <nova:disk>1</nova:disk>                                                                                                                                                                                     
        <nova:swap>0</nova:swap>                                                                                                                                                                                     
        <nova:ephemeral>0</nova:ephemeral>                                                                                                                                                                           
        <nova:vcpus>1</nova:vcpus>                                                                                                                                                                                   
      </nova:flavor>                                                                                                                                                                                                 
      <nova:owner>                                                                                                                                                                                                   
        <nova:user uuid="c7e08bf99b1f46f587f849c8e7603b8e">master</nova:user>                                                                                                                                        
        <nova:project uuid="cd166989ce974c6181b5b456ecadde65">master</nova:project>                                                                                                                                  
      </nova:owner>                                                                                                                                                                                                  
      <nova:root type="image" uuid="eb717993-317b-4853-8139-65bf9ba939d5"/>                                                                                                                                          
    </nova:instance>                                                                                                                                                                                                 
  </metadata>                                                                                                                                                                                                        
  <memory unit='KiB'>524288</memory>                                                                                                                                                                                 
  <currentMemory unit='KiB'>524288</currentMemory>                                                                                                                                                                   
  <vcpu placement='static'>1</vcpu>                                                                                                                                                                                  
  <cputune>                                                                                                                                                                                                          
    <shares>1024</shares>                                                                                                                                                                                            
  </cputune>                                                                                                                                                                                                         
  <sysinfo type='smbios'>                                                                                                                                                                                            
    <system>                                                                                                                                                                                                         
      <entry name='manufacturer'>Red Hat</entry>                                                                                                                                                                     
      <entry name='product'>OpenStack Compute</entry>                                                                                                                                                                
      <entry name='version'>14.0.0-0.20160823074012.21312bf.el7ost</entry>                                                                                                                                           
      <entry name='serial'>1a713901-c563-4931-9910-0b380166f1dc</entry>                                                                                                                                              
      <entry name='uuid'>8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6</entry>                                                                                                                                                
      <entry name='family'>Virtual Machine</entry>                                                                                                                                                                   
    </system>                                                                                                                                                                                                        
  </sysinfo>                                                                                                                                                                                                         
  <os>                                                                                                                                                                                                               
    <type arch='x86_64' machine='pc-i440fx-rhel7.3.0'>hvm</type>                                                                                                                                                     
    <boot dev='hd'/>                                                                                                                                                                                                 
    <smbios mode='sysinfo'/>                                                                                                                                                                                         
  </os>                                                                                                                                                                                                              
  <features>                                                                                                                                                                                                         
    <acpi/>                                                                                                                                                                                                          
    <apic/>                                                                                                                                                                                                          
  </features>                                                                                                                                                                                                        
  <cpu mode='host-model'>                                                                                                                                                                                            
    <model fallback='allow'/>                                                                                                                                                                                        
    <topology sockets='1' cores='1' threads='1'/>                                                                                                                                                                    
  </cpu>                                                                                                                                                                                                             
  <clock offset='utc'>                                                                                                                                                                                               
    <timer name='pit' tickpolicy='delay'/>                                                                                                                                                                           
    <timer name='rtc' tickpolicy='catchup'/>                                                                                                                                                                         
    <timer name='hpet' present='no'/>                                                                                                                                                                                
  </clock>                                                                                                                                                                                                           
  <on_poweroff>destroy</on_poweroff>                                                                                                                                                                                 
  <on_reboot>restart</on_reboot>                                                                                                                                                                                     
  <on_crash>destroy</on_crash>                                                                                                                                                                                       
  <devices>                                                                                                                                                                                                          
    <emulator>/usr/libexec/qemu-kvm</emulator>                                                                                                                                                                       
    <disk type='network' device='disk'>                                                                                                                                                                              
      <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>                                                                                                                                             
      <auth username='openstack'>                                                                                                                                                                                    
        <secret type='ceph' uuid='74c2353a-7a1a-11e6-90da-5254003ec994'/>                                                                                                                                            
      </auth>                                                                                                                                                                                                        
      <source protocol='rbd' name='vms/8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6_disk'>                                                                                                                                   
        <host name='10.19.95.15' port='6789'/>                                                                                                                                                                       
        <host name='10.19.95.19' port='6789'/>                                                                                                                                                                       
        <host name='10.19.95.26' port='6789'/>                                                                                                                                                                       
      </source>                                                                                                                                                                                                      
      <target dev='vda' bus='virtio'/>                                                                                                                                                                               
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>                                                                                                                                    
    </disk>                                                                                                                                                                                                          
    <controller type='usb' index='0'>                                                                                                                                                                                
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>                                                                                                                                    
    </controller>                                                                                                                                                                                                    
    <controller type='pci' index='0' model='pci-root'/>                                                                                                                                                              
    <interface type='bridge'>                                                                                                                                                                                        
      <mac address='fa:16:3e:35:a1:a7'/>                                                                                                                                                                             
      <source bridge='qbr4db8b2f0-96'/>                                                                                                                                                                              
      <target dev='tap4db8b2f0-96'/>                                                                                                                                                                                 
      <model type='virtio'/>                                                                                                                                                                                         
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>                                                                                                                                    
    </interface>                                                                                                                                                                                                     
    <serial type='file'>                                                                                                                                                                                             
      <source path='/var/lib/nova/instances/8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6/console.log'/>                                                                                                                      
      <target port='0'/>                                                                                                                                                                                             
    </serial>                                                                                                                                                                                                        
    <serial type='pty'>                                                                                                                                                                                              
      <target port='1'/>                                                                                                                                                                                             
    </serial>                                                                                                                                                                                                        
    <console type='file'>                                                                                                                                                                                            
      <source path='/var/lib/nova/instances/8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6/console.log'/>                                                                                                                      
      <target type='serial' port='0'/>                                                                                                                                                                               
    </console>                                                                                                                                                                                                       
    <input type='tablet' bus='usb'>                                                                                                                                                                                  
      <address type='usb' bus='0' port='1'/>                                                                                                                                                                         
    </input>                                                                                                                                                                                                         
    <input type='mouse' bus='ps2'/>                                                                                                                                                                                  
    <input type='keyboard' bus='ps2'/>                                                                                                                                                                               
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='en-us'>                                                                                                                                   
      <listen type='address' address='0.0.0.0'/>                                                                                                                                                                     
    </graphics>                                                                                                                                                                                                      
    <video>                                                                                                                                                                                                          
      <model type='cirrus' vram='16384' heads='1' primary='yes'/>                                                                                                                                                    
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>                                                                                                                                    
    </video>                                                                                                                                                                                                         
    <memballoon model='virtio'>                                                                                                                                                                                      
      <stats period='10'/>                                                                                                                                                                                           
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>                                                                                                                                    
    </memballoon>                                                                                                                                                                                                    
  </devices>                                                                                                                                                                                                         
</domain>                                                                                                                                                                                                            

2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [req-ee39700f-b5a6-4aed-9bb6-37ab09256b24 c7e08bf99b1f46f587f849c8e7603b8e cd166989ce974c6181b5b456ecadde65 - - -] [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6] Instance failed to spawn                                                                                                                                                                                     
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6] Traceback (most recent call last):                                                                         
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2075, in _build_resources          
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]     yield resources                                                                                        
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1919, in _build_and_run_instance   
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]     block_device_info=block_device_info)                                                                   
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2583, in spawn                 
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]     post_xml_callback=gen_confdrive)                                                                       
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4812, in _create_domain_and_network                                                                                                                                                                                                                 
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]     post_xml_callback=post_xml_callback)                                                                   
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4741, in _create_domain        
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]     guest.launch(pause=pause)                                                                              
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch                  
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]     self._encoded_xml, errors='ignore')                                                                    
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__                    
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]     self.force_reraise()                                                                                   
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise               
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]     six.reraise(self.type_, self.value, self.tb)                                                           
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch                  
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]     return self._domain.createWithFlags(flags)                                                             
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit                             
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]     result = proxy_call(self._autowrap, f, *args, **kwargs)                                                
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call                       
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]     rv = execute(f, *args, **kwargs)                                                                       
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute                          
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]     six.reraise(c, e, tb)                                                                                  
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker                           
rhel-osp-director:   Unable to launch instance
Environment:
openstack-nova-compute-14.0.0-0.20160823074012.21312bf.el7ost.noarch
openstack-puppet-modules-9.0.0-0.20160802183056.8c758d6.el7ost.noarch
instack-undercloud-5.0.0-0.20160818065636.41ef775.el7ost.noarch
openstack-tripleo-heat-templates-5.0.0-0.20160823140311.72404b.1.el7ost.noarch

Steps to reproduce:
1. Deploy overcloud with:
openstack overcloud deploy --templates --control-scale 3 --control-flavor control --compute-scale 2 --compute-flavor compute --ceph-storage-scale 3 --ceph-storage-flavor ceph-storage --neutron-network-type vxlan --neutron-tunnel-types vxlan -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml  -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e network-environment.yaml --ntp-server clock.redhat.com -e ceph-layout.yaml


2. 
Populate the overcloud with networks, image for vm.
3.
Attempt to launch a vm.


Result.

The launched VM ends up with status Error.

Looking in /var/log/nova/nova-compute.log, I see this error:

2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]     rv = meth(*args, **kwargs)                                                                             
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1065, in createWithFlags                      
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)                       
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6] libvirtError: internal error: qemu unexpectedly closed the monitor: -vnc 0.0.0.0:0 -k en-us -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6] 2016-09-14T14:44:28.230977Z qemu-kvm: -drive file=rbd:vms/8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6_disk:id=openstack:auth_supported=cephx\;none:mon_host=10.19.95.15\:6789\;10.19.95.19\:6789\;10.19.95.26\:6789,password-secret=virtio-disk0-secret0,format=raw,if=none,id=drive-virtio-disk0,cache=writeback,discard=unmap: Block format 'raw' does not support the option 'password-secret'
2016-09-14 14:44:28.448 29392 ERROR nova.compute.manager [instance: 8ea0ec40-cdf0-453b-99ed-3c5950bcfbc6]


Expected result:
Successfully launched instance.

Comment 2 Daniel Berrangé 2016-09-14 15:48:14 UTC
No info on livirt version is shown, but I'm guessing the reporter is using an outdated build from 7.3, as this issue is fixed in libvirt-2.0.0-6.el7 as per https://bugzilla.redhat.com/show_bug.cgi?id=1182074 comment 24

Comment 3 Alexander Chuzhoy 2016-09-14 18:12:13 UTC
Was able to launch instance after upgrading the libvirt* on compute to libvirt-2.0.0-9

Comment 4 Mike Burns 2016-09-15 15:33:11 UTC
This is fixed in libvirt as sasha and danpb noted.  We're going to carry the newer libvirt for our beta.  

Since this is fixed in RHEL, I'm going to mark it a duplicate.

*** This bug has been marked as a duplicate of bug 1182074 ***