Attachment 1316147 Details for Bug 1483466 – nova log

nova log

novalog.txt (text/plain), 17.76 KB, created by Eran Kuris on 2017-08-21 08:11:33 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Eran Kuris

Created: 2017-08-21 08:11:33 UTC

Size: 17.76 KB

patch

obsolete

>[root@compute-0 ~]# tail -f /var/log/nova/nova-compute.log 
>2017-08-21 08:01:07.224 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
>2017-08-21 08:01:07.224 2179 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
>2017-08-21 08:01:07.224 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
>2017-08-21 08:01:07.224 2179 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
>2017-08-21 08:01:07.224 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
>2017-08-21 08:01:07.224 2179 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
>2017-08-21 08:01:07.224 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, in createWithFlags
>2017-08-21 08:01:07.224 2179 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
>2017-08-21 08:01:07.224 2179 ERROR oslo_messaging.rpc.server libvirtError: Unable to delete file /var/lib/nova/instances/8d448c2b-1d76-435d-a885-e45ef2c43a46/console.log: Permission denied
>2017-08-21 08:01:07.224 2179 ERROR oslo_messaging.rpc.server 
>
>
>2017-08-21 08:01:54.195 2179 INFO nova.compute.resource_tracker [req-6bf767d5-dec7-4901-bf2f-34575e1d66a1 - - - - -] Auditing locally available compute resources for node compute-0.localdomain
>2017-08-21 08:01:54.568 2179 WARNING nova.scheduler.client.report [req-6bf767d5-dec7-4901-bf2f-34575e1d66a1 - - - - -] Unable to refresh my resource provider record
>2017-08-21 08:01:54.741 2179 INFO nova.compute.resource_tracker [req-6bf767d5-dec7-4901-bf2f-34575e1d66a1 - - - - -] Total usable vcpus: 2, total allocated vcpus: 4
>2017-08-21 08:01:54.742 2179 INFO nova.compute.resource_tracker [req-6bf767d5-dec7-4901-bf2f-34575e1d66a1 - - - - -] Final resource view: name=compute-0.localdomain phys_ram=12287MB used_ram=6144MB phys_disk=49GB used_disk=20GB total_vcpus=2 used_vcpus=4 pci_stats=[]
>2017-08-21 08:01:54.787 2179 WARNING nova.scheduler.client.report [req-6bf767d5-dec7-4901-bf2f-34575e1d66a1 - - - - -] Unable to refresh my resource provider record
>2017-08-21 08:01:54.788 2179 INFO nova.compute.resource_tracker [req-6bf767d5-dec7-4901-bf2f-34575e1d66a1 - - - - -] Compute_service record updated for compute-0.localdomain:compute-0.localdomain
>2017-08-21 08:01:57.168 2179 INFO nova.virt.libvirt.driver [-] [instance: 8d448c2b-1d76-435d-a885-e45ef2c43a46] During wait destroy, instance disappeared.
>2017-08-21 08:01:57.171 2179 WARNING nova.virt.osinfo [req-6e9d3221-8f10-4cf2-b0cf-18d5a4f0225e e38dfa89f2b84b29b263b57e4239699f 6751ce461b51462e8486c0d8cca9c3ec - - -] Cannot find OS information - Reason: (No configuration information found for operating system Empty)
>2017-08-21 08:01:57.241 2179 WARNING nova.virt.osinfo [req-6e9d3221-8f10-4cf2-b0cf-18d5a4f0225e e38dfa89f2b84b29b263b57e4239699f 6751ce461b51462e8486c0d8cca9c3ec - - -] Cannot find OS information - Reason: (No configuration information found for operating system Empty)
>2017-08-21 08:01:57.678 2179 INFO os_vif [req-6e9d3221-8f10-4cf2-b0cf-18d5a4f0225e e38dfa89f2b84b29b263b57e4239699f 6751ce461b51462e8486c0d8cca9c3ec - - -] Successfully plugged vif VIFBridge(active=False,address=fa:16:3e:7e:fd:b8,bridge_name='qbr16fb572c-2a',has_traffic_filtering=True,id=16fb572c-2a91-497e-b753-02d1c9ea1f95,network=Network(df08a89a-2b0a-4940-8981-1b842b979763),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap16fb572c-2a')
>2017-08-21 08:01:57.794 2179 ERROR nova.virt.libvirt.guest [req-6e9d3221-8f10-4cf2-b0cf-18d5a4f0225e e38dfa89f2b84b29b263b57e4239699f 6751ce461b51462e8486c0d8cca9c3ec - - -] Error launching a defined domain with XML: <domain type='kvm'>
>  <name>instance-00000003</name>
>  <uuid>8d448c2b-1d76-435d-a885-e45ef2c43a46</uuid>
>  <metadata>
>    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
>      <nova:package version="14.0.7-11.el7ost"/>
>      <nova:name>VM1</nova:name>
>      <nova:creationTime>2017-08-21 08:01:57</nova:creationTime>
>      <nova:flavor name="m1.medium">
>        <nova:memory>2048</nova:memory>
>        <nova:disk>10</nova:disk>
>        <nova:swap>0</nova:swap>
>        <nova:ephemeral>0</nova:ephemeral>
>        <nova:vcpus>2</nova:vcpus>
>      </nova:flavor>
>      <nova:owner>
>        <nova:user uuid="e38dfa89f2b84b29b263b57e4239699f">admin</nova:user>
>        <nova:project uuid="6751ce461b51462e8486c0d8cca9c3ec">admin</nova:project>
>      </nova:owner>
>      <nova:root type="image" uuid="9cff87f2-045b-4a9e-b123-66d949f39bba"/>
>    </nova:instance>
>  </metadata>
>  <memory unit='KiB'>2097152</memory>
>  <currentMemory unit='KiB'>2097152</currentMemory>
>  <vcpu placement='static'>2</vcpu>
>  <cputune>
>    <shares>2048</shares>
>  </cputune>
>  <sysinfo type='smbios'>
>    <system>
>      <entry name='manufacturer'>Red Hat</entry>
>      <entry name='product'>OpenStack Compute</entry>
>      <entry name='version'>14.0.7-11.el7ost</entry>
>      <entry name='serial'>aeb90f2d-4a18-4516-8605-d9e7fd12f87f</entry>
>      <entry name='uuid'>8d448c2b-1d76-435d-a885-e45ef2c43a46</entry>
>      <entry name='family'>Virtual Machine</entry>
>    </system>
>  </sysinfo>
>  <os>
>    <type arch='x86_64' machine='pc-i440fx-rhel7.4.0'>hvm</type>
>    <boot dev='hd'/>
>    <smbios mode='sysinfo'/>
>  </os>
>  <features>
>    <acpi/>
>    <apic/>
>  </features>
>  <cpu mode='host-model' check='partial'>
>    <model fallback='allow'/>
>    <topology sockets='2' cores='1' threads='1'/>
>  </cpu>
>  <clock offset='utc'>
>    <timer name='pit' tickpolicy='delay'/>
>    <timer name='rtc' tickpolicy='catchup'/>
>    <timer name='hpet' present='no'/>
>  </clock>
>  <on_poweroff>destroy</on_poweroff>
>  <on_reboot>restart</on_reboot>
>  <on_crash>destroy</on_crash>
>  <devices>
>    <emulator>/usr/libexec/qemu-kvm</emulator>
>    <disk type='file' device='disk'>
>      <driver name='qemu' type='qcow2' cache='none'/>
>      <source file='/var/lib/nova/instances/8d448c2b-1d76-435d-a885-e45ef2c43a46/disk'/>
>      <target dev='vda' bus='virtio'/>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
>    </disk>
>    <controller type='usb' index='0' model='piix3-uhci'>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
>    </controller>
>    <controller type='pci' index='0' model='pci-root'/>
>    <interface type='bridge'>
>      <mac address='fa:16:3e:7e:fd:b8'/>
>      <source bridge='qbr16fb572c-2a'/>
>      <target dev='tap16fb572c-2a'/>
>      <model type='virtio'/>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
>    </interface>
>    <serial type='file'>
>      <source path='/var/lib/nova/instances/8d448c2b-1d76-435d-a885-e45ef2c43a46/console.log'/>
>      <target port='0'/>
>    </serial>
>    <serial type='pty'>
>      <target port='1'/>
>    </serial>
>    <console type='file'>
>      <source path='/var/lib/nova/instances/8d448c2b-1d76-435d-a885-e45ef2c43a46/console.log'/>
>      <target type='serial' port='0'/>
>    </console>
>    <input type='tablet' bus='usb'>
>      <address type='usb' bus='0' port='1'/>
>    </input>
>    <input type='mouse' bus='ps2'/>
>    <input type='keyboard' bus='ps2'/>
>    <graphics type='vnc' port='-1' autoport='yes' listen='172.17.1.19' keymap='en-us'>
>      <listen type='address' address='172.17.1.19'/>
>    </graphics>
>    <video>
>      <model type='cirrus' vram='16384' heads='1' primary='yes'/>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
>    </video>
>    <memballoon model='virtio'>
>      <stats period='10'/>
>      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
>    </memballoon>
>  </devices>
></domain>
>
>2017-08-21 08:01:57.795 2179 ERROR nova.virt.libvirt.driver [req-6e9d3221-8f10-4cf2-b0cf-18d5a4f0225e e38dfa89f2b84b29b263b57e4239699f 6751ce461b51462e8486c0d8cca9c3ec - - -] [instance: 8d448c2b-1d76-435d-a885-e45ef2c43a46] Failed to start libvirt guest
>2017-08-21 08:01:57.869 2179 INFO os_vif [req-6e9d3221-8f10-4cf2-b0cf-18d5a4f0225e e38dfa89f2b84b29b263b57e4239699f 6751ce461b51462e8486c0d8cca9c3ec - - -] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:7e:fd:b8,bridge_name='qbr16fb572c-2a',has_traffic_filtering=True,id=16fb572c-2a91-497e-b753-02d1c9ea1f95,network=Network(df08a89a-2b0a-4940-8981-1b842b979763),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tap16fb572c-2a')
>2017-08-21 08:01:58.089 2179 INFO nova.compute.manager [req-6e9d3221-8f10-4cf2-b0cf-18d5a4f0225e e38dfa89f2b84b29b263b57e4239699f 6751ce461b51462e8486c0d8cca9c3ec - - -] [instance: 8d448c2b-1d76-435d-a885-e45ef2c43a46] Successfully reverted task state from powering-on on failure for instance.
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server [req-6e9d3221-8f10-4cf2-b0cf-18d5a4f0225e e38dfa89f2b84b29b263b57e4239699f 6751ce461b51462e8486c0d8cca9c3ec - - -] Exception during message handling
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 75, in wrapped
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     self.force_reraise()
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 188, in decorated_function
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     self.force_reraise()
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 157, in decorated_function
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/utils.py", line 613, in decorated_function
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 216, in decorated_function
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     self.force_reraise()
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 204, in decorated_function
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2480, in start_instance
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     self._power_on(context, instance)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2450, in _power_on
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     block_device_info)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2455, in power_on
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     self._hard_reboot(context, instance, network_info, block_device_info)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2344, in _hard_reboot
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     vifs_already_plugged=True)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4948, in _create_domain_and_network
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     destroy_disks_on_failure)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     self.force_reraise()
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4920, in _create_domain_and_network
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     post_xml_callback=post_xml_callback)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4838, in _create_domain
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     guest.launch(pause=pause)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     self._encoded_xml, errors='ignore')
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     self.force_reraise()
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     return self._domain.createWithFlags(flags)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1069, in createWithFlags
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server libvirtError: Unable to delete file /var/lib/nova/instances/8d448c2b-1d76-435d-a885-e45ef2c43a46/console.log: Permission denied
>2017-08-21 08:01:58.101 2179 ERROR oslo_messaging.rpc.server 
>

Actions: View

Attachments on bug 1483466: 1316147