Bug 436328 - Migration doesn't work with RHEL4 VMs on RHEL5 Dom0s
Summary: Migration doesn't work with RHEL4 VMs on RHEL5 Dom0s
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: xen
Version: 5.1
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: ---
Assignee: Xen Maintainance List
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2008-03-06 15:30 UTC by Ege Turgay
Modified: 2009-12-14 21:14 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2008-03-07 15:44:30 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Ege Turgay 2008-03-06 15:30:12 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.12) Gecko/20071122 Red Hat/1.5.0.12-7.el5 Firefox/1.5.0.12

Description of problem:
While trying to migrate RHEL4 VM on RHEL5 Dom0s
the VM's state becomes "Shut Off" on the source Dom0  and "No State" on the target Dom0

Version-Release number of selected component (if applicable):
xen-3.0.3-41.el5

How reproducible:
Always


Steps to Reproduce:
1. xm create rhel4-vm
2. xm migrate rhel4-vm corpha2-int
3.

Actual Results:
the VM's state becomes "Shut Off" on the source Dom0  and "No State" on the target Dom0
xm list shows on 
dom0(1)
rhel4-vm                                  45      299     1 ---s--      8.0
dom(2)
rhel4-vm                                  16      300     0 -bp---      0.0



Expected Results:


Additional info:
$ xm list --long
(domain
    (domid 0)
    (uuid 00000000-0000-0000-0000-000000000000)
    (vcpus 8)
    (cpu_weight 1.0)
    (memory 7158)
    (shadow_memory 0)
    (maxmem 7158)
    (features )
    (name Domain-0)
    (on_poweroff destroy)
    (on_reboot restart)
    (on_crash restart)
    (state r-----)
    (shutdown_reason poweroff)
    (cpu_time 20680.9275378)
    (online_vcpus 8)
)
(domain
    (domid 45)
    (uuid 6f7e670b-2075-0174-7766-bd2f44e7de7a)
    (vcpus 1)
    (cpu_weight 1.0)
    (memory 300)
    (shadow_memory 0)
    (maxmem 300)
    (bootloader /usr/bin/pygrub)
    (features )
    (name rhel4-vm)
    (on_poweroff destroy)
    (on_reboot restart)
    (on_crash restart)
    (image
        (linux
            (ramdisk /var/lib/xen/boot_ramdisk.L_XoT-)
            (kernel /var/lib/xen/boot_kernel.MQx2kg)
            (args 'ro root=/dev/VolGroup00/rootfs rhgb quiet')
        )
    )
    (device
        (vif
            (backend 0)
            (script vif-bridge)
            (bridge xenbr0)
            (mac 00:16:3e:23:a1:04)
        )
    )
    (device (vbd (backend 0) (dev xvda:disk) (uname phy:/dev/sdg) (mode w)))
    (device (vkbd (backend 0)))
    (device
        (vfb
            (backend 0)
            (type vnc)
            (vncunused 1)
            (display localhost:10.0)
            (xauthority /root/.Xauthority)
        )
    )
    (state ---s--)
    (shutdown_reason suspend)
    (cpu_time 7.993818084)
    (online_vcpus 1)
    (up_time 832.86420393)
    (start_time 1204816325.17)
    (store_mfn 2275795)
    (console_mfn 2275794)
)

$cat xend-debug.log 
Running with no password
06/03/2008 10:12:06 Autoprobing TCP port 
06/03/2008 10:12:06 Autoprobing selected port 5900
ERROR Internal error: Couldn't map p2m_frame_list_list


on the target dom0(2)

$ xm  list --long
(domain
    (domid 0)
    (uuid 00000000-0000-0000-0000-000000000000)
    (vcpus 8)
    (cpu_weight 1.0)
    (memory 7158)
    (shadow_memory 0)
    (maxmem 7158)
    (features )
    (name Domain-0)
    (on_poweroff destroy)
    (on_reboot restart)
    (on_crash restart)
    (state r-----)
    (shutdown_reason poweroff)
    (cpu_time 3836.34362988)
    (online_vcpus 8)
)
(domain
    (domid 16)
    (uuid 6f7e670b-2075-0174-7766-bd2f44e7de7a)
    (vcpus 1)
    (cpu_weight 1.0)
    (memory 300)
    (shadow_memory 0)
    (maxmem 300)
    (bootloader /usr/bin/pygrub)
    (features )
    (name rhel4-vm)
    (on_poweroff destroy)
    (on_reboot restart)
    (on_crash restart)
    (image
        (linux
            (ramdisk /var/lib/xen/boot_ramdisk.L_XoT-)
            (kernel /var/lib/xen/boot_kernel.MQx2kg)
            (args 'ro root=/dev/VolGroup00/rootfs rhgb quiet')
        )
    )
    (device
        (vif
            (backend 0)
            (script vif-bridge)
            (bridge xenbr0)
            (mac 00:16:3e:23:a1:04)
        )
    )
    (device (vbd (backend 0) (dev xvda:disk) (uname phy:/dev/sdg) (mode w)))
    (device (vkbd (backend 0)))
    (device
        (vfb
            (backend 0)
            (type vnc)
            (vncunused 1)
            (display localhost:10.0)
            (xauthority /root/.Xauthority)
        )
    )
    (state -bp---)
    (shutdown_reason poweroff)
    (cpu_time 0.0)
    (online_vcpus 0)
    (up_time 890.467780828)
    (start_time 1204816325.17)
)



 $ cat xend.log
[2008-03-06 10:11:51 xend 9933] INFO (SrvDaemon:283) Xend Daemon started
[2008-03-06 10:11:51 xend 9933] INFO (SrvDaemon:287) Xend changeset: unavailable.
[2008-03-06 10:11:51 xend.XendDomainInfo 9933] DEBUG (XendDomainInfo:222) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 3833116535168L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 0, 'dying': 0, 'mem_kb': 7328876L, 'domid': 0, 'max_vcpu_id': 7, 'crashed': 0, 'running': 1, 'maxmem_kb': 17179869180L, 'shutdown': 0, 'online_vcpus': 8, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'blocked': 0})
[2008-03-06 10:11:51 xend.XendDomainInfo 9933] INFO (XendDomainInfo:234) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000.
[2008-03-06 10:11:51 xend 9933] DEBUG (XendDomain:153) number of vcpus to use is 0
[2008-03-06 10:11:51 xend 9933] INFO (SrvServer:116) unix path=/var/lib/xend/xend-socket
[2008-03-06 10:11:51 xend.XendDomainInfo 9933] DEBUG (XendDomainInfo:988) XendDomainInfo.handleShutdownWatch
[2008-03-06 10:12:36 xend 9933] DEBUG (tcp:66) Match corpha1-int.rdu.redhat.com ^192\.168\.1\.1$
[2008-03-06 10:12:36 xend.XendDomainInfo 9933] DEBUG (XendDomainInfo:275) XendDomainInfo.restore(['domain', ['domid', '45'], ['uuid', '6f7e670b-2075-0174-7766-bd2f44e7de7a'], ['vcpus', '1'], ['vcpu_avail', '1'], ['cpu_weight', '1.0'], ['memory', '300'], ['shadow_memory', '0'], ['maxmem', '300'], ['bootloader', '/usr/bin/pygrub'], ['features'], ['name', 'rhel4-vm'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['image', ['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.L_XoT-'], ['kernel', '/var/lib/xen/boot_kernel.MQx2kg'], ['args', 'ro root=/dev/VolGroup00/rootfs rhgb quiet']]], ['device', ['vif', ['backend', '0'], ['script', 'vif-bridge'], ['bridge', 'xenbr0'], ['mac', '00:16:3e:23:a1:04']]], ['device', ['vbd', ['backend', '0'], ['dev', 'xvda:disk'], ['uname', 'phy:/dev/sdg'], ['mode', 'w']]], ['device', ['vkbd', ['backend', '0']]], ['device', ['vfb', ['backend', '0'], ['type', 'vnc'], ['vncunused', '1'], ['display', 'localhost:10.0'], ['xauthority', '/root/.Xauthority']]], ['state', '-b----'], ['shutdown_reason', 'poweroff'], ['cpu_time', '5.627690237'], ['online_vcpus', '1'], ['up_time', '33.8372058868'], ['start_time', '1204816325.17'], ['store_mfn', '2275795'], ['console_mfn', '2275794']])
[2008-03-06 10:12:36 xend.XendDomainInfo 9933] DEBUG (XendDomainInfo:306) parseConfig: config is ['domain', ['domid', '45'], ['uuid', '6f7e670b-2075-0174-7766-bd2f44e7de7a'], ['vcpus', '1'], ['vcpu_avail', '1'], ['cpu_weight', '1.0'], ['memory', '300'], ['shadow_memory', '0'], ['maxmem', '300'], ['bootloader', '/usr/bin/pygrub'], ['features'], ['name', 'rhel4-vm'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['image', ['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.L_XoT-'], ['kernel', '/var/lib/xen/boot_kernel.MQx2kg'], ['args', 'ro root=/dev/VolGroup00/rootfs rhgb quiet']]], ['device', ['vif', ['backend', '0'], ['script', 'vif-bridge'], ['bridge', 'xenbr0'], ['mac', '00:16:3e:23:a1:04']]], ['device', ['vbd', ['backend', '0'], ['dev', 'xvda:disk'], ['uname', 'phy:/dev/sdg'], ['mode', 'w']]], ['device', ['vkbd', ['backend', '0']]], ['device', ['vfb', ['backend', '0'], ['type', 'vnc'], ['vncunused', '1'], ['display', 'localhost:10.0'], ['xauthority', '/root/.Xauthority']]], ['state', '-b----'], ['shutdown_reason', 'poweroff'], ['cpu_time', '5.627690237'], ['online_vcpus', '1'], ['up_time', '33.8372058868'], ['start_time', '1204816325.17'], ['store_mfn', '2275795'], ['console_mfn', '2275794']]
[2008-03-06 10:12:36 xend.XendDomainInfo 9933] DEBUG (XendDomainInfo:411) parseConfig: result is {'shadow_memory': 0, 'start_time': 1204816325.1700001, 'uuid': '6f7e670b-2075-0174-7766-bd2f44e7de7a', 'on_crash': 'restart', 'on_reboot': 'restart', 'localtime': None, 'image': ['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.L_XoT-'], ['kernel', '/var/lib/xen/boot_kernel.MQx2kg'], ['args', 'ro root=/dev/VolGroup00/rootfs rhgb quiet']], 'on_poweroff': 'destroy', 'bootloader_args': None, 'cpus': None, 'name': 'rhel4-vm', 'backend': [], 'vcpus': 1, 'cpu_weight': 1.0, 'features': None, 'vcpu_avail': 1, 'memory': 300, 'device': [('vif', ['vif', ['backend', '0'], ['script', 'vif-bridge'], ['bridge', 'xenbr0'], ['mac', '00:16:3e:23:a1:04']]), ('vbd', ['vbd', ['backend', '0'], ['dev', 'xvda:disk'], ['uname', 'phy:/dev/sdg'], ['mode', 'w']]), ('vkbd', ['vkbd', ['backend', '0']]), ('vfb', ['vfb', ['backend', '0'], ['type', 'vnc'], ['vncunused', '1'], ['display', 'localhost:10.0'], ['xauthority', '/root/.Xauthority']])], 'bootloader': '/usr/bin/pygrub', 'cpu': None, 'maxmem': 300}
[2008-03-06 10:12:36 xend.XendDomainInfo 9933] DEBUG (XendDomainInfo:1349) XendDomainInfo.construct: None
[2008-03-06 10:12:36 xend 9933] DEBUG (balloon:127) Balloon: 922220 KiB free; need 2048; done.
[2008-03-06 10:12:36 xend.XendDomainInfo 9933] DEBUG (XendDomainInfo:756) Storing VM details: {'shadow_memory': '0', 'uuid': '6f7e670b-2075-0174-7766-bd2f44e7de7a', 'on_reboot': 'restart', 'start_time': '1204816325.17', 'on_poweroff': 'destroy', 'name': 'rhel4-vm', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '300', 'on_crash': 'restart', 'image': "(linux (ramdisk /var/lib/xen/boot_ramdisk.L_XoT-) (kernel /var/lib/xen/boot_kernel.MQx2kg) (args 'ro root=/dev/VolGroup00/rootfs rhgb quiet'))", 'maxmem': '300'}
[2008-03-06 10:12:36 xend 9933] DEBUG (DevController:110) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/vkbd/16/0'} to /local/domain/16/device/vkbd/0.
[2008-03-06 10:12:36 xend 9933] DEBUG (DevController:112) DevController: writing {'frontend-id': '16', 'domain': 'rhel4-vm', 'frontend': '/local/domain/16/device/vkbd/0', 'state': '1', 'online': '1'} to /local/domain/0/backend/vkbd/16/0.
[2008-03-06 10:12:36 xend 9933] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'mac': '00:16:3e:23:a1:04', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/16/0'} to /local/domain/16/device/vif/0.
[2008-03-06 10:12:36 xend 9933] DEBUG (DevController:112) DevController: writing {'bridge': 'xenbr0', 'domain': 'rhel4-vm', 'handle': '0', 'script': '/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': '/local/domain/16/device/vif/0', 'mac': '00:16:3e:23:a1:04', 'online': '1', 'frontend-id': '16'} to /local/domain/0/backend/vif/16/0.
[2008-03-06 10:12:36 xend 9933] DEBUG (blkif:24) exception looking up device number for xvda: [Errno 2] No such file or directory: '/dev/xvda'
[2008-03-06 10:12:36 xend 9933] DEBUG (DevController:110) DevController: writing {'backend-id': '0', 'virtual-device': '51712', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/16/51712'} to /local/domain/16/device/vbd/51712.
[2008-03-06 10:12:36 xend 9933] DEBUG (DevController:112) DevController: writing {'domain': 'rhel4-vm', 'frontend': '/local/domain/16/device/vbd/51712', 'dev': 'xvda', 'state': '1', 'params': '/dev/sdg', 'mode': 'w', 'online': '1', 'frontend-id': '16', 'type': 'phy'} to /local/domain/0/backend/vbd/16/51712.
[2008-03-06 10:12:36 xend 9933] DEBUG (DevController:110) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/vfb/16/0'} to /local/domain/16/device/vfb/0.
[2008-03-06 10:12:36 xend 9933] DEBUG (DevController:112) DevController: writing {'vncunused': '1', 'domain': 'rhel4-vm', 'frontend': '/local/domain/16/device/vfb/0', 'xauthority': '/root/.Xauthority', 'state': '1', 'online': '1', 'frontend-id': '16', 'type': 'vnc', 'display': 'localhost:10.0'} to /local/domain/0/backend/vfb/16/0.
[2008-03-06 10:12:36 xend 9933] DEBUG (vfbif:64) No VNC passwd configured for vfb access
[2008-03-06 10:12:36 xend.XendDomainInfo 9933] DEBUG (XendDomainInfo:791) Storing domain details: {'console/port': '2', 'name': 'rhel4-vm', 'console/limit': '1048576', 'vm': '/vm/6f7e670b-2075-0174-7766-bd2f44e7de7a', 'domid': '16', 'cpu/0/availability': 'online', 'memory/target': '307200', 'store/port': '1'}
[2008-03-06 10:12:36 xend 9933] DEBUG (XendCheckpoint:198) restore:shadow=0x0, _static_max=0x12c, _static_min=0x12c, 
[2008-03-06 10:12:36 xend 9933] DEBUG (balloon:127) Balloon: 922212 KiB free; need 307200; done.
[2008-03-06 10:12:36 xend 9933] DEBUG (XendCheckpoint:215) [xc_restore]: /usr/lib64/xen/bin/xc_restore 15 16 1 2 0 0 0
[2008-03-06 10:12:36 xend 9933] INFO (XendCheckpoint:351) xc_domain_restore start: p2m_size = 13400

Comment 1 Don Dutile (Red Hat) 2008-03-06 16:09:05 UTC
Please provide dom0 kernel rev (uname -a) as well as domU rhel4-guest kernel rev.

Comment 2 Ege Turgay 2008-03-06 16:28:49 UTC
dom0(1)
Linux corpha1.rdu.redhat.com 2.6.18-53.1.13.el5xen #1 SMP Mon Feb 11 13:41:50
EST 2008 x86_64 x86_64 x86_64 GNU/Linux
dom0(2)
Linux corpha2.rdu.redhat.com 2.6.18-53.1.13.el5xen #1 SMP Mon Feb 11 13:41:50
EST 2008 x86_64 x86_64 x86_64 GNU/Linux

RHEL4 VM
Linux syslog-test.rdu.redhat.com 2.6.9-67.ELxenU #1 SMP Wed Nov 7 14:19:15 EST
2007 i686 i686 i386 GNU/Linux

[~] $ cat /etc/xen/xend-config.sxp 
# -*- sh -*-

#
# Xend configuration file.
#

# This example configuration is appropriate for an installation that 
# utilizes a bridged network configuration. Access to xend via http
# is disabled.  

# Commented out entries show the default for that entry, unless otherwise
# specified.

(logfile /var/log/xen/xend.log)
(loglevel DEBUG)

#(xend-http-server no)
(xend-unix-server yes)
#(xend-tcp-xmlrpc-server no)
#(xend-unix-xmlrpc-server yes)
(xend-relocation-server yes)
# The relocation server should be kept desactivated unless using a trusted
# network, the domain virtual memory will be exchanged in raw form without
# encryption of the communication. See also xend-relocation-hosts-allow option

(xend-unix-path /var/lib/xend/xend-socket)

# Port xend should use for the HTTP interface, if xend-http-server is set.
#(xend-port            8000)

# Port xend should use for the relocation interface, if xend-relocation-server
# is set.
(xend-relocation-port 8002)

# Address xend should listen on for HTTP connections, if xend-http-server is
# set.
# Specifying 'localhost' prevents remote connections.
# Specifying the empty string '' (the default) allows all connections.
#(xend-address '')
#(xend-address localhost)

# Address xend should listen on for relocation-socket connections, if
# xend-relocation-server is set.
# Meaning and default as for xend-address above.
#(xend-relocation-address '')
(xend-relocation-address 192.168.1.2)

# The hosts allowed to talk to the relocation port.  If this is empty (the
# default), then all connections are allowed (assuming that the connection
# arrives on a port and interface on which we are listening; see
# xend-relocation-port and xend-relocation-address above).  Otherwise, this
# should be a space-separated sequence of regular expressions.  Any host with
# a fully-qualified domain name or an IP address that matches one of these
# regular expressions will be accepted.
#
# For example:
#  (xend-relocation-hosts-allow '^localhost$ ^.*\.example\.org$')
#
#(xend-relocation-hosts-allow '')
#(xend-relocation-hosts-allow '^localhost$ ^localhost\\.localdomain$')
(xend-relocation-hosts-allow ^192\.168\.1\.1$)

# The limit (in kilobytes) on the size of the console buffer
#(console-limit 1024)

##
# To bridge network traffic, like this:
#
# dom0: fake eth0 -> vif0.0 -+
#                            |
#                          bridge -> real eth0 -> the network
#                            |
# domU: fake eth0 -> vifN.0 -+
#
# use
#
# (network-script network-bridge)
#
# Your default ethernet device is used as the outgoing interface, by default. 
# To use a different one (e.g. eth1) use
#
# (network-script 'network-bridge netdev=eth1')
#
# The bridge is named xenbr0, by default.  To rename the bridge, use
#
# (network-script 'network-bridge bridge=<name>')
#
# It is possible to use the network-bridge script in more complicated
# scenarios, such as having two outgoing interfaces, with two bridges, and
# two fake interfaces per guest domain.  To do things like this, write
# yourself a wrapper script, and call network-bridge from it, as appropriate.
#
(network-script network-bridge)

# The script used to control virtual interfaces.  This can be overridden on a
# per-vif basis when creating a domain or a configuring a new vif.  The
# vif-bridge script is designed for use with the network-bridge script, or
# similar configurations.
#
# If you have overridden the bridge name using
# (network-script 'network-bridge bridge=<name>') then you may wish to do the
# same here.  The bridge name can also be set when creating a domain or
# configuring a new vif, but a value specified here would act as a default.
#
# If you are using only one bridge, the vif-bridge script will discover that,
# so there is no need to specify it explicitly.
#
(vif-script vif-bridge)


## Use the following if network traffic is routed, as an alternative to the
# settings for bridged networking given above.
#(network-script network-route)
#(vif-script     vif-route)


## Use the following if network traffic is routed with NAT, as an alternative
# to the settings for bridged networking given above.
#(network-script network-nat)
#(vif-script     vif-nat)


# Dom0 will balloon out when needed to free memory for domU.
# dom0-min-mem is the lowest memory level (in MB) dom0 will get down to.
# If dom0-min-mem=0, dom0 will never balloon out.
(dom0-min-mem 256)

# In SMP system, dom0 will use dom0-cpus # of CPUS
# If dom0-cpus = 0, dom0 will take all cpus available
(dom0-cpus 0)

# Whether to enable core-dumps when domains crash.
#(enable-dump no)

# The tool used for initiating virtual TPM migration
#(external-migration-tool '')

# The interface for VNC servers to listen on. Defaults
# to 127.0.0.1  To restore old 'listen everywhere' behaviour
# set this to 0.0.0.0
#(vnc-listen '127.0.0.1')
  
# The default password for VNC console on HVM domain.
# Empty string is no authentication.
(vncpasswd '')




Comment 3 Don Dutile (Red Hat) 2008-03-06 17:13:48 UTC

This could be a dupe of 431081.

Are you getting domU crash message as shown in 431081 ?

unfortunately, I'm waiting (begging) for simple ACK flags to be set on
this bz, so it can go into 4.7.





Comment 4 Don Dutile (Red Hat) 2008-03-06 22:21:23 UTC
Skip comment #3 .... I didn't carefully read the uname -r output,
and see that both dom's are x86_64.
bz 431081 is a 32-on-32 (only) bug.



Comment 5 Don Dutile (Red Hat) 2008-03-07 15:44:30 UTC
My bad for not reading the details correctly, again!

32-on-64 is tech-preview on 5.1.

Migration (save, restore) of 32-on-64 has known problems & not supported.
the guest is rhel4 i686; the dom0's are x86_64.

This won't be fixed on 5.1. 
It may be fixed in a future version of rhel5.


Note You need to log in before you can comment on or make changes to this bug.