Bug 667283

Summary: migrate fails - ERROR Internal error: Couldn't allocate p2m_frame_list array
Product: [Fedora] Fedora Reporter: Jerry Amundson <jamundso>
Component: xenAssignee: Xen Maintainance List <xen-maint>
Status: CLOSED WONTFIX QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 14CC: berrange, crobinso, jforbes, kraxel, m.a.young, virt-maint, xen-maint
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2012-06-06 13:51:25 EDT Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Description Jerry Amundson 2011-01-04 23:06:12 EST
Description of problem:
migrate fails - ERROR Internal error: Couldn't allocate p2m_frame_list array

Version-Release number of selected component (if applicable):
[root@elm ~]# rpm -q xen ; xm dmesg | grep kernel:
xen-4.0.1-6.fc14.i686
(XEN)  Xen  kernel: 32-bit, PAE, lsb
(XEN)  Dom0 kernel: 32-bit, PAE, lsb, paddr 0x400000 -> 0xce1000

How reproducible:
always

Steps to Reproduce:
1.create vm
2.attempt migrate
3.
  
Actual results:
fail

Expected results:
success

Additional info:
Guest domU is 2.6.18-194.26.1.el5xen (CentOS 5.5)

Cluster node is 
[root@oak ~]# rpm -q xen ; xm dmesg | grep kernel:
xen-4.0.1-6.fc14.x86_64
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1b38000
Comment 1 Jerry Amundson 2011-01-05 19:30:07 EST
I'm stumped. The domU runs fine on either dom0, but will not migrate between them.

With the domU "alpha" running on dom0 "elm" :

[root@elm ~]# xm migrate -l alpha oak
Error: /usr/lib64/xen/bin/xc_restore 18 1 1 2 0 0 0 0 failed (from oak)
Usage: xm migrate <Domain> <Host>

Migrate a domain to another machine.

Options:

-h, --help           Print this help.
-l, --live           Use live migration.
-p=portnum, --port=portnum
                     Use specified port for migration.
-n=nodenum, --node=nodenum
                     Use specified NUMA node on target.
-s, --ssl            Use ssl connection for migration.
-c, --change_home_server
                     Change home server for managed domains.

From xend.log on dom0 elm :

[2011-01-04 20:45:20 2436] DEBUG (XendCheckpoint:124) [xc_save]: /usr/lib/xen/bin/xc_save 44 1 0 0 1
[2011-01-04 20:45:20 2436] INFO (XendCheckpoint:423) xc_save: failed to get the suspend evtchn port
[2011-01-04 20:45:20 2436] INFO (XendCheckpoint:423) 
[2011-01-04 20:45:20 2436] INFO (XendCheckpoint:423) Had 0 unexplained entries in p2m table
[2011-01-04 20:45:23 2436] INFO (XendCheckpoint:423) Saving memory pages: iter 1   0%ERROR Internal error: Error when writing to state file (4b) (errno 104)
[2011-01-04 20:45:23 2436] INFO (XendCheckpoint:423) Save exit rc=1
[2011-01-04 20:45:23 2436] ERROR (XendCheckpoint:178) Save failed on domain alpha (1) - resuming.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/xen/xend/XendCheckpoint.py", line 146, in save
    forkHelper(cmd, fd, saveInputHandler, False)
  File "/usr/lib/python2.7/site-packages/xen/xend/XendCheckpoint.py", line 411, in forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib/xen/bin/xc_save 44 1 0 0 1 failed
[2011-01-04 20:45:23 2436] DEBUG (XendDomainInfo:3117) XendDomainInfo.resumeDomain(1)

From xend.log on dom0 oak :

[2011-01-04 20:45:20 2237] DEBUG (XendDomainInfo:235) XendDomainInfo.restore(['domain', ['domid', '1'], ['cpu_weight', '256'], ['cpu_cap', '0'], ['bootloader', '/usr/bin/pygrub'], ['on_crash', 'destroy'], ['uuid', 'fb1a01b7-ba22-2d62-5cdf-d11da4c371df'], ['bootloader_args'], ['vcpus', '1'], ['description'], ['name', 'alpha'], ['cpus', [[]]], ['on_reboot', 'destroy'], ['on_poweroff', 'destroy'], ['maxmem', '1024'], ['memory', '1024'], ['shadow_memory', '0'], ['vcpu_avail', '1'], ['features'], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['start_time', '1294189027.31'], ['cpu_time', '43.979506087'], ['online_vcpus', '1'], ['image', ['linux', ['kernel'], ['superpages', '0'], ['videoram', '4'], ['pci', []], ['nomigrate', '0'], ['tsc_mode', '0'], ['notes', ['FEATURES', 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel'], ['VIRT_BASE', '3221225472'], ['GUEST_VERSION', '2.6'], ['PADDR_OFFSET', '3221225472'], ['GUEST_OS', 'linux'], ['HYPERCALL_PAGE', '3225423872'], ['LOADER', 'generic'], ['PAE_MODE', 'yes'], ['ENTRY', '3225419776'], ['XEN_VERSION', 'xen-3.0']]]], ['status', '2'], ['state', '-b----'], ['store_mfn', '495362'], ['console_mfn', '495361'], ['device', ['vif', ['bridge', 'br0'], ['mac', '00:16:3E:4B:55:56'], ['script', '/etc/xen/scripts/vif-bridge'], ['uuid', '9e2787dd-bd05-2bd3-d188-7877afeab211'], ['backend', '0']]], ['device', ['console', ['protocol', 'vt100'], ['location', '2'], ['uuid', '9fcf3cf2-f917-fc14-c360-35c32b2918b7']]], ['device', ['vbd', ['protocol', 'x86_32-abi'], ['uuid', '3c7a01aa-a5cc-eb5d-29e4-22046cc1d155'], ['bootable', '1'], ['dev', 'xvda:disk'], ['uname', 'drbd:r0'], ['mode', 'w'], ['backend', '0'], ['VDI']]], ['change_home_server', 'False']])
[2011-01-04 20:45:21 2237] DEBUG (XendDomainInfo:2508) XendDomainInfo.constructDomain
[2011-01-04 20:45:21 2237] DEBUG (balloon:220) Balloon: 1371468 KiB free; need 16384; done.
[2011-01-04 20:45:21 2237] DEBUG (XendDomain:464) Adding Domain: 1
[2011-01-04 20:45:21 2237] DEBUG (XendDomainInfo:3400) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': 'fb1a01b7-ba22-2d62-5cdf-d11da4c371df', 'on_reboot': 'destroy', 'start_time': '1294189027.31', 'on_poweroff': 'destroy', 'bootloader_args': '', 'on_xend_start': 'ignore', 'on_crash': 'destroy', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'bootloader': '/usr/bin/pygrub', 'image': "(linux (kernel ) (superpages 0) (videoram 4) (pci ()) (nomigrate 0) (tsc_mode 0) (notes (FEATURES 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE 3221225472) (GUEST_VERSION 2.6) (PADDR_OFFSET 3221225472) (GUEST_OS linux) (HYPERCALL_PAGE 3225423872) (LOADER generic) (PAE_MODE yes) (ENTRY 3225419776) (XEN_VERSION xen-3.0)))", 'name': 'alpha'}
[2011-01-04 20:45:21 2237] INFO (XendDomainInfo:2367) createDevice: console : {'protocol': 'vt100', 'location': '2', 'uuid': '9fcf3cf2-f917-fc14-c360-35c32b2918b7'}
[2011-01-04 20:45:21 2237] DEBUG (DevController:95) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/1/0'} to /local/domain/1/device/console/0.
[2011-01-04 20:45:21 2237] DEBUG (DevController:97) DevController: writing {'domain': 'alpha', 'frontend': '/local/domain/1/device/console/0', 'uuid': '9fcf3cf2-f917-fc14-c360-35c32b2918b7', 'frontend-id': '1', 'state': '1', 'location': '2', 'online': '1', 'protocol': 'vt100'} to /local/domain/0/backend/console/1/0.
[2011-01-04 20:45:21 2237] INFO (XendDomainInfo:2367) createDevice: vbd : {'protocol': 'x86_32-abi', 'uuid': '3c7a01aa-a5cc-eb5d-29e4-22046cc1d155', 'bootable': 1, 'driver': 'paravirtualised', 'dev': 'xvda:disk', 'uname': 'drbd:r0', 'mode': 'w', 'backend': '0'}
[2011-01-04 20:45:21 2237] DEBUG (DevController:95) DevController: writing {'virtual-device': '51712', 'protocol': 'x86_32-abi', 'device-type': 'disk', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/1/51712'} to /local/domain/1/device/vbd/51712.
[2011-01-04 20:45:21 2237] DEBUG (DevController:97) DevController: writing {'domain': 'alpha', 'frontend': '/local/domain/1/device/vbd/51712', 'uuid': '3c7a01aa-a5cc-eb5d-29e4-22046cc1d155', 'bootable': '1', 'dev': 'xvda', 'state': '1', 'params': 'r0', 'mode': 'w', 'online': '1', 'frontend-id': '1', 'type': 'drbd'} to /local/domain/0/backend/vbd/1/51712.
[2011-01-04 20:45:22 2237] INFO (XendDomainInfo:2367) createDevice: vif : {'bridge': 'br0', 'mac': '00:16:3E:4B:55:56', 'script': '/etc/xen/scripts/vif-bridge', 'uuid': '9e2787dd-bd05-2bd3-d188-7877afeab211', 'backend': '0'}
[2011-01-04 20:45:22 2237] DEBUG (DevController:95) DevController: writing {'backend-id': '0', 'mac': '00:16:3E:4B:55:56', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/1/0'} to /local/domain/1/device/vif/0.
[2011-01-04 20:45:22 2237] DEBUG (DevController:97) DevController: writing {'bridge': 'br0', 'domain': 'alpha', 'handle': '0', 'uuid': '9e2787dd-bd05-2bd3-d188-7877afeab211', 'script': '/etc/xen/scripts/vif-bridge', 'mac': '00:16:3E:4B:55:56', 'frontend-id': '1', 'state': '1', 'online': '1', 'frontend': '/local/domain/1/device/vif/0'} to /local/domain/0/backend/vif/1/0.
[2011-01-04 20:45:22 2237] DEBUG (XendDomainInfo:1804) Storing domain details: {'image/entry': '3225419776', 'console/port': '2', 'image/loader': 'generic', 'vm': '/vm/fb1a01b7-ba22-2d62-5cdf-d11da4c371df', 'control/platform-feature-multiprocessor-suspend': '1', 'image/guest-os': 'linux', 'image/features/writable-descriptor-tables': '1', 'image/virt-base': '3221225472', 'memory/target': '1048576', 'image/guest-version': '2.6', 'image/features/supervisor-mode-kernel': '1', 'image/pae-mode': 'yes', 'description': '', 'console/limit': '1048576', 'image/paddr-offset': '3221225472', 'image/hypercall-page': '3225423872', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '1', 'console/type': 'xenconsoled', 'image/features/auto-translated-physmap': '1', 'name': 'alpha', 'domid': '1', 'image/xen-version': 'xen-3.0', 'store/port': '1'}
[2011-01-04 20:45:22 2237] DEBUG (XendCheckpoint:286) restore:shadow=0x0, _static_max=0x40000000, _static_min=0x0, 
[2011-01-04 20:45:22 2237] DEBUG (XendCheckpoint:305) [xc_restore]: /usr/lib64/xen/bin/xc_restore 18 1 1 2 0 0 0 0
[2011-01-04 20:45:22 2237] INFO (XendCheckpoint:423) xc_domain_restore start: p2m_size = ffffffff00040800
[2011-01-04 20:45:22 2237] INFO (XendCheckpoint:423) ERROR Internal error: Couldn't allocate p2m_frame_list array
[2011-01-04 20:45:22 2237] INFO (XendCheckpoint:423) Restore exit with rc=1
[2011-01-04 20:45:22 2237] DEBUG (XendDomainInfo:3053) XendDomainInfo.destroy: domid=1
[2011-01-04 20:45:22 2237] ERROR (XendDomainInfo:3067) XendDomainInfo.destroy: domain destruction failed.
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomainInfo.py", line 3060, in destroy
    xc.domain_pause(self.domid)
Error: (3, 'No such process')
[2011-01-04 20:45:22 2237] DEBUG (XendDomainInfo:2416) No device model
[2011-01-04 20:45:22 2237] DEBUG (XendDomainInfo:2418) Releasing devices
[2011-01-04 20:45:22 2237] DEBUG (XendDomainInfo:2424) Removing vif/0
[2011-01-04 20:45:22 2237] DEBUG (XendDomainInfo:1286) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2011-01-04 20:45:22 2237] DEBUG (XendDomainInfo:2424) Removing console/0
[2011-01-04 20:45:22 2237] DEBUG (XendDomainInfo:1286) XendDomainInfo.destroyDevice: deviceClass = console, device = console/0
[2011-01-04 20:45:22 2237] DEBUG (XendDomainInfo:2424) Removing vbd/51712
[2011-01-04 20:45:22 2237] DEBUG (XendDomainInfo:1286) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/51712
[2011-01-04 20:45:23 2237] INFO (XendDomain:1107) Domain alpha (fb1a01b7-ba22-2d62-5cdf-d11da4c371df) deleted.
[2011-01-04 20:45:23 2237] ERROR (XendCheckpoint:357) /usr/lib64/xen/bin/xc_restore 18 1 1 2 0 0 0 0 failed
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendCheckpoint.py", line 309, in restore
    forkHelper(cmd, fd, handler.handler, True)
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendCheckpoint.py", line 411, in forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib64/xen/bin/xc_restore 18 1 1 2 0 0 0 0 failed
[2011-01-04 20:45:23 2237] ERROR (XendDomain:1175) Restore failed
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendDomain.py", line 1159, in domain_restore_fd
    dominfo = XendCheckpoint.restore(self, fd, paused=paused, relocating=relocating)
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendCheckpoint.py", line 358, in restore
    raise exn
XendError: /usr/lib64/xen/bin/xc_restore 18 1 1 2 0 0 0 0 failed
Comment 2 Jerry Amundson 2011-01-15 00:59:31 EST
As it stands I'm the only one on the planet for which migration fails under in this environment, so I'm OK with that. :)
Seriously. Big "ping" here. I don't think the basics of what I'm trying here are so unique to attain "ignore it" status...
Comment 3 Cole Robinson 2012-06-06 13:51:25 EDT
Jerry, sorry you never got a response, that's life in bugzilla land sometimes though.

Since F14 has been end-of-life for a few months, I'm  closing this bug. If you can reproduce with F16 or F17, please reopen.