Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
According Comment 2 and Bug 1172522 this isn't a libvirt fault, because the error message comes from spice server. The error "early end of file from monitor" usually happens when qemu-kvm exit while starting a domain.
Re-assigning to spice.
Comment 6Marc-Andre Lureau
2015-05-13 16:28:14 UTC
I don't think the "statistics" code is being used by anyone, so we should probably disable it in RHEL.
One option could be easily disabling statistics.
The problem came from stale files and permissions.
The shm file name is based on the pid number (/spice.XXX where XXX is the pid number so at the end you get a file /dev/shm/spice.XXX). If a qemu process die or does not manage to remove (calling shm_unlink) that file on next execution of a qemu with the same pid if possible to have this issue. The file is created with 444 as permissions so trying to opening again it fails. So there could be different solutions
- disable statistics
- create the file with 644 permission so on next shm_open call (a new qemu process) shm_open does not fail
- call shm_unlink before shm_open. The file should not be there.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory, and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://rhn.redhat.com/errata/RHBA-2015-2429.html
Description of problem: Failed to start domain due to "Spice-ERROR **: reds.c:3975:do_spice_init: statistics shm_open failed, Permission denied" Version-Release number of selected component (if applicable): libvirt-1.2.8-10.el7.x86_64 spice-server-0.12.4-8.el7.x86_64 qemu-kvm-rhev-2.1.2-16.el7.x86_64 How reproducible: low(<10%) Steps to Reproduce: [root@ibm-x3850x5-02 images]# cat /etc/libvirt/libvirtd.conf | grep "^[^#]" listen_tcp = 1 auth_tcp = "none" auth_tls = "none" max_anonymous_clients = 20000 max_workers = 20000 max_requests = 20000 max_client_requests = 5000 log_level = 1 log_outputs="1:file:/var/log/libvirt/libvirtd.log" log_filters="1:qemu 1:libvirt 1:conf 1:json" [root@ibm-x3850x5-02 images]# cat /etc/libvirt/qemu.conf | grep "^[^#]" max_processes = 1000000 max_files = 1000000 [root@ibm-x3850x5-02 images]# cat /etc/systemd/system/multi-user.target.wants/libvirtd.service| grep Limit LimitNOFILE=2048000 [root@ibm-x3850x5-02 images]# [root@dhcp-66-83-39 testing]# for i in `seq 1 1024`; do qemu-img create -f qcow2 guest$i.img 1M; done [root@ibm-x3850x5-02 scalability]# cat define.sh #! /bin/sh - for i in `seq 1 1024` do sed -i "s,#num#,guest$i,g" r7.xml virsh define r7.xml sed -i "s,guest$i,#num#,g" r7.xml #sleep 1 done [root@ibm-x3850x5-02 scalability]# for i in {1..1024};do virsh start guest$i;done ... Domain guest569 started error: Failed to start domain guest570 error: internal error: process exited while connecting to monitor: ((null):13809): Spice-ERROR **: reds.c:3975:do_spice_init: statistics shm_open failed, Permission denied Domain guest571 started Domain guest572 started Domain guest573 started Domain guest574 started error: Failed to start domain guest575 error: internal error: early end of file from monitor: possible problem: ((null):13872): Spice-ERROR **: reds.c:3975:do_spice_init: statistics shm_open failed, Permission denied Domain guest576 started Domain guest577 started Domain guest578 started Domain guest579 started Domain guest580 started Domain guest581 started Domain guest582 started ... Actual results: As shown above steps, start 1024 domains, hit this issue 28 times. Try to start it again, the failed domain can be started successfully. [root@ibm-x3850x5-02 scalability]# virsh start guest575 Domain guest575 started +++++++++++ [root@ibm-x3850x5-02 scalability]# cat /var/log/libvirt/qemu/guest575.log ... 2014-12-26 06:24:32.568+0000: 13872: debug : qemuProcessHook:3083 : Hook complete ret=0 2014-12-26 06:24:32.568+0000: 13872: debug : virExec:691 : Done hook 0 2014-12-26 06:24:32.568+0000: 13872: debug : virExec:698 : Setting child security label to system_u:system_r:svirt_t:s0:c140,c239 2014-12-26 06:24:32.568+0000: 13872: debug : virExec:728 : Setting child uid:gid to 107:107 with caps 0 2014-12-26 06:24:32.569+0000: 13872: debug : virCommandHandshakeChild:431 : Notifying parent for handshake start on 583 2014-12-26 06:24:32.570+0000: 13872: debug : virCommandHandshakeChild:439 : Waiting on parent for handshake complete on 584 2014-12-26 06:24:32.676+0000: 13872: debug : virFileClose:99 : Closed fd 583 2014-12-26 06:24:32.676+0000: 13872: debug : virFileClose:99 : Closed fd 584 2014-12-26 06:24:32.676+0000: 13872: debug : virCommandHandshakeChild:459 : Handshake with parent is done char device redirected to /dev/pts/548 (label charserial0) ((null):13872): Spice-ERROR **: reds.c:3975:do_spice_init: statistics shm_open failed, Permission denied 2014-12-26 06:24:33.559+0000: shutting down ... +++++++++++ Expected results: Start a lot of domain continually, each domain should be started normally. Additional info: [root@ibm-x3850x5-02 scalability]# cat r7.xml <domain type='kvm'> <name>#num#</name> <memory unit='M'>128</memory> <vcpu placement='static'>4</vcpu> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.1.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/home/images/#num#.img'/> <target dev='hda' bus='ide'/> </disk> <controller type='pci' index='0' model='pci-root'/> <controller type='virtio-serial' index='0'> </controller> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='spicevmc'> <target type='virtio' name='com.redhat.spice.0'/> </channel> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <graphics type='spice' autoport='yes'/> <video> <model type='qxl' ram='65536' vram='65536' heads='1'/> </video> <memballoon model='none'> </memballoon> </devices> </domain>