RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1177326 - Failed to start domain due to "Spice-ERROR **: reds.c:3975:do_spice_init: statistics shm_open failed, Permission denied"
Summary: Failed to start domain due to "Spice-ERROR **: reds.c:3975:do_spice_init: sta...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: spice
Version: 7.2
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: 7.2
Assignee: Default Assignee for SPICE Bugs
QA Contact: SPICE QE bug list
URL:
Whiteboard:
: 1243313 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-12-26 06:52 UTC by Hu Jianwei
Modified: 2015-11-19 14:34 UTC (History)
11 users (show)

Fixed In Version: spice-0.12.4-13.el7
Doc Type: Bug Fix
Doc Text:
No doc needed
Clone Of:
Environment:
Last Closed: 2015-11-19 14:34:31 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
PoC of possible solution (third option in previous comment) (388 bytes, text/x-csrc)
2015-07-15 09:46 UTC, Frediano Ziglio
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2429 0 normal SHIPPED_LIVE spice bug fix update 2015-11-19 11:34:35 UTC

Description Hu Jianwei 2014-12-26 06:52:59 UTC
Description of problem:
Failed to start domain due to "Spice-ERROR **: reds.c:3975:do_spice_init: statistics shm_open failed, Permission denied"

Version-Release number of selected component (if applicable):
libvirt-1.2.8-10.el7.x86_64
spice-server-0.12.4-8.el7.x86_64
qemu-kvm-rhev-2.1.2-16.el7.x86_64

How reproducible:
low(<10%)

Steps to Reproduce:
[root@ibm-x3850x5-02 images]# cat /etc/libvirt/libvirtd.conf | grep "^[^#]"
listen_tcp = 1
auth_tcp = "none"
auth_tls = "none"
max_anonymous_clients = 20000
max_workers = 20000
max_requests = 20000
max_client_requests = 5000
log_level = 1
log_outputs="1:file:/var/log/libvirt/libvirtd.log" 
log_filters="1:qemu 1:libvirt 1:conf 1:json"
[root@ibm-x3850x5-02 images]# cat /etc/libvirt/qemu.conf | grep "^[^#]"
max_processes = 1000000
max_files = 1000000
[root@ibm-x3850x5-02 images]# cat /etc/systemd/system/multi-user.target.wants/libvirtd.service| grep Limit
LimitNOFILE=2048000
[root@ibm-x3850x5-02 images]# 

[root@dhcp-66-83-39 testing]# for i in `seq 1 1024`; do qemu-img create -f qcow2 guest$i.img 1M; done

[root@ibm-x3850x5-02 scalability]# cat define.sh 
#! /bin/sh -

for i in `seq 1 1024`
do
sed -i "s,#num#,guest$i,g" r7.xml
virsh define r7.xml
sed -i "s,guest$i,#num#,g" r7.xml
#sleep 1
done

[root@ibm-x3850x5-02 scalability]# for i in {1..1024};do virsh start guest$i;done
...
Domain guest569 started

error: Failed to start domain guest570
error: internal error: process exited while connecting to monitor: ((null):13809): Spice-ERROR **: reds.c:3975:do_spice_init: statistics shm_open failed, Permission denied


Domain guest571 started

Domain guest572 started

Domain guest573 started

Domain guest574 started

error: Failed to start domain guest575
error: internal error: early end of file from monitor: possible problem:
((null):13872): Spice-ERROR **: reds.c:3975:do_spice_init: statistics shm_open failed, Permission denied


Domain guest576 started

Domain guest577 started

Domain guest578 started

Domain guest579 started

Domain guest580 started

Domain guest581 started

Domain guest582 started
...

Actual results:
As shown above steps, start 1024 domains, hit this issue 28 times. Try to start it again, the failed domain can be started successfully. 
[root@ibm-x3850x5-02 scalability]# virsh start guest575
Domain guest575 started

+++++++++++
[root@ibm-x3850x5-02 scalability]# cat /var/log/libvirt/qemu/guest575.log
...
2014-12-26 06:24:32.568+0000: 13872: debug : qemuProcessHook:3083 : Hook complete ret=0
2014-12-26 06:24:32.568+0000: 13872: debug : virExec:691 : Done hook 0
2014-12-26 06:24:32.568+0000: 13872: debug : virExec:698 : Setting child security label to system_u:system_r:svirt_t:s0:c140,c239
2014-12-26 06:24:32.568+0000: 13872: debug : virExec:728 : Setting child uid:gid to 107:107 with caps 0
2014-12-26 06:24:32.569+0000: 13872: debug : virCommandHandshakeChild:431 : Notifying parent for handshake start on 583
2014-12-26 06:24:32.570+0000: 13872: debug : virCommandHandshakeChild:439 : Waiting on parent for handshake complete on 584
2014-12-26 06:24:32.676+0000: 13872: debug : virFileClose:99 : Closed fd 583
2014-12-26 06:24:32.676+0000: 13872: debug : virFileClose:99 : Closed fd 584
2014-12-26 06:24:32.676+0000: 13872: debug : virCommandHandshakeChild:459 : Handshake with parent is done
char device redirected to /dev/pts/548 (label charserial0)
((null):13872): Spice-ERROR **: reds.c:3975:do_spice_init: statistics shm_open failed, Permission denied
2014-12-26 06:24:33.559+0000: shutting down
...
+++++++++++

Expected results:
Start a lot of domain continually, each domain should be started normally.

Additional info:
[root@ibm-x3850x5-02 scalability]# cat r7.xml 
<domain type='kvm'>
  <name>#num#</name>
  <memory unit='M'>128</memory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.1.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
   <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/home/images/#num#.img'/>
      <target dev='hda' bus='ide'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
    </controller>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <video>
      <model type='qxl' ram='65536' vram='65536' heads='1'/>
    </video>
    <memballoon model='none'>
    </memballoon>
  </devices>
</domain>

Comment 2 Hu Jianwei 2014-12-26 06:58:32 UTC
Do migration also can hit this issue, please see the comment.
https://bugzilla.redhat.com/show_bug.cgi?id=1172522#c4

If this is not libvirt component, please change to correct place, thanks.

Comment 5 Pavel Hrdina 2015-05-13 13:28:39 UTC
According Comment 2 and Bug 1172522 this isn't a libvirt fault, because the error message comes from spice server.  The error "early end of file from monitor" usually happens when qemu-kvm exit while starting a domain.

Re-assigning to spice.

Comment 6 Marc-Andre Lureau 2015-05-13 16:28:14 UTC
I don't think the "statistics" code is being used by anyone, so we should probably disable it in RHEL.

Comment 8 Frediano Ziglio 2015-07-14 17:33:40 UTC
One option could be easily disabling statistics.
The problem came from stale files and permissions.
The shm file name is based on the pid number (/spice.XXX where XXX is the pid number so at the end you get a file /dev/shm/spice.XXX). If a qemu process die or does not manage to remove (calling shm_unlink) that file on next execution of a qemu with the same pid if possible to have this issue. The file is created with 444 as permissions so trying to opening again it fails. So there could be different solutions
- disable statistics
- create the file with 644 permission so on next shm_open call (a new qemu process) shm_open does not fail
- call shm_unlink before shm_open. The file should not be there.

Comment 9 Frediano Ziglio 2015-07-15 09:46:43 UTC
Created attachment 1052277 [details]
PoC of possible solution (third option in previous comment)

Comment 10 Frediano Ziglio 2015-07-15 10:06:21 UTC
Post a patch upstream, see http://lists.freedesktop.org/archives/spice-devel/2015-July/020863.html

Comment 11 Victor Toso 2015-07-16 15:39:55 UTC
*** Bug 1243313 has been marked as a duplicate of this bug. ***

Comment 15 errata-xmlrpc 2015-11-19 14:34:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2429.html


Note You need to log in before you can comment on or make changes to this bug.