Description of problem: After installation VDSM log is flooded with similar errors libvirtError: Requested operation is not valid: cgroup CPU controller is not mounted Thread-251::DEBUG::2015-02-19 12:44:02,465::libvirtconnection::143::root::(wrapper) Unknown libvirterror: ecode: 55 edom: 10 level: 2 message: Requested operation is not valid: cgroup CPUACCT controller is not mounted Thread-251::ERROR::2015-02-19 12:44:02,465::sampling::475::vm.Vm::(collect) vmId=`fdc09d48-d800-4138-95e0-db18596882e9`::Stats function failed: <AdvancedStatsFunction _sampleCpu at 0x2a00490> Traceback (most recent call last): File "/usr/share/vdsm/virt/sampling.py", line 471, in collect File "/usr/share/vdsm/virt/sampling.py", line 346, in __call__ File "/usr/share/vdsm/virt/vm.py", line 303, in _sampleCpu File "/usr/share/vdsm/virt/vm.py", line 689, in f File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1911, in getCPUStats libvirtError: Requested operation is not valid: cgroup CPUACCT controller is not mounted Thread-251::DEBUG::2015-02-19 12:44:02,481::libvirtconnection::143::root::(wrapper) Unknown libvirterror: ecode: 55 edom: 10 level: 2 message: Requested operation is not valid: cgroup CPU controller is not mounted Thread-251::ERROR::2015-02-19 12:44:02,481::sampling::475::vm.Vm::(collect) vmId=`fdc09d48-d800-4138-95e0-db18596882e9`::Stats function failed: <AdvancedStatsFunction _sampleCpuTune at 0x2a006c0> Traceback (most recent call last): File "/usr/share/vdsm/virt/sampling.py", line 471, in collect File "/usr/share/vdsm/virt/sampling.py", line 346, in __call__ File "/usr/share/vdsm/virt/vm.py", line 349, in _sampleCpuTune File "/usr/share/vdsm/virt/vm.py", line 689, in f File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1991, in schedulerParameters ----------- Migration of VM to affected RHEV-H version fails with error: Thread-1204872::DEBUG::2015-03-03 07:51:27,493::stompReactor::163::yajsonrpc.StompServer::(send) Sending response Thread-1204864::ERROR::2015-03-03 07:51:27,887::migration::260::vm.Vm::(run) vmId=`c45ee223-1ab7-43a7-89fb-1d401ac4c08c`::Failed to migrate Traceback (most recent call last): File "/usr/share/vdsm/virt/migration.py", line 246, in run File "/usr/share/vdsm/virt/migration.py", line 325, in _startUnderlyingMigration File "/usr/share/vdsm/virt/vm.py", line 689, in f File "/usr/lib/python2.6/site-packages/vdsm/libvirtconnection.py", line 111, in wrapper File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1210, in migrateToURI2 libvirtError: unsupported configuration: cgroup cpu is required for scheduler tuning VM Channels Listener::DEBUG::2015-03-03 07:51:28,229::vmchannels::96::vds::(_handle_timeouts) Timeout on fileno 64. Version-Release number of selected component (if applicable): Red Hat Enterprise Virtualization Hypervisor 6.6 (20150128.0.el6ev)
Can you confirm that the host version is RHEL 6.6 or RHEV-H 6.6? Looks like backporting this patch to RHEL6 might help. Bug 1097028 - Don't fail starting domain without cpu, cpuset and cpuacct cgroups controllers https://bugzilla.redhat.com/show_bug.cgi?id=1097028 commit 0c04906fa8872e4f7cdb19f59cbcb26b9bd02b0e Author: Martin Kletzander <mkletzan> Date: Wed Jul 9 09:55:29 2014 +0200 qemu: don't error out when cgroups don't exist
The initial comment states that it's RHEV-H 6.6, which is based on RHEL 6.6.z packages.
(In reply to Karen Noel from comment #5) > commit 0c04906fa8872e4f7cdb19f59cbcb26b9bd02b0e > Author: Martin Kletzander <mkletzan> > Date: Wed Jul 9 09:55:29 2014 +0200 > > qemu: don't error out when cgroups don't exist Unfortunately, this commit won't really help. VDSM wants to use features/APIs (CPU tunning, CPU stats) that require these crgoups to be mounted. We need to investigate why libvirtd does not see them. Since restarting libvirtd solves the issue and restarting the host does not fix it, it looks like cgroups are mounted after libvirtd starts... But that's just a guess, we need to look at the logs.
Hmm, vdsm apparently turned off libvirt's debug logs so I can't see anything in the logs. Debug logs are present until Feb 17 morning but then they suddenly stop; libvirtd was not upgraded at that time, it stayed at 0.10.2-29.el6_5.13 and it was upgraded to 0.10.2-46.el6_6.2 about 45 minutes later. I can't tell if cgroups were mounted at the time libvirtd originally started, according to the sosreport they are currently mounted which explains why restarting libvirtd helps. IMHO this does not look like a libvirt bug (everything works when libvirtd starts at the time we know cgroups are all properly mounted).
libvirtd cannot be upgraded for RHEV-H on it's own. What RHEV does, is deploys new ISO and then reboots the hypervisor, so whatever issue is present it happens at some point during normal operation. https://access.redhat.com/solutions/509343
I'm unable to reproduce the issue locally. I tried installing rhevh-6.6-20150128.0.el6ev.iso and everything works as expected there. Only after stopping cgconfig service I was able to get the errors about missing cgroups, which is of course expected. That said, we'd need some additional data from the host where this reproduces. Please follow the steps below: 1) open /etc/sysconfig/libvirtd in vi and add the following line at the end: cat /proc/mounts >/tmp/libvirtd-mounts 2>/dev/null 2) open /etc/libvirt/libvirtd.conf in vi and add the following lines at the end: log_level=1 log_outputs="1:file:/var/log/libvirt/libvirtd.log" 3) reboot the host 4) get /tmp/libvirtd-mounts from the host and rename it as libvirtd-mounts-boot 5) restart libvirtd service 6) get the /tmp/libvirtd-mounts and /var/log/libvirt/libvirtd.log files from the host 7) attach all files (libvirtd-mounts-boot, libvirt-mounts, libvirtd.log) to this bugzilla 8) revert both changes to the configuration files :-)
(In reply to nijin ashok from comment #12) > Created attachment 1001203 [details] > libvirtd mount point Was this file generated according to the steps in comment #10 (libvirtd-mounts-boot)? In other words, is this a list of mounted filesystems at the time libvirtd was about to start? It's unclear because I asked for two files (before and after libvirtd starts), although it seems to be correct considering the fact that restarting libvirtd service did not help.
(In reply to Jiri Denemark from comment #15) > (In reply to nijin ashok from comment #12) > > Created attachment 1001203 [details] > > libvirtd mount point > > Was this file generated according to the steps in comment #10 > (libvirtd-mounts-boot)? In other words, is this a list of mounted filesystems > at the time libvirtd was about to start? It's unclear because I asked for two > files (before and after libvirtd starts), although it seems to be correct > considering the fact that restarting libvirtd service did not help. Yes, the attached libvirtd-mounts file was generated during the booting process.
Since I'm unable to reproduce the issue and virCgroupDetectMounts works just fine on the data from libvirtd-mounts, I will need some more data to help me understand what's going on here. I added a lot of debugging messages into the code which is supposed to deal with cgroups and I made a scratch build with these changes: http://brewweb.devel.redhat.com/brew/taskinfo?taskID=8861139 Could you install the packages, turn on debug logs (see step 2 in comment 10), reboot the host, and attach the debug logs generated during boot?
Just a reminder... since rebooting the host removes the manually installed test packages from the system, we need to get data from systems where restarting libvirtd service does *not* fix the issue. If restarting libvirtd fixes the issue, the logs we get won't help us much. It doesn't hurt to have them but we desperately need the data from the other hosts.
Also, it would help to get an sosreport from a system which is currently in a failed state. This is important because only at that point we can capture the correct mounts. Please also provide the complete contents of /proc during the failed state.
It was also reported that using RHEL 6.6 on the hosts instead of RHEV-H 6.6 works around the issue. Allan, can you check what versions of vdsm and libvirt packages they installed on the RHEL 6.6 hosts?
Created attachment 1005910 [details] scratch libvirt build with additional debugging.
(In reply to Jiri Denemark from comment #10) … > That said, we'd need some additional data from the host where this > reproduces. Please follow the steps below: > > 1) open /etc/sysconfig/libvirtd in vi and add the following line at the end: > > cat /proc/mounts >/tmp/libvirtd-mounts 2>/dev/null > > 2) open /etc/libvirt/libvirtd.conf in vi and add the following lines at the > end: > > log_level=1 > log_outputs="1:file:/var/log/libvirt/libvirtd.log" > > 3) reboot the host > 4) get /tmp/libvirtd-mounts from the host and rename it as > libvirtd-mounts-boot > 5) restart libvirtd service > 6) get the /tmp/libvirtd-mounts and /var/log/libvirt/libvirtd.log files from > the host > 7) attach all files (libvirtd-mounts-boot, libvirt-mounts, libvirtd.log) to > this bugzilla > 8) revert both changes to the configuration files :-) After reviewing this comment we noticed that this will only catch th elatest call. Please append the line: (date ; cat /proc/mounts ) >> /tmp/libvirtd-mounts in step 1 and reboot.
Problem did not reproduce on following flows: 1. Have 2 hosts installed from USB with rhevh 6.6 20150128 (for rhevm-3.5), engine - vt13.4 2. have 2 hosts installed from USB with rhev-h 6.6 20150123.1 (for rhevm-3.4), engine - vt13.4 have a VM run on host1. Upgrade host2 to rhevh 6.6 20150128, migrate the VM to host2, migrate VM to host1, migrate VM to host 2, upgrade host1 to rhevh 6.6 20150128, migrate VM back to host 1. 3. Same as 2 with engine vt14.1, and hosts installed from USB with rhev-h6.5-20150115.0.el6ev (libvirt 0.10.2-29.el6_5.13) (Since in comment #8 it is mentioned that the previous libvirt version was 0.10.2-29.el6_5.13) I did had an Error seen in vdsm.log/var/log/messages for flow 3: "ERROR::2015-03-26 15:05:53,605::vm::5626::vm.Vm::(_updateDevicesDomxmlCache) vmId=`26e46959-754c-4dd8-943a-39f581184ead`::Alias not found for device type graphics during migration at destination host"
Cont. testing Can not reproduce this bug. Test scenario 4: to run more VMs on it. Hardware info: RHEV-H Host1 on IBM System x3650 M4 CPU Type:Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz CPU Sockets: 2 CPU Cores per Socket: 6 CPU Threads per Core: 2 (SMT Enabled) RHEV-H Host2 on Dell PowerEdge R210 CPU Type: Intel(R) Xeon(R) CPU X3470 @ 2.93GHz CPU Sockets: 1 CPU Cores per Socket: 4 CPU Threads per Core: 2 (SMT Enabled) Test steps: 1. Two hosts are installed RHEVH 6.5 20150115 and added them via rhevm 3.4.5-0.3.el6ev portal, and UP. 2. Run 11 VMs. 3. Upgrade rhevm 3.4.5-0.3.el6ev to rhevm 3.5.0-0.34.el6ev 4. After RHEVM upgrade, two hosts and VMs are still running, check the log, did _not_ contain any cgroup CPUACCT is not mounted error. 5. migrated VMs. 6. Maintenance Host 2, upgrade host 2 to rhevh 6.6 20150128 successful. 7. Host 2 UP, and migrated VMs to Host 2. 8. Maintenance Host 1, upgrade host 1 to rhevh 6.6 20150128 successful. 9. Host 1 UP, and migrated VMs to Host 1. 10. Change Clusters compatibility version from 3.4 to 3.5. 11. Change Data Centers compatibility version from 3.4 to 3.5. 12. Migrated VMs between two hosts. 13. lscgroup can list all cgroups successful, no cgroup CPUACCT is not mounted error. Above always check vdsm.log and messages log on two rhevh 6.6 hosts. Still did not contain any cgroup CPUACCT is not mounted error. And not sure which actions will cause the cgroup CPUACCT is not mounted automatically. The server are still running, we continue tailing it, once we encounter it, we will update the bug comment.
Find something different between rhevh qe's test env and customer's sosreport. Why libvirtd start twice quickly in RHEV-H qe's test env ? 1. RHEV-H libvirtd.log: libvirtd start twice in 1 min. The 2nd time is later than cgroup initialized. ==> 2015-03-27 07:52:22.487+0000: 4589: info : libvirt version: 0.10.2, package: 46.el6_6.2 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2014-11-07-08:34:58, x86-029.build.eng.bos.redhat.com) 2015-03-27 07:52:22.487+0000: 4589: error : virNetSocketNewListenTCP:239 : Unable to resolve address '0.0.0.0' service '16514': Address family for hostname not supported ==> 2015-03-27 07:53:03.880+0000: 5223: info : libvirt version: 0.10.2, package: 46.el6_6.2 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2014-11-07-08:34:58, x86-029.build.eng.bos.redhat.com) root 5212 0.7 0.0 931268 18148 ? - 07:53 1:04 libvirtd --daemon --listen Mar 27 07:52:29 ibm-x3650m4-01 kernel: Initializing cgroup subsys cpuset Mar 27 07:52:29 ibm-x3650m4-01 kernel: Initializing cgroup subsys cpu 2. in customer's sosreport Only 1 time before the cgroup initialized. ==> 2015-02-19 10:09:49.793+0000: 7556: info : libvirt version: 0.10.2, package: 46.el6_6.2 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2014-11-07-08:34:58, x86-029.build.eng.bos.redhat.com) root 7545 0.5 0.0 996876 16848 ? - 10:09 0:54 libvirtd --daemon --listen Feb 19 10:09:58 rhev1 kernel: Initializing cgroup subsys cpuset Feb 19 10:09:58 rhev1 kernel: Initializing cgroup subsys cpu
(In reply to dyuan from comment #49) > Find something different between rhevh qe's test env and customer's > sosreport. > > Why libvirtd start twice quickly in RHEV-H qe's test env ? > > 1. RHEV-H > > libvirtd.log: > > libvirtd start twice in 1 min. The 2nd time is later than cgroup initialized. > > ==> 2015-03-27 07:52:22.487+0000: 4589: info : libvirt version: 0.10.2, > package: 46.el6_6.2 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, > 2014-11-07-08:34:58, x86-029.build.eng.bos.redhat.com) > 2015-03-27 07:52:22.487+0000: 4589: error : virNetSocketNewListenTCP:239 : > Unable to resolve address '0.0.0.0' service '16514': Address family for > hostname not supported > ==> 2015-03-27 07:53:03.880+0000: 5223: info : libvirt version: 0.10.2, > package: 46.el6_6.2 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, > 2014-11-07-08:34:58, x86-029.build.eng.bos.redhat.com) Sorry, please ignore the comment. The 2nd time may be due to the error.
From what I was able to find out, this looks like a bug in RHEV-H because libvirtd service is started twice: - once by vdsm service which is running in parallel to rc3 init sequence (vdsm was perhaps started in background by S01ovirt-early but this is just a guess) - and second time when rc3 init sequence comes to S97libvirtd - since the rc3 init sequence runs S05cgconfig, libvirtd might actually be started by the vdsm service earlier than cgroups are mounted We still have no direct evidence of this happening, but I hope to have it soon.
Perfect, we finally have a proof for my theory in comment 54: root 1 0 2 08:31 ? 00:00:02 /sbin/init root 646 1 6 08:31 ? 00:00:04 /bin/plymouthd --attach-to-session root 2129 1 0 08:31 ? 00:00:00 /sbin/udevd -d root 5300 2129 0 08:32 ? 00:00:00 /sbin/udevd -d root 5303 2129 0 08:32 ? 00:00:00 /sbin/udevd -d root 5627 2129 0 08:32 ? 00:00:00 /sbin/udevd -d root 4822 1 0 08:32 ? 00:00:00 /bin/bash /etc/rc.d/rc 3 root 5535 4822 0 08:32 ? 00:00:00 /bin/bash /etc/rc3.d/S02lvm2-monitor start root 5751 5535 19 08:32 ? 00:00:00 /sbin/vgchange --monitor y --poll y --ignoreskippedcluster --config log{command_names=0 prefix=" "} rhel root 5489 1 0 08:32 ? 00:00:00 /bin/sh /sbin/service vdsmd start root 5494 5489 0 08:32 ? 00:00:00 /bin/sh /etc/init.d/vdsmd start root 5499 5494 0 08:32 ? 00:00:00 /bin/sh /etc/init.d/vdsmd start root 5774 5499 1 08:32 ? 00:00:00 /bin/sh /sbin/service libvirtd start root 5779 5774 1 08:32 ? 00:00:00 /bin/sh /etc/init.d/libvirtd start root 5782 5779 0 08:32 ? 00:00:00 /bin/sh /etc/init.d/libvirtd start libvirtd is started before S05cgconfig when cgroups are not mounted yet; /proc/mounts contains: rootfs / rootfs rw 0 0 proc /proc proc rw,relatime 0 0 sysfs /sys sysfs rw,seclabel,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,relatime,size=132241908k,nr_inodes=33060477,mode=755 0 0 devpts /dev/pts devpts rw,seclabel,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /dev/shm tmpfs rw,seclabel,relatime 0 0 /dev/sdi2 /dev/.initramfs/live ext2 ro,seclabel,relatime,errors=continue 0 0 /dev/mapper/live-rw / ext2 ro,seclabel,relatime,errors=continue,user_xattr,acl 0 0 none /selinux selinuxfs rw,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,relatime,size=132241908k,nr_inodes=33060477,mode=755 0 0 /proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0 none /var/lib/stateless/writable tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/cache/man tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/lock tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/log tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/run tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/lib/dbus tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/lib/nfs tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /tmp tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/cache/hald tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/lib/dhclient tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/tmp tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /media tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/lib/iscsi tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/lib/logrotate.status tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/lib/ntp tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/lib/random-seed tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/spool tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /etc tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/lib/net-snmp tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/lib/dnsmasq tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /root/.ssh tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /root/.uml tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/cache/libvirt tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/lib/libvirt tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/cache/multipathd tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /mnt tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /boot tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /boot-kdump tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /cgroup tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/lib/yum tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/cache/yum tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /usr/share/snmp/mibs tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/lib/lldpad tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/cache/rpcbind tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /usr/share/snmp/mibs tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/lib/lldpad tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/cache/rpcbind tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/cache/rhn tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/db tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /usr/libexec/vdsm/hooks tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /var/lib/vdsm tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 none /rhev/data-center tmpfs rw,rootcontext=system_u:object_r:var_lib_t:s0,seclabel,relatime 0 0 /dev/mapper/HostVG-Config /config ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/fstab ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/shadow ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/default/ovirt ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/vdsm/vdsm.conf ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/sysconfig/keyboard ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /var/lib/random-seed ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/iscsi/initiatorname.iscsi ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/rsyslog.conf ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/libvirt/passwd.db ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/passwd ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/sysconfig/network ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/libvirt/qemu/networks ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/ssh/sshd_config ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/pki ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/hosts ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/resolv.conf ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/ntp.conf ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/sysconfig/iptables ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /var/lib/glusterd ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/sysconfig/virt-who ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/shadow ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/profile ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/sysconfig/network-scripts/ifcfg-lo ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /var/lib/vdsm/persistence ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/vdsm-reg/vdsm-reg.conf ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/multipath.conf ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/sysconfig/rhn/up2date ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /usr/libexec/ovirt-init-functions.sh ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 tmpfs /dev/shm tmpfs rw,rootcontext=system_u:object_r:tmpfs_t:s0,seclabel,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,relatime 0 0 /dev/mapper/HostVG-Logging /var/log ext4 rw,seclabel,noatime,barrier=1,stripe=64,data=ordered 0 0 /dev/mapper/HostVG-Data /data ext4 rw,seclabel,noatime,barrier=1,stripe=64,data=ordered 0 0 /dev/mapper/HostVG-Data /var/lib/libvirt/images ext4 rw,seclabel,noatime,barrier=1,stripe=64,data=ordered 0 0 /dev/mapper/HostVG-Data /var/log/core ext4 rw,seclabel,noatime,barrier=1,stripe=64,data=ordered 0 0 none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0 /dev/mapper/HostVG-Logging /tmp/early-logs ext4 rw,seclabel,noatime,barrier=1,stripe=64,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/ssh/ssh_host_key ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/ssh/ssh_host_key.pub ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/ssh/ssh_host_dsa_key ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/ssh/ssh_host_dsa_key.pub ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/ssh/ssh_host_rsa_key ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/ssh/ssh_host_rsa_key.pub ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/udev/rules.d/71-persistent-node-net.rules ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/sysconfig/network-scripts/ifcfg-eth3 ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/sysconfig/network-scripts/ifcfg-eth2 ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/sysconfig/network-scripts/ifcfg-eth1 ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/sysconfig/network-scripts/ifcfg-eth0 ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /var/lib/vdsm/upgrade/upgrade-unified-persistence ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/vdsm/vdsm.id ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/udev/rules.d/12-ovirt-iosched.rules ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /root/.ssh/authorized_keys ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/sysconfig/rhn/systemid ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 tmpfs /dev/shm tmpfs rw,rootcontext=system_u:object_r:tmpfs_t:s0,seclabel,relatime 0 0 /dev/mapper/HostVG-Data /var/lib/libvirt/images ext4 rw,seclabel,noatime,barrier=1,stripe=64,data=ordered 0 0 /dev/mapper/HostVG-Data /var/log/core ext4 rw,seclabel,noatime,barrier=1,stripe=64,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/libvirt/libvirtd.conf ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/sysconfig/libvirtd ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/libvirt/qemu.conf ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/libvirt/qemu-sanlock.conf ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0 /dev/mapper/HostVG-Config /etc/logrotate.d/libvirtd ext4 rw,seclabel,noatime,barrier=1,stripe=256,data=ordered 0 0
(In reply to Jiri Denemark from comment #63) > Perfect, we finally have a proof for my theory in comment 54: > Jiri - What's the relevant output in this post, and what how can I verify an image? It's not clear from your comment whether you interrupted service startup, ran it after login, or somewhere else, and I've never been able to reproduce this bug.
The libvirtd service should not ever be started before cgconfig (ideally, it should just be started at the end of init sequence as would the S97libvirtd siggest). I've never reproduced this either, but it looks like there is a race between the standard init sequence and the hacks which start libvirtd in parallel to it. Except for asking the customer who hit the bug to try with the new image, the only way (which I can think of) to verify this is to make the standard init sequence slower... something like "sleep 60" at the beginning of start section in cgconfig init script. I'm not sure if this is possible on RHEV-H, though.
(In reply to Jiri Denemark from comment #71) > The libvirtd service should not ever be started before cgconfig (ideally, it > should just be started at the end of init sequence as would the S97libvirtd > siggest). I've never reproduced this either, but it looks like there is a > race between the standard init sequence and the hacks which start libvirtd > in parallel to it. Except for asking the customer who hit the bug to try > with the new image, the only way (which I can think of) to verify this is to > make the standard init sequence slower... something like "sleep 60" at the > beginning of start section in cgconfig init script. I'm not sure if this is > possible on RHEV-H, though. Well, it is possible, but: I'm asked because a recent change moved the hooks which start vdsm (which starts libvirt) to S98ovirt-post, with libvirtd as a Required-Start, which should resolve/remove any race conditions there. We certainly can patch init scripts, but hopefully we wouldn't need to. Thanks.
I see, that sounds like the right solution. One more thought, the log from comment 63 shows that libvirtd was started when lvm was still being initialized so I guess the bug could be reproduced on a host where it takes a long time to initialize lvm (I'm not sure what setup would do that, though)?
the cgroups issue isn't related to https://bugzilla.redhat.com/show_bug.cgi?id=1209486
According to comment 74 this specific bug is fixed, thus I am moving this bug to MODIFIED.
According to comment 67, comment 68, comment 74 and comment 77, the test builds are already provided to customer, and customer no longer encounter CPUACCT controller is not mounted errors flood in vdsm.log. We can not reproduce it in-house, so here add OtherQA keywords, and QE have to do sanity testing to verify this bug. sanity testing will be executed as the following scenarios: 1. restart rhevh 6.6 for 3.5.1 build more times(20+), to check the race condition whether or not happen. 2. upgrade rhevh 6.5 to rhevh 6.6 build, then check vdsm.log and message log. 3. upgrade rhevh 6.6 3.5.0 to rhevh 6.6 3.5.1, then check vdsm.log and message log. Currently we already knew this build rhev-hypervisor6-6.6-20150402.0.el6ev has network issue, see bug 1209486, so sanity testing to verify this bug will be executed when rhevh grab the coming vdsm which include the fix of bug 1209486. Thanks.
the bug was probably cloned incorrectly, but there is a z-stream bug 1213431 for this issue
*** Bug 1235718 has been marked as a duplicate of this bug. ***