Description of problem: In order to verify the https://bugzilla.redhat.com/show_bug.cgi?id=1317468, I've tried to create 300 SDs. Creation of 300 storage domains, failed during creation of 54th storage domain. VDSM alma04.qa.lab.tlv.redhat.com command failed: Cannot reconstruct master domain: (u"spUUID=00000001-0001-0001-0001-00000000011d, masterDom=4bf872c6-c931-4203-aac4-436f19d9e683, masterVersion=3, clusterlock params: ({'LEASETIMESEC': 60, 'IOOPTIMEOUTSEC': 10, 'LEASERETRIES': 3, 'LOCKRENEWALINTERVALSEC': 5})",) systemctl status libvirtd -l ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/libvirtd.service.d └─unlimited-core.conf Active: active (running) since Tue 2016-05-24 08:24:38 IDT; 7h ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 1605 (libvirtd) CGroup: /system.slice/libvirtd.service └─1605 /usr/sbin/libvirtd --listen May 24 08:24:36 alma04.qa.lab.tlv.redhat.com systemd[1]: Starting Virtualization daemon... May 24 08:24:37 alma04.qa.lab.tlv.redhat.com libvirtd[1605]: libvirt version: 1.2.17, package: 13.el7_2.4 (CentOS BuildSystem <http://bugs.centos.org>, 2016-03-31-16:56:26, worker1.bsys.centos.org) May 24 08:24:37 alma04.qa.lab.tlv.redhat.com libvirtd[1605]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so not accessible May 24 08:24:38 alma04.qa.lab.tlv.redhat.com systemd[1]: Started Virtualization daemon. May 24 08:34:16 alma04.qa.lab.tlv.redhat.com libvirtd[1605]: Failed to acquire lock: File exists May 24 15:53:08 alma04.qa.lab.tlv.redhat.com libvirtd[1605]: End of file while reading data: Input/output error systemctl status vdsmd -l ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2016-05-24 08:24:41 IDT; 7h ago Process: 2840 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) Main PID: 2913 (vdsm) CGroup: /system.slice/vdsmd.service ├─ 1043 /usr/libexec/ioprocess --read-pipe-fd 666 --write-pipe-fd 663 --max-threads 10 --max-queued-requests 10 ├─ 2288 /usr/libexec/ioprocess --read-pipe-fd 674 --write-pipe-fd 672 --max-threads 10 --max-queued-requests 10 ├─ 2876 /usr/libexec/ioprocess --read-pipe-fd 683 --write-pipe-fd 682 --max-threads 10 --max-queued-requests 10 ├─ 2913 /usr/bin/python /usr/share/vdsm/vdsm ├─ 3125 /usr/libexec/ioprocess --read-pipe-fd 71 --write-pipe-fd 70 --max-threads 10 --max-queued-requests 10 ├─ 3826 /usr/libexec/ioprocess --read-pipe-fd 690 --write-pipe-fd 688 --max-threads 10 --max-queued-requests 10 ├─ 4827 /usr/libexec/ioprocess --read-pipe-fd 699 --write-pipe-fd 698 --max-threads 10 --max-queued-requests 10 ├─ 5897 /usr/libexec/ioprocess --read-pipe-fd 712 --write-pipe-fd 711 --max-threads 10 --max-queued-requests 10 ├─ 6853 /usr/libexec/ioprocess --read-pipe-fd 719 --write-pipe-fd 717 --max-threads 10 --max-queued-requests 10 ├─ 7857 /usr/libexec/ioprocess --read-pipe-fd 729 --write-pipe-fd 728 --max-threads 10 --max-queued-requests 10 ├─ 8755 /usr/libexec/ioprocess --read-pipe-fd 736 --write-pipe-fd 734 --max-threads 10 --max-queued-requests 10 ├─ 9831 /usr/libexec/ioprocess --read-pipe-fd 746 --write-pipe-fd 745 --max-threads 10 --max-queued-requests 10 ├─ 10864 /usr/libexec/ioprocess --read-pipe-fd 754 --write-pipe-fd 753 --max-threads 10 --max-queued-requests 10 ├─ 12212 /usr/libexec/ioprocess --read-pipe-fd 763 --write-pipe-fd 762 --max-threads 10 --max-queued-requests 10 ├─ 13006 /usr/libexec/ioprocess --read-pipe-fd 771 --write-pipe-fd 770 --max-threads 10 --max-queued-requests 10 ├─ 14050 /usr/libexec/ioprocess --read-pipe-fd 782 --write-pipe-fd 781 --max-threads 10 --max-queued-requests 10 ├─ 15370 /usr/libexec/ioprocess --read-pipe-fd 784 --write-pipe-fd 782 --max-threads 10 --max-queued-requests 10 ├─ 16660 /usr/libexec/ioprocess --read-pipe-fd 792 --write-pipe-fd 789 --max-threads 10 --max-queued-requests 10 ├─ 17661 /usr/libexec/ioprocess --read-pipe-fd 803 --write-pipe-fd 802 --max-threads 10 --max-queued-requests 10 ├─ 19145 /usr/libexec/ioprocess --read-pipe-fd 811 --write-pipe-fd 810 --max-threads 10 --max-queued-requests 10 ├─ 20430 /usr/libexec/ioprocess --read-pipe-fd 819 --write-pipe-fd 818 --max-threads 10 --max-queued-requests 10 ├─ 21603 /usr/libexec/ioprocess --read-pipe-fd 828 --write-pipe-fd 827 --max-threads 10 --max-queued-requests 10 ├─ 22673 /usr/libexec/ioprocess --read-pipe-fd 837 --write-pipe-fd 836 --max-threads 10 --max-queued-requests 10 ├─ 23788 /usr/libexec/ioprocess --read-pipe-fd 846 --write-pipe-fd 845 --max-threads 10 --max-queued-requests 10 ├─ 25266 /usr/libexec/ioprocess --read-pipe-fd 856 --write-pipe-fd 855 --max-threads 10 --max-queued-requests 10 ├─ 26449 /usr/libexec/ioprocess --read-pipe-fd 866 --write-pipe-fd 865 --max-threads 10 --max-queued-requests 10 ├─ 27866 /usr/libexec/ioprocess --read-pipe-fd 875 --write-pipe-fd 874 --max-threads 10 --max-queued-requests 10 ├─ 29236 /usr/libexec/ioprocess --read-pipe-fd 884 --write-pipe-fd 883 --max-threads 10 --max-queued-requests 10 ├─ 30486 /usr/libexec/ioprocess --read-pipe-fd 892 --write-pipe-fd 891 --max-threads 10 --max-queued-requests 10 ├─ 31840 /usr/libexec/ioprocess --read-pipe-fd 900 --write-pipe-fd 899 --max-threads 10 --max-queued-requests 10 ├─ 33230 /usr/libexec/ioprocess --read-pipe-fd 908 --write-pipe-fd 907 --max-threads 10 --max-queued-requests 10 ├─ 34428 /usr/libexec/ioprocess --read-pipe-fd 916 --write-pipe-fd 915 --max-threads 10 --max-queued-requests 10 ├─ 36039 /usr/libexec/ioprocess --read-pipe-fd 924 --write-pipe-fd 923 --max-threads 10 --max-queued-requests 10 ├─ 37356 /usr/libexec/ioprocess --read-pipe-fd 932 --write-pipe-fd 931 --max-threads 10 --max-queued-requests 10 ├─ 38783 /usr/libexec/ioprocess --read-pipe-fd 941 --write-pipe-fd 940 --max-threads 10 --max-queued-requests 10 ├─ 40028 /usr/libexec/ioprocess --read-pipe-fd 950 --write-pipe-fd 949 --max-threads 10 --max-queued-requests 10 ├─ 41754 /usr/libexec/ioprocess --read-pipe-fd 958 --write-pipe-fd 957 --max-threads 10 --max-queued-requests 10 ├─ 43305 /usr/libexec/ioprocess --read-pipe-fd 969 --write-pipe-fd 968 --max-threads 10 --max-queued-requests 10 ├─ 44681 /usr/libexec/ioprocess --read-pipe-fd 971 --write-pipe-fd 969 --max-threads 10 --max-queued-requests 10 ├─ 46296 /usr/libexec/ioprocess --read-pipe-fd 983 --write-pipe-fd 982 --max-threads 10 --max-queued-requests 10 ├─ 47736 /usr/libexec/ioprocess --read-pipe-fd 988 --write-pipe-fd 985 --max-threads 10 --max-queued-requests 10 ├─ 49326 /usr/libexec/ioprocess --read-pipe-fd 999 --write-pipe-fd 998 --max-threads 10 --max-queued-requests 10 ├─ 51123 /usr/libexec/ioprocess --read-pipe-fd 1006 --write-pipe-fd 1004 --max-threads 10 --max-queued-requests 10 ├─ 53969 /usr/libexec/ioprocess --read-pipe-fd 490 --write-pipe-fd 242 --max-threads 10 --max-queued-requests 10 ├─ 65118 /usr/libexec/ioprocess --read-pipe-fd 93 --write-pipe-fd 92 --max-threads 10 --max-queued-requests 10 ├─ 92547 /usr/libexec/ioprocess --read-pipe-fd 40 --write-pipe-fd 39 --max-threads 10 --max-queued-requests 10 ├─ 92573 /usr/libexec/ioprocess --read-pipe-fd 48 --write-pipe-fd 47 --max-threads 10 --max-queued-requests 10 ├─ 92580 /usr/libexec/ioprocess --read-pipe-fd 55 --write-pipe-fd 53 --max-threads 10 --max-queued-requests 10 ├─ 92588 /usr/libexec/ioprocess --read-pipe-fd 65 --write-pipe-fd 63 --max-threads 10 --max-queued-requests 10 ├─ 92595 /usr/libexec/ioprocess --read-pipe-fd 85 --write-pipe-fd 83 --max-threads 10 --max-queued-requests 10 ├─ 92602 /usr/libexec/ioprocess --read-pipe-fd 109 --write-pipe-fd 108 --max-threads 10 --max-queued-requests 10 ├─ 93544 /usr/libexec/ioprocess --read-pipe-fd 125 --write-pipe-fd 124 --max-threads 10 --max-queued-requests 10 ├─ 93597 /usr/libexec/ioprocess --read-pipe-fd 128 --write-pipe-fd 126 --max-threads 10 --max-queued-requests 10 ├─ 93870 /usr/libexec/ioprocess --read-pipe-fd 140 --write-pipe-fd 139 --max-threads 10 --max-queued-requests 10 ├─ 93944 /usr/libexec/ioprocess --read-pipe-fd 148 --write-pipe-fd 147 --max-threads 10 --max-queued-requests 10 ├─ 94217 /usr/libexec/ioprocess --read-pipe-fd 161 --write-pipe-fd 160 --max-threads 10 --max-queued-requests 10 ├─ 94298 /usr/libexec/ioprocess --read-pipe-fd 161 --write-pipe-fd 157 --max-threads 10 --max-queued-requests 10 ├─ 94611 /usr/libexec/ioprocess --read-pipe-fd 172 --write-pipe-fd 170 --max-threads 10 --max-queued-requests 10 ├─ 94684 /usr/libexec/ioprocess --read-pipe-fd 181 --write-pipe-fd 180 --max-threads 10 --max-queued-requests 10 ├─ 94960 /usr/libexec/ioprocess --read-pipe-fd 189 --write-pipe-fd 188 --max-threads 10 --max-queued-requests 10 ├─ 95100 /usr/libexec/ioprocess --read-pipe-fd 198 --write-pipe-fd 197 --max-threads 10 --max-queued-requests 10 ├─ 95379 /usr/libexec/ioprocess --read-pipe-fd 205 --write-pipe-fd 203 --max-threads 10 --max-queued-requests 10 ├─ 95499 /usr/libexec/ioprocess --read-pipe-fd 214 --write-pipe-fd 213 --max-threads 10 --max-queued-requests 10 ├─ 95837 /usr/libexec/ioprocess --read-pipe-fd 221 --write-pipe-fd 219 --max-threads 10 --max-queued-requests 10 ├─ 95974 /usr/libexec/ioprocess --read-pipe-fd 230 --write-pipe-fd 229 --max-threads 10 --max-queued-requests 10 ├─ 96295 /usr/libexec/ioprocess --read-pipe-fd 235 --write-pipe-fd 232 --max-threads 10 --max-queued-requests 10 ├─ 96491 /usr/libexec/ioprocess --read-pipe-fd 248 --write-pipe-fd 247 --max-threads 10 --max-queued-requests 10 ├─ 96861 /usr/libexec/ioprocess --read-pipe-fd 258 --write-pipe-fd 257 --max-threads 10 --max-queued-requests 10 ├─ 97058 /usr/libexec/ioprocess --read-pipe-fd 265 --write-pipe-fd 263 --max-threads 10 --max-queued-requests 10 ├─ 97435 /usr/libexec/ioprocess --read-pipe-fd 273 --write-pipe-fd 272 --max-threads 10 --max-queued-requests 10 ├─ 97600 /usr/libexec/ioprocess --read-pipe-fd 282 --write-pipe-fd 281 --max-threads 10 --max-queued-requests 10 ├─ 98002 /usr/libexec/ioprocess --read-pipe-fd 291 --write-pipe-fd 290 --max-threads 10 --max-queued-requests 10 ├─ 98232 /usr/libexec/ioprocess --read-pipe-fd 297 --write-pipe-fd 295 --max-threads 10 --max-queued-requests 10 ├─ 98658 /usr/libexec/ioprocess --read-pipe-fd 305 --write-pipe-fd 304 --max-threads 10 --max-queued-requests 10 ├─ 98896 /usr/libexec/ioprocess --read-pipe-fd 314 --write-pipe-fd 313 --max-threads 10 --max-queued-requests 10 ├─ 99337 /usr/libexec/ioprocess --read-pipe-fd 321 --write-pipe-fd 319 --max-threads 10 --max-queued-requests 10 ├─ 99588 /usr/libexec/ioprocess --read-pipe-fd 330 --write-pipe-fd 329 --max-threads 10 --max-queued-requests 10 ├─ 99985 /usr/libexec/ioprocess --read-pipe-fd 340 --write-pipe-fd 339 --max-threads 10 --max-queued-requests 10 ├─100273 /usr/libexec/ioprocess --read-pipe-fd 345 --write-pipe-fd 342 --max-threads 10 --max-queued-requests 10 ├─100790 /usr/libexec/ioprocess --read-pipe-fd 362 --write-pipe-fd 361 --max-threads 10 --max-queued-requests 10 ├─101086 /usr/libexec/ioprocess --read-pipe-fd 365 --write-pipe-fd 363 --max-threads 10 --max-queued-requests 10 ├─101554 /usr/libexec/ioprocess --read-pipe-fd 378 --write-pipe-fd 377 --max-threads 10 --max-queued-requests 10 ├─101906 /usr/libexec/ioprocess --read-pipe-fd 383 --write-pipe-fd 380 --max-threads 10 --max-queued-requests 10 ├─102428 /usr/libexec/ioprocess --read-pipe-fd 396 --write-pipe-fd 395 --max-threads 10 --max-queued-requests 10 ├─102739 /usr/libexec/ioprocess --read-pipe-fd 398 --write-pipe-fd 396 --max-threads 10 --max-queued-requests 10 ├─103272 /usr/libexec/ioprocess --read-pipe-fd 414 --write-pipe-fd 413 --max-threads 10 --max-queued-requests 10 ├─103620 /usr/libexec/ioprocess --read-pipe-fd 414 --write-pipe-fd 413 --max-threads 10 --max-queued-requests 10 ├─104164 /usr/libexec/ioprocess --read-pipe-fd 425 --write-pipe-fd 423 --max-threads 10 --max-queued-requests 10 ├─104613 /usr/libexec/ioprocess --read-pipe-fd 436 --write-pipe-fd 435 --max-threads 10 --max-queued-requests 10 ├─105232 /usr/libexec/ioprocess --read-pipe-fd 441 --write-pipe-fd 438 --max-threads 10 --max-queued-requests 10 ├─105572 /usr/libexec/ioprocess --read-pipe-fd 450 --write-pipe-fd 449 --max-threads 10 --max-queued-requests 10 ├─106157 /usr/libexec/ioprocess --read-pipe-fd 457 --write-pipe-fd 455 --max-threads 10 --max-queued-requests 10 ├─106567 /usr/libexec/ioprocess --read-pipe-fd 466 --write-pipe-fd 465 --max-threads 10 --max-queued-requests 10 ├─107201 /usr/libexec/ioprocess --read-pipe-fd 471 --write-pipe-fd 468 --max-threads 10 --max-queued-requests 10 ├─107668 /usr/libexec/ioprocess --read-pipe-fd 481 --write-pipe-fd 480 --max-threads 10 --max-queued-requests 10 ├─108259 /usr/libexec/ioprocess --read-pipe-fd 488 --write-pipe-fd 486 --max-threads 10 --max-queued-requests 10 ├─108737 /usr/libexec/ioprocess --read-pipe-fd 497 --write-pipe-fd 496 --max-threads 10 --max-queued-requests 10 ├─109396 /usr/libexec/ioprocess --read-pipe-fd 506 --write-pipe-fd 505 --max-threads 10 --max-queued-requests 10 ├─109884 /usr/libexec/ioprocess --read-pipe-fd 514 --write-pipe-fd 513 --max-threads 10 --max-queued-requests 10 ├─110656 /usr/libexec/ioprocess --read-pipe-fd 525 --write-pipe-fd 524 --max-threads 10 --max-queued-requests 10 ├─111238 /usr/libexec/ioprocess --read-pipe-fd 530 --write-pipe-fd 527 --max-threads 10 --max-queued-requests 10 ├─111900 /usr/libexec/ioprocess --read-pipe-fd 545 --write-pipe-fd 544 --max-threads 10 --max-queued-requests 10 ├─112586 /usr/libexec/ioprocess --read-pipe-fd 544 --write-pipe-fd 541 --max-threads 10 --max-queued-requests 10 ├─113241 /usr/libexec/ioprocess --read-pipe-fd 557 --write-pipe-fd 556 --max-threads 10 --max-queued-requests 10 ├─113814 /usr/libexec/ioprocess --read-pipe-fd 565 --write-pipe-fd 564 --max-threads 10 --max-queued-requests 10 ├─114548 /usr/libexec/ioprocess --read-pipe-fd 576 --write-pipe-fd 575 --max-threads 10 --max-queued-requests 10 ├─115139 /usr/libexec/ioprocess --read-pipe-fd 581 --write-pipe-fd 578 --max-threads 10 --max-queued-requests 10 ├─115867 /usr/libexec/ioprocess --read-pipe-fd 592 --write-pipe-fd 591 --max-threads 10 --max-queued-requests 10 ├─116583 /usr/libexec/ioprocess --read-pipe-fd 600 --write-pipe-fd 599 --max-threads 10 --max-queued-requests 10 ├─117561 /usr/libexec/ioprocess --read-pipe-fd 608 --write-pipe-fd 607 --max-threads 10 --max-queued-requests 10 ├─118314 /usr/libexec/ioprocess --read-pipe-fd 616 --write-pipe-fd 615 --max-threads 10 --max-queued-requests 10 ├─119238 /usr/libexec/ioprocess --read-pipe-fd 623 --write-pipe-fd 621 --max-threads 10 --max-queued-requests 10 ├─120037 /usr/libexec/ioprocess --read-pipe-fd 632 --write-pipe-fd 631 --max-threads 10 --max-queued-requests 10 ├─121020 /usr/libexec/ioprocess --read-pipe-fd 641 --write-pipe-fd 640 --max-threads 10 --max-queued-requests 10 ├─121683 /usr/libexec/ioprocess --read-pipe-fd 650 --write-pipe-fd 649 --max-threads 10 --max-queued-requests 10 └─122603 /usr/libexec/ioprocess --read-pipe-fd 661 --write-pipe-fd 660 --max-threads 10 --max-queued-requests 10 May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter memFree is not uint type May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter free is not uint type May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter free is not uint type May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter free is not uint type May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter free is not uint type May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter cpuSysVdsmd is not float type May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Required property haStatus is not provided when calling Host.getStats May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter bootTime is not uint type May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter rxDropped is not uint type May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter anonHugePages is not uint type Version-Release number of selected component (if applicable): Engine: rhevm-setup-plugins-4.0.0-0.3.alpha.el7ev.noarch rhevm-dependencies-4.0.0-0.1.alpha.git9ae0cc3.el7ev.noarch rhevm-4.0.0-0.6.master.el7ev.noarch rhevm-doc-3.6.0-7.el6eng.noarch rhevm-branding-rhev-4.0.0-0.0.master.20160219183625.el7ev.noarch postfix-2.10.1-6.el7.x86_64 ovirt-engine-extensions-api-impl-4.0.0-0.6.master.el7ev.noarch ovirt-host-deploy-1.5.0-0.2.alpha1.git968f12d.el7ev.noarch ovirt-engine-tools-backup-4.0.0-0.6.master.el7ev.noarch ovirt-engine-tools-4.0.0-0.6.master.el7ev.noarch ovirt-iso-uploader-4.0.0-0.2.master.git98e29da.el7ev.noarch ovirt-host-deploy-java-1.5.0-0.2.alpha1.git968f12d.el7ev.noarch ovirt-vmconsole-1.0.2-1.el7ev.noarch ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.0.0-0.6.master.el7ev.noarch ovirt-setup-lib-1.0.1-1.el7ev.noarch ovirt-engine-dwh-setup-4.0.0-1.el7ev.noarch ovirt-engine-websocket-proxy-4.0.0-0.6.master.el7ev.noarch ovirt-engine-restapi-4.0.0-0.6.master.el7ev.noarch ovirt-engine-vmconsole-proxy-helper-4.0.0-0.6.master.el7ev.noarch ovirt-engine-backend-4.0.0-0.6.master.el7ev.noarch ovirt-engine-setup-base-4.0.0-0.6.master.el7ev.noarch ovirt-log-collector-4.0.0-0.1.master.git668ba27.el7ev.noarch ovirt-engine-sdk-python-3.6.5.0-1.el7ev.noarch ovirt-engine-extension-aaa-jdbc-1.1.0-0.3.el7ev.noarch ovirt-engine-setup-plugin-ovirt-engine-common-4.0.0-0.6.master.el7ev.noarch ovirt-engine-setup-plugin-websocket-proxy-4.0.0-0.6.master.el7ev.noarch ovirt-engine-dbscripts-4.0.0-0.6.master.el7ev.noarch ovirt-engine-setup-4.0.0-0.6.master.el7ev.noarch ovirt-engine-userportal-4.0.0-0.6.master.el7ev.noarch ovirt-image-uploader-4.0.0-0.1.master.gita3e72cd.el7ev.noarch ovirt-vmconsole-proxy-1.0.2-1.el7ev.noarch ovirt-engine-setup-plugin-ovirt-engine-4.0.0-0.6.master.el7ev.noarch ovirt-engine-dwh-4.0.0-1.el7ev.noarch ovirt-engine-webadmin-portal-4.0.0-0.6.master.el7ev.noarch ovirt-engine-4.0.0-0.6.master.el7ev.noarch ovirt-engine-cli-3.6.2.0-1.el7ev.noarch ovirt-engine-lib-4.0.0-0.6.master.el7ev.noarch Linux version 3.10.0-327.22.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Mon May 16 13:31:48 EDT 2016 Linux 3.10.0-327.22.1.el7.x86_64 #1 SMP Mon May 16 13:31:48 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 7.2 (Maipo) Host: ovirt-vmconsole-host-1.0.2-0.0.master.20160517094103.git06df50a.el7.noarch ovirt-setup-lib-1.0.2-0.0.master.20160502125738.gitf05af9e.el7.centos.noarch ovirt-release40-4.0.0-0.3.beta1.noarch ovirt-vmconsole-1.0.2-0.0.master.20160517094103.git06df50a.el7.noarch ovirt-engine-sdk-python-3.6.5.1-0.1.20160507.git5fb7e0e.el7.centos.noarch ovirt-host-deploy-1.5.0-0.1.alpha1.el7.centos.noarch ovirt-hosted-engine-setup-2.0.0-0.1.beta1.el7.centos.noarch ovirt-release-host-node-4.0.0-0.3.beta1.el7.noarch ovirt-hosted-engine-ha-2.0.0-0.1.beta1.el7.centos.noarch ovirt-node-ng-image-update-placeholder-4.0.0-0.3.beta1.el7.noarch libvirt-client-1.2.17-13.el7_2.4.x86_64 mom-0.5.3-1.1.el7.noarch vdsm-4.17.999-1155.gitcf216a0.el7.centos.x86_64 sanlock-3.2.4-2.el7_2.x86_64 qemu-kvm-tools-ev-2.3.0-31.el7_2.10.1.x86_64 qemu-kvm-ev-2.3.0-31.el7_2.10.1.x86_64 qemu-kvm-common-ev-2.3.0-31.el7_2.10.1.x86_64 Linux version 3.10.0-327.18.2.el7.x86_64 (builder.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu May 12 11:03:55 UTC 2016 Linux 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux CentOS Linux release 7.2.1511 (Core) How reproducible: 100% Steps to Reproduce: 1.Deploy 4.0HE over NFS on NGN 4.0. 2.Add one data storage domain (SD) to get hosted_storage auto-imported and to get the HE-VM visible. 3.Create 300 NFS SDs. Actual results: On creation of 54th SD the host failed to continue and all SDs previously active, went in to Inactive state. Expected results: Creation of 300 SDs should succeed. Additional info:
Created attachment 1161044 [details] sosreport from engine
Adding external link to host's logs https://drive.google.com/a/redhat.com/file/d/0B85BEaDBcF88bFpYOGItN2lOcUU/view?usp=sharing
Freddy, as this weeks QE contact please have a look at the logs. Targeting for now to 4.1 till we know better what's going on here
The relevant part of the log: Traceback (most recent call last): File "/usr/share/vdsm/storage/task.py", line 875, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/storage/hsm.py", line 987, in connectStoragePool spUUID, hostID, msdUUID, masterVersion, domainsMap) File "/usr/share/vdsm/storage/hsm.py", line 1052, in _connectStoragePool res = pool.connect(hostID, msdUUID, masterVersion) File "/usr/share/vdsm/storage/sp.py", line 664, in connect self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) File "/usr/share/vdsm/storage/sp.py", line 1237, in __rebuild self.setMasterDomain(msdUUID, masterVersion) File "/usr/share/vdsm/storage/sp.py", line 1445, in setMasterDomain domain = sdCache.produce(msdUUID) File "/usr/share/vdsm/storage/sdc.py", line 101, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce self.refreshStorage() File "/usr/lib/python2.7/site-packages/vdsm/storage/misc.py", line 616, in helper return sm(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/misc.py", line 601, in __call__ self.__lastResult = self.__func(*args, **kwargs) File "/usr/share/vdsm/storage/sdc.py", line 84, in refreshStorage multipath.rescan() File "/usr/share/vdsm/storage/multipath.py", line 74, in rescan iscsi.rescan() File "/usr/lib/python2.7/site-packages/vdsm/storage/misc.py", line 616, in helper return sm(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/misc.py", line 601, in __call__ self.__lastResult = self.__func(*args, **kwargs) File "/usr/share/vdsm/storage/iscsi.py", line 435, in rescan rescanOp = iscsiadm.session_rescan_async() File "/usr/share/vdsm/storage/iscsiadm.py", line 340, in session_rescan_async proc = _runCmd(["-m", "session", "-R"], sync=False) File "/usr/share/vdsm/storage/iscsiadm.py", line 119, in _runCmd return misc.execCmd(cmd, printable=printCmd, sudo=True, sync=sync) File "/usr/lib/python2.7/site-packages/vdsm/commands.py", line 71, in execCmd deathSignal=deathSignal, childUmask=childUmask) File "/usr/lib64/python2.7/site-packages/cpopen/__init__.py", line 63, in __init__ **kw) File "/usr/lib64/python2.7/subprocess.py", line 703, in __init__ errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr) File "/usr/lib64/python2.7/subprocess.py", line 1112, in _get_handles p2cread, p2cwrite = self.pipe_cloexec() File "/usr/lib64/python2.7/subprocess.py", line 1168, in pipe_cloexec r, w = os.pipe() OSError: [Errno 24] Too many open files I wonder if we're leaking an FD somewhere.
Reducing severity. >50 SDs is quite rare.
Is this a regression?
Created attachment 1161328 [details] lsof output
How did you create the Storage Domains ?
vdsm process is limited to 1024 fd [root@RHEL7 ~]# cat /proc/23522/limits Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 8388608 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 7280 7280 processes Max open files 1024 4096 files Max locked memory 65536 65536 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 7280 7280 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
I also reached the "too may open files" when creating + 55 File SD. The ulimit of the vdsm process is 1024 (see comment #9). It seems it has been changed a long time ago by this patch: https://gerrit.ovirt.org/#/c/914/2 It sets the limit to the user not to the process.
(In reply to Yaniv Kaul from comment #6) > Is this a regression? Yes, it is a regression, as previously at least 100 SDs creation at a single servere was possible and worked. (In reply to Fred Rolland from comment #8) > How did you create the Storage Domains ? By using Selenium and proper REST-APIs. I've seen live that first 53 were created OK, then they were attached to DC and became Active, no problem, but at the 54th they all became Inactive.
What type of SDs (File/Block) were created on the 100 SDs case ?
Related to bug 1339777?
(In reply to Allon Mureinik from comment #13) > Related to bug 1339777? No, this this is leak of exactly one fd for each ioprocess, so this is 2 fds max per storage domain. We create 2 ioprocess per domain - one for the mount point, stopped after 1 minute of idle time, and the second for the domain, kept alive by the monitor.
(In reply to Nikolai Sednev from comment #0) > Description of problem: > In order to verify the https://bugzilla.redhat.com/show_bug.cgi?id=1317468, > I've tried to create 300 SDs. We don't support this amount. > Expected results: > Creation of 300 SDs should succeed. It is not specified, and the system will not be useable in this state. Nikolai, check with Yaniv what are the official limits.
Workaround : - Edit : /lib/systemd/system/vdsmd.service - Add following lines to ‘[Service]‘: LimitNOFILE=4096 - Reload daemon: systemctl daemon-reload - Reload service: systemctl restart vdsmd
(In reply to Nir Soffer from comment #15) > (In reply to Nikolai Sednev from comment #0) > > Description of problem: > > In order to verify the https://bugzilla.redhat.com/show_bug.cgi?id=1317468, > > I've tried to create 300 SDs. > > We don't support this amount. What is maximum supported amount for VGs in RHEVM? > > > Expected results: > > Creation of 300 SDs should succeed. > > It is not specified, and the system will not be useable in this state. > > Nikolai, check with Yaniv what are the official limits. Anyway, system should no hang or disconnect the existing resources, instead of this it should bring warning of maximum number of VGs been reached, further addition is not supported. Yaniv, can you please clarify this issue?
(In reply to Nikolai Sednev from comment #17) > (In reply to Nir Soffer from comment #15) > > (In reply to Nikolai Sednev from comment #0) > > > Description of problem: > > > In order to verify the https://bugzilla.redhat.com/show_bug.cgi?id=1317468, > > > I've tried to create 300 SDs. > > > > We don't support this amount. > > What is maximum supported amount for VGs in RHEVM? VGs ? we should aim to support more than 50. This depends on the HW in any case. > > > > > Expected results: > > > Creation of 300 SDs should succeed. > > > > It is not specified, and the system will not be useable in this state. > > > > Nikolai, check with Yaniv what are the official limits. > Anyway, system should no hang or disconnect the existing resources, instead > of this it should bring warning of maximum number of VGs been reached, > further addition is not supported. > > Yaniv, can you please clarify this issue? We can't enforce this. It depends on the storage and hosts specs.
(In reply to Yaniv Dary from comment #18) > (In reply to Nikolai Sednev from comment #17) > > (In reply to Nir Soffer from comment #15) > > > (In reply to Nikolai Sednev from comment #0) > > > > Description of problem: > > > > In order to verify the https://bugzilla.redhat.com/show_bug.cgi?id=1317468, > > > > I've tried to create 300 SDs. > > > > > > We don't support this amount. > > > > What is maximum supported amount for VGs in RHEVM? > > VGs ? we should aim to support more than 50. This depends on the HW in any > case. > > > > > > > > Expected results: > > > > Creation of 300 SDs should succeed. > > > > > > It is not specified, and the system will not be useable in this state. > > > > > > Nikolai, check with Yaniv what are the official limits. > > Anyway, system should no hang or disconnect the existing resources, instead > > of this it should bring warning of maximum number of VGs been reached, > > further addition is not supported. > > > > Yaniv, can you please clarify this issue? > > We can't enforce this. It depends on the storage and hosts specs. VGs=Volume Groups=Storage Domains within the engine. I've seen that on host the memory was used ~6Gig from 32Gig available, all other resources were freely available. IMHO the system crushed and that what should not happen, no active data storage domains should be disconnected. So what customer should expect and are there any strict limitations available and documented?
(In reply to Nikolai Sednev from comment #19) > > So what customer should expect and are there any strict limitations > available and documented? We don't have a published sizing guide, but it is the number we have been requesting scale team to test and development to make sure works well.
Well I have one engine with currently around 40 Storage domains in a live 3.3.3. deployment and plan to go much beyond that in my (now at testing stage) 3.6.z deployment. So I really hope you can push the numbers at least to 300 :) I think (from reading this BZ, please correct me if I'm wrong) we have 2 issues at hand: 1. ioprocess leaking 2 fds per SD (can this be stopped?) 2. fd limit of 1024 of vdsm, which should maybe get increased (each administrator can increase it himself, as a workaround, but I bet it get's overwritten on every vdsm update?) HTH Sven
(In reply to Sven Kieske from comment #21) > Well I have one engine with currently around 40 Storage domains in a live > 3.3.3. > deployment and plan to go much beyond that in my (now at testing stage) > 3.6.z deployment. > > So I really hope you can push the numbers at least to 300 :) Storage domain is heavyweight, using one thread per storage domain and for file based storage, and 1 ioprocess thread. We also perform io every 10 seconds on each domain. With this design, 300 storage domain are not possible. > I think (from reading this BZ, please correct me if I'm wrong) > > we have 2 issues at hand: > > 1. ioprocess leaking 2 fds per SD (can this be stopped?) We don't have any fd leak, we just use too many of them for each ioprocess. Current ioprocess uses 8 fds per instance (in vdsm). This was decreased to 6 in upstream patches: - https://gerrit.ovirt.org/58612 - https://gerrit.ovirt.org/58613 The lowest number of descriptors we can have is 2: - read pipe - write pipe To get there, we need to rewrite ipprocess client to use an event loop instead of one thread per ioprocess. We have the needed infrastructure now in 4.0, so we can have this in 4.1 if we work on this. This will allow doubling of the number of storage domains without any change in the way storage domains and mounts are implemented. > 2. fd limit of 1024 of vdsm, which should maybe get increased (each > administrator can increase it himself, as a workaround, but I bet it get's > overwritten on every vdsm update?) This is already fixed in master and 3.6, the limit is now 4096: - https://gerrit.ovirt.org/58123 - https://gerrit.ovirt.org/58321 There is a third issue that is not described in this bug - we are using more then one ioprocess per storage domain. Currently we open 1 iorpocess for storage domain, and 1 for mountpoint, used when refreshing storage. The ioprocess for mount points are automatically closed after 60 seconds, since they are used once every 5 minutes during storage refresh, but this means we are creating 100 iprocesses for 50 storage domains, using 800 fds just for this. Even worse, we actually create one ioprocess per directory in /rhev/data-center/mnt, so if you have large amount of stale storage domains directories there (probably removed storage domains), we create additional usless ioprocesses. This is the root cause for this bug, and it should be fixed by increasing the number of open files. We are working on fixing this issue in this topic, creating one ioprocess per mountpoint: https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:oop-per-mount These changes are invasive, no chance to get them before 4.1. For file based storage domains, if we want to support bigger number of storage domains, we must separate storage domains from mounts, like we do in block storage. Then we can have 50-100 mounts checking accessibility of the mounts, and unlimited of number of storage domains, where each mount provide lot of storage domains. Can you explain why you need 300 storage domains? and which kind of storage - file? block? Please open RFE for this if you have a good use case, we may work on it for a future version. You may also start a thread about this on the users mailing list, there may be others that need this functionality, and we may get useful info.
Created attachment 1168368 [details] vdsm log
I have run creation of 300 SD on RHEVM 3.6.7.2-0.1.el6, up to 100 all run O.k then reached Storage pool limit : Failed to attach Storage Domain storage_scale_101 to Data Center Default. (User: admin@internal) VDSM command failed: Too many domains in Storage pool: () But, 100 SD where created successfully with non of the reported symptoms, thus bug is fixed. Further info : systemctl status libvirtd -l: ● libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/libvirtd.service.d └─unlimited-core.conf Active: active (running) since Thu 2016-06-09 10:42:00 UTC; 6 days ago Docs: man:libvirtd(8) http://libvirt.org Main PID: 1163 (libvirtd) CGroup: /system.slice/libvirtd.service └─1163 /usr/sbin/libvirtd --listen Jun 09 10:41:59 host12-rack04.scale.openstack.engineering.redhat.com systemd[1]: Starting Virtualization daemon... Jun 09 10:41:59 host12-rack04.scale.openstack.engineering.redhat.com libvirtd[1163]: libvirt version: 1.2.17, package: 13.el7_2.5 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2016-05-26-07:48:46, x86-020.build.eng.bos.redhat.com) Jun 09 10:41:59 host12-rack04.scale.openstack.engineering.redhat.com libvirtd[1163]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so not accessible Jun 09 10:42:00 host12-rack04.scale.openstack.engineering.redhat.com systemd[1]: Started Virtualization daemon. Jun 13 09:09:13 host12-rack04.scale.openstack.engineering.redhat.com libvirtd[1163]: Received unexpected event 3 Jun 13 09:09:52 host12-rack04.scale.openstack.engineering.redhat.com libvirtd[1163]: Received unexpected event 3 Jun 13 09:54:15 host12-rack04.scale.openstack.engineering.redhat.com libvirtd[1163]: Received unexpected event 3 Jun 13 09:54:16 host12-rack04.scale.openstack.engineering.redhat.com libvirtd[1163]: Received unexpected event 3 Jun 13 10:05:43 host12-rack04.scale.openstack.engineering.redhat.com libvirtd[1163]: Received unexpected event 3 systemctl status vdsmd -l: ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2016-06-09 10:42:06 UTC; 6 days ago Main PID: 2535 (vdsm) CGroup: /system.slice/vdsmd.service ├─ 358 /usr/libexec/ioprocess --read-pipe-fd 880 --write-pipe-fd 879 --max-threads 10 --max-queued-requests 10 ├─ 625 /usr/libexec/ioprocess --read-pipe-fd 1513 --write-pipe-fd 1512 --max-threads 10 --max-queued-requests 10 ├─ 876 /usr/libexec/ioprocess --read-pipe-fd 1583 --write-pipe-fd 1582 --max-threads 10 --max-queued-requests 10 ├─ 942 /usr/libexec/ioprocess --read-pipe-fd 281 --write-pipe-fd 280 --max-threads 10 --max-queued-requests 10 ├─ 1515 /usr/libexec/ioprocess --read-pipe-fd 1707 --write-pipe-fd 1706 --max-threads 10 --max-queued-requests 10 ├─ 1777 /usr/libexec/ioprocess --read-pipe-fd 297 --write-pipe-fd 296 --max-threads 10 --max-queued-requests 10 ├─ 1900 /usr/libexec/ioprocess --read-pipe-fd 685 --write-pipe-fd 684 --max-threads 10 --max-queued-requests 10 ├─ 2380 /usr/libexec/ioprocess --read-pipe-fd 1167 --write-pipe-fd 1166 --max-threads 10 --max-queued-requests 10 ├─ 2535 /usr/bin/python /usr/share/vdsm/vdsm ├─ 2704 /usr/libexec/ioprocess --read-pipe-fd 313 --write-pipe-fd 312 --max-threads 10 --max-queued-requests 10 ├─ 3357 /usr/libexec/ioprocess --read-pipe-fd 321 --write-pipe-fd 320 --max-threads 10 --max-queued-requests 10 ├─ 3483 /usr/libexec/ioprocess --read-pipe-fd 328 --write-pipe-fd 326 --max-threads 10 --max-queued-requests 10 ├─ 3539 /usr/libexec/ioprocess --read-pipe-fd 639 --write-pipe-fd 637 --max-threads 10 --max-queued-requests 10 ├─ 3543 /usr/libexec/ioprocess --read-pipe-fd 648 --write-pipe-fd 643 --max-threads 10 --max-queued-requests 10 ├─ 3554 /usr/libexec/ioprocess --read-pipe-fd 669 --write-pipe-fd 664 --max-threads 10 --max-queued-requests 10 ├─ 3563 /usr/libexec/ioprocess --read-pipe-fd 702 --write-pipe-fd 697 --max-threads 10 --max-queued-requests 10 ├─ 3571 /usr/libexec/ioprocess --read-pipe-fd 719 --write-pipe-fd 714 --max-threads 10 --max-queued-requests 10 ├─ 3579 /usr/libexec/ioprocess --read-pipe-fd 733 --write-pipe-fd 731 --max-threads 10 --max-queued-requests 10 ├─ 3587 /usr/libexec/ioprocess --read-pipe-fd 749 --write-pipe-fd 747 --max-threads 10 --max-queued-requests 10 ├─ 3596 /usr/libexec/ioprocess --read-pipe-fd 770 --write-pipe-fd 763 --max-threads 10 --max-queued-requests 10 ├─ 3604 /usr/libexec/ioprocess --read-pipe-fd 786 --write-pipe-fd 781 --max-threads 10 --max-queued-requests 10 ├─ 3612 /usr/libexec/ioprocess --read-pipe-fd 800 --write-pipe-fd 798 --max-threads 10 --max-queued-requests 10 ├─ 3620 /usr/libexec/ioprocess --read-pipe-fd 816 --write-pipe-fd 814 --max-threads 10 --max-queued-requests 10 ├─ 3628 /usr/libexec/ioprocess --read-pipe-fd 835 --write-pipe-fd 832 --max-threads 10 --max-queued-requests 10 ├─ 3636 /usr/libexec/ioprocess --read-pipe-fd 853 --write-pipe-fd 851 --max-threads 10 --max-queued-requests 10 ├─ 3645 /usr/libexec/ioprocess --read-pipe-fd 870 --write-pipe-fd 868 --max-threads 10 --max-queued-requests 10 ├─ 3655 /usr/libexec/ioprocess --read-pipe-fd 1048 --write-pipe-fd 1047 --max-threads 10 --max-queued-requests 10 ├─ 3661 /usr/libexec/ioprocess --read-pipe-fd 885 --write-pipe-fd 882 --max-threads 10 --max-queued-requests 10 ├─ 3668 /usr/libexec/ioprocess --read-pipe-fd 903 --write-pipe-fd 901 --max-threads 10 --max-queued-requests 10 ├─ 3672 /usr/libexec/ioprocess --read-pipe-fd 896 --write-pipe-fd 895 --max-threads 10 --max-queued-requests 10 ├─ 3681 /usr/libexec/ioprocess --read-pipe-fd 937 --write-pipe-fd 935 --max-threads 10 --max-queued-requests 10 ├─ 3688 /usr/libexec/ioprocess --read-pipe-fd 954 --write-pipe-fd 952 --max-threads 10 --max-queued-requests 10 ├─ 3695 /usr/libexec/ioprocess --read-pipe-fd 969 --write-pipe-fd 966 --max-threads 10 --max-queued-requests 10 ├─ 3702 /usr/libexec/ioprocess --read-pipe-fd 985 --write-pipe-fd 982 --max-threads 10 --max-queued-requests 10 ├─ 3792 /usr/libexec/ioprocess --read-pipe-fd 1357 --write-pipe-fd 1356 --max-threads 10 --max-queued-requests 10 ├─ 4091 /usr/libexec/ioprocess --read-pipe-fd 1007 --write-pipe-fd 1005 --max-threads 10 --max-queued-requests 10 ├─ 4096 /usr/libexec/ioprocess --read-pipe-fd 1021 --write-pipe-fd 1010 --max-threads 10 --max-queued-requests 10 ├─ 4105 /usr/libexec/ioprocess --read-pipe-fd 1032 --write-pipe-fd 1030 --max-threads 10 --max-queued-requests 10 ├─ 4115 /usr/libexec/ioprocess --read-pipe-fd 1058 --write-pipe-fd 1056 --max-threads 10 --max-queued-requests 10 ├─ 4123 /usr/libexec/ioprocess --read-pipe-fd 1069 --write-pipe-fd 1067 --max-threads 10 --max-queued-requests 10 ├─ 4132 /usr/libexec/ioprocess --read-pipe-fd 1085 --write-pipe-fd 1082 --max-threads 10 --max-queued-requests 10 ├─ 4140 /usr/libexec/ioprocess --read-pipe-fd 1118 --write-pipe-fd 1115 --max-threads 10 --max-queued-requests 10 ├─ 4148 /usr/libexec/ioprocess --read-pipe-fd 1131 --write-pipe-fd 1129 --max-threads 10 --max-queued-requests 10 ├─ 4156 /usr/libexec/ioprocess --read-pipe-fd 1149 --write-pipe-fd 1147 --max-threads 10 --max-queued-requests 10 ├─ 4164 /usr/libexec/ioprocess --read-pipe-fd 1165 --write-pipe-fd 1164 --max-threads 10 --max-queued-requests 10 ├─ 4172 /usr/libexec/ioprocess --read-pipe-fd 1198 --write-pipe-fd 1197 --max-threads 10 --max-queued-requests 10 ├─ 4180 /usr/libexec/ioprocess --read-pipe-fd 1217 --write-pipe-fd 1214 --max-threads 10 --max-queued-requests 10 ├─ 4188 /usr/libexec/ioprocess --read-pipe-fd 1254 --write-pipe-fd 1252 --max-threads 10 --max-queued-requests 10 ├─ 4196 /usr/libexec/ioprocess --read-pipe-fd 1273 --write-pipe-fd 1272 --max-threads 10 --max-queued-requests 10 ├─ 4204 /usr/libexec/ioprocess --read-pipe-fd 1291 --write-pipe-fd 1290 --max-threads 10 --max-queued-requests 10 ├─ 4212 /usr/libexec/ioprocess --read-pipe-fd 1306 --write-pipe-fd 1303 --max-threads 10 --max-queued-requests 10 ├─ 4220 /usr/libexec/ioprocess --read-pipe-fd 1337 --write-pipe-fd 1336 --max-threads 10 --max-queued-requests 10 ├─ 4228 /usr/libexec/ioprocess --read-pipe-fd 1372 --write-pipe-fd 1371 --max-threads 10 --max-queued-requests 10 ├─ 4236 /usr/libexec/ioprocess --read-pipe-fd 1387 --write-pipe-fd 1385 --max-threads 10 --max-queued-requests 10 ├─ 4245 /usr/libexec/ioprocess --read-pipe-fd 1421 --write-pipe-fd 1419 --max-threads 10 --max-queued-requests 10 ├─ 4253 /usr/libexec/ioprocess --read-pipe-fd 1440 --write-pipe-fd 1439 --max-threads 10 --max-queued-requests 10 ├─ 4261 /usr/libexec/ioprocess --read-pipe-fd 1455 --write-pipe-fd 1454 --max-threads 10 --max-queued-requests 10 ├─ 4269 /usr/libexec/ioprocess --read-pipe-fd 1468 --write-pipe-fd 1466 --max-threads 10 --max-queued-requests 10 ├─ 4277 /usr/libexec/ioprocess --read-pipe-fd 1488 --write-pipe-fd 1487 --max-threads 10 --max-queued-requests 10 ├─ 4285 /usr/libexec/ioprocess --read-pipe-fd 1504 --write-pipe-fd 1502 --max-threads 10 --max-queued-requests 10 ├─ 4295 /usr/libexec/ioprocess --read-pipe-fd 1522 --write-pipe-fd 1521 --max-threads 10 --max-queued-requests 10 ├─ 4303 /usr/libexec/ioprocess --read-pipe-fd 1534 --write-pipe-fd 1530 --max-threads 10 --max-queued-requests 10 ├─ 4311 /usr/libexec/ioprocess --read-pipe-fd 1550 --write-pipe-fd 1549 --max-threads 10 --max-queued-requests 10 ├─ 4319 /usr/libexec/ioprocess --read-pipe-fd 1572 --write-pipe-fd 1569 --max-threads 10 --max-queued-requests 10 ├─ 4325 /usr/libexec/ioprocess --read-pipe-fd 348 --write-pipe-fd 347 --max-threads 10 --max-queued-requests 10 ├─ 4333 /usr/libexec/ioprocess --read-pipe-fd 1593 --write-pipe-fd 1591 --max-threads 10 --max-queued-requests 10 ├─ 4340 /usr/libexec/ioprocess --read-pipe-fd 1612 --write-pipe-fd 1611 --max-threads 10 --max-queued-requests 10 ├─ 4348 /usr/libexec/ioprocess --read-pipe-fd 1625 --write-pipe-fd 1624 --max-threads 10 --max-queued-requests 10 ├─ 4355 /usr/libexec/ioprocess --read-pipe-fd 1646 --write-pipe-fd 1645 --max-threads 10 --max-queued-requests 10 ├─ 4362 /usr/libexec/ioprocess --read-pipe-fd 1667 --write-pipe-fd 1665 --max-threads 10 --max-queued-requests 10 ├─ 4369 /usr/libexec/ioprocess --read-pipe-fd 1692 --write-pipe-fd 1691 --max-threads 10 --max-queued-requests 10 ├─ 4409 /usr/libexec/ioprocess --read-pipe-fd 1363 --write-pipe-fd 1360 --max-threads 10 --max-queued-requests 10 ├─ 4448 /usr/libexec/ioprocess --read-pipe-fd 685 --write-pipe-fd 681 --max-threads 10 --max-queued-requests 10 ├─ 4700 /usr/libexec/ioprocess --read-pipe-fd 697 --write-pipe-fd 696 --max-threads 10 --max-queued-requests 10 ├─ 4840 /usr/libexec/ioprocess --read-pipe-fd 1443 --write-pipe-fd 1442 --max-threads 10 --max-queued-requests 10 ├─ 5205 /usr/libexec/ioprocess --read-pipe-fd 362 --write-pipe-fd 361 --max-threads 10 --max-queued-requests 10 ├─ 5787 /usr/libexec/ioprocess --read-pipe-fd 1654 --write-pipe-fd 1653 --max-threads 10 --max-queued-requests 10 ├─ 6206 /usr/libexec/ioprocess --read-pipe-fd 379 --write-pipe-fd 378 --max-threads 10 --max-queued-requests 10 ├─ 6436 /usr/libexec/ioprocess --read-pipe-fd 1280 --write-pipe-fd 1279 --max-threads 10 --max-queued-requests 10 ├─ 6821 /usr/libexec/ioprocess --read-pipe-fd 714 --write-pipe-fd 713 --max-threads 10 --max-queued-requests 10 ├─ 6935 /usr/libexec/ioprocess --read-pipe-fd 913 --write-pipe-fd 912 --max-threads 10 --max-queued-requests 10 ├─ 7024 /usr/libexec/ioprocess --read-pipe-fd 385 --write-pipe-fd 382 --max-threads 10 --max-queued-requests 10 ├─ 7058 /usr/libexec/ioprocess --read-pipe-fd 1181 --write-pipe-fd 1180 --max-threads 10 --max-queued-requests 10 ├─ 7145 /usr/libexec/ioprocess --read-pipe-fd 395 --write-pipe-fd 394 --max-threads 10 --max-queued-requests 10 ├─ 7610 /usr/libexec/ioprocess --read-pipe-fd 1679 --write-pipe-fd 1677 --max-threads 10 --max-queued-requests 10 ├─ 8019 /usr/libexec/ioprocess --read-pipe-fd 1060 --write-pipe-fd 1057 --max-threads 10 --max-queued-requests 10 ├─ 8070 /usr/libexec/ioprocess --read-pipe-fd 1733 --write-pipe-fd 1732 --max-threads 10 --max-queued-requests 10 ├─ 8156 /usr/libexec/ioprocess --read-pipe-fd 409 --write-pipe-fd 406 --max-threads 10 --max-queued-requests 10 ├─ 8619 /usr/libexec/ioprocess --read-pipe-fd 1538 --write-pipe-fd 1537 --max-threads 10 --max-queued-requests 10 ├─ 9041 /usr/libexec/ioprocess --read-pipe-fd 731 --write-pipe-fd 730 --max-threads 10 --max-queued-requests 10 ├─ 9186 /usr/libexec/ioprocess --read-pipe-fd 430 --write-pipe-fd 429 --max-threads 10 --max-queued-requests 10 ├─10007 /usr/libexec/ioprocess --read-pipe-fd 922 --write-pipe-fd 921 --max-threads 10 --max-queued-requests 10 ├─10302 /usr/libexec/ioprocess --read-pipe-fd 930 --write-pipe-fd 929 --max-threads 10 --max-queued-requests 10 ├─10452 /usr/libexec/ioprocess --read-pipe-fd 1379 --write-pipe-fd 1376 --max-threads 10 --max-queued-requests 10 ├─10531 /usr/libexec/ioprocess --read-pipe-fd 448 --write-pipe-fd 447 --max-threads 10 --max-queued-requests 10 ├─10551 /usr/libexec/ioprocess --read-pipe-fd 1604 --write-pipe-fd 1603 --max-threads 10 --max-queued-requests 10 ├─11237 /usr/libexec/ioprocess --read-pipe-fd 1465 --write-pipe-fd 1464 --max-threads 10 --max-queued-requests 10 ├─11322 /usr/libexec/ioprocess --read-pipe-fd 746 --write-pipe-fd 743 --max-threads 10 --max-queued-requests 10 ├─11441 /usr/libexec/ioprocess --read-pipe-fd 1188 --write-pipe-fd 1186 --max-threads 10 --max-queued-requests 10 ├─11746 /usr/libexec/ioprocess --read-pipe-fd 1198 --write-pipe-fd 1197 --max-threads 10 --max-queued-requests 10 ├─11775 /usr/libexec/ioprocess --read-pipe-fd 464 --write-pipe-fd 463 --max-threads 10 --max-queued-requests 10 ├─12038 /usr/libexec/ioprocess --read-pipe-fd 1294 --write-pipe-fd 1292 --max-threads 10 --max-queued-requests 10 ├─12215 /usr/libexec/ioprocess --read-pipe-fd 1080 --write-pipe-fd 1079 --max-threads 10 --max-queued-requests 10 ├─12783 /usr/libexec/ioprocess --read-pipe-fd 1724 --write-pipe-fd 1721 --max-threads 10 --max-queued-requests 10 ├─12977 /usr/libexec/ioprocess --read-pipe-fd 480 --write-pipe-fd 479 --max-threads 10 --max-queued-requests 10 ├─13567 /usr/libexec/ioprocess --read-pipe-fd 947 --write-pipe-fd 946 --max-threads 10 --max-queued-requests 10 ├─13678 /usr/libexec/ioprocess --read-pipe-fd 763 --write-pipe-fd 762 --max-threads 10 --max-queued-requests 10 ├─14375 /usr/libexec/ioprocess --read-pipe-fd 496 --write-pipe-fd 495 --max-threads 10 --max-queued-requests 10 ├─15078 /usr/libexec/ioprocess --read-pipe-fd 1672 --write-pipe-fd 1671 --max-threads 10 --max-queued-requests 10 ├─15695 /usr/libexec/ioprocess --read-pipe-fd 513 --write-pipe-fd 512 --max-threads 10 --max-queued-requests 10 ├─16179 /usr/libexec/ioprocess --read-pipe-fd 781 --write-pipe-fd 780 --max-threads 10 --max-queued-requests 10 ├─16235 /usr/libexec/ioprocess --read-pipe-fd 1097 --write-pipe-fd 1096 --max-threads 10 --max-queued-requests 10 ├─16564 /usr/libexec/ioprocess --read-pipe-fd 1554 --write-pipe-fd 1553 --max-threads 10 --max-queued-requests 10 ├─16617 /usr/libexec/ioprocess --read-pipe-fd 1396 --write-pipe-fd 1395 --max-threads 10 --max-queued-requests 10 ├─16831 /usr/libexec/ioprocess --read-pipe-fd 1212 --write-pipe-fd 1211 --max-threads 10 --max-queued-requests 10 ├─16897 /usr/libexec/ioprocess --read-pipe-fd 964 --write-pipe-fd 963 --max-threads 10 --max-queued-requests 10 ├─16914 /usr/libexec/ioprocess --read-pipe-fd 523 --write-pipe-fd 521 --max-threads 10 --max-queued-requests 10 ├─17100 /usr/libexec/ioprocess --read-pipe-fd 528 --write-pipe-fd 525 --max-threads 10 --max-queued-requests 10 ├─17903 /usr/libexec/ioprocess --read-pipe-fd 1313 --write-pipe-fd 1312 --max-threads 10 --max-queued-requests 10 ├─18519 /usr/libexec/ioprocess --read-pipe-fd 545 --write-pipe-fd 544 --max-threads 10 --max-queued-requests 10 ├─18609 /usr/libexec/ioprocess --read-pipe-fd 1479 --write-pipe-fd 1476 --max-threads 10 --max-queued-requests 10 ├─18703 /usr/libexec/ioprocess --read-pipe-fd 798 --write-pipe-fd 797 --max-threads 10 --max-queued-requests 10 ├─18842 /usr/libexec/ioprocess --read-pipe-fd 1625 --write-pipe-fd 1624 --max-threads 10 --max-queued-requests 10 ├─20293 /usr/libexec/ioprocess --read-pipe-fd 567 --write-pipe-fd 566 --max-threads 10 --max-queued-requests 10 ├─20305 /usr/libexec/ioprocess --read-pipe-fd 1105 --write-pipe-fd 1103 --max-threads 10 --max-queued-requests 10 ├─20374 /usr/libexec/ioprocess --read-pipe-fd 980 --write-pipe-fd 979 --max-threads 10 --max-queued-requests 10 ├─20640 /usr/libexec/ioprocess --read-pipe-fd 1113 --write-pipe-fd 1112 --max-threads 10 --max-queued-requests 10 ├─21159 /usr/libexec/ioprocess --read-pipe-fd 814 --write-pipe-fd 813 --max-threads 10 --max-queued-requests 10 ├─21709 /usr/libexec/ioprocess --read-pipe-fd 1228 --write-pipe-fd 1227 --max-threads 10 --max-queued-requests 10 ├─21890 /usr/libexec/ioprocess --read-pipe-fd 579 --write-pipe-fd 578 --max-threads 10 --max-queued-requests 10 ├─22310 /usr/libexec/ioprocess --read-pipe-fd 40 --write-pipe-fd 39 --max-threads 10 --max-queued-requests 10 ├─22319 /usr/libexec/ioprocess --read-pipe-fd 60 --write-pipe-fd 58 --max-threads 10 --max-queued-requests 10 ├─22335 /usr/libexec/ioprocess --read-pipe-fd 68 --write-pipe-fd 67 --max-threads 10 --max-queued-requests 10 ├─22353 /usr/libexec/ioprocess --read-pipe-fd 77 --write-pipe-fd 76 --max-threads 10 --max-queued-requests 10 ├─22363 /usr/libexec/ioprocess --read-pipe-fd 83 --write-pipe-fd 79 --max-threads 10 --max-queued-requests 10 ├─22374 /usr/libexec/ioprocess --read-pipe-fd 98 --write-pipe-fd 97 --max-threads 10 --max-queued-requests 10 ├─22384 /usr/libexec/ioprocess --read-pipe-fd 112 --write-pipe-fd 111 --max-threads 10 --max-queued-requests 10 ├─22395 /usr/libexec/ioprocess --read-pipe-fd 129 --write-pipe-fd 127 --max-threads 10 --max-queued-requests 10 ├─22406 /usr/libexec/ioprocess --read-pipe-fd 150 --write-pipe-fd 148 --max-threads 10 --max-queued-requests 10 ├─22421 /usr/libexec/ioprocess --read-pipe-fd 166 --write-pipe-fd 164 --max-threads 10 --max-queued-requests 10 ├─22433 /usr/libexec/ioprocess --read-pipe-fd 183 --write-pipe-fd 181 --max-threads 10 --max-queued-requests 10 ├─22445 /usr/libexec/ioprocess --read-pipe-fd 198 --write-pipe-fd 197 --max-threads 10 --max-queued-requests 10 ├─22456 /usr/libexec/ioprocess --read-pipe-fd 213 --write-pipe-fd 211 --max-threads 10 --max-queued-requests 10 ├─22466 /usr/libexec/ioprocess --read-pipe-fd 229 --write-pipe-fd 228 --max-threads 10 --max-queued-requests 10 ├─22478 /usr/libexec/ioprocess --read-pipe-fd 244 --write-pipe-fd 243 --max-threads 10 --max-queued-requests 10 ├─22492 /usr/libexec/ioprocess --read-pipe-fd 263 --write-pipe-fd 261 --max-threads 10 --max-queued-requests 10 ├─22502 /usr/libexec/ioprocess --read-pipe-fd 279 --write-pipe-fd 277 --max-threads 10 --max-queued-requests 10 ├─22518 /usr/libexec/ioprocess --read-pipe-fd 295 --write-pipe-fd 293 --max-threads 10 --max-queued-requests 10 ├─22528 /usr/libexec/ioprocess --read-pipe-fd 311 --write-pipe-fd 309 --max-threads 10 --max-queued-requests 10 ├─22541 /usr/libexec/ioprocess --read-pipe-fd 343 --write-pipe-fd 342 --max-threads 10 --max-queued-requests 10 ├─22553 /usr/libexec/ioprocess --read-pipe-fd 360 --write-pipe-fd 358 --max-threads 10 --max-queued-requests 10 ├─22564 /usr/libexec/ioprocess --read-pipe-fd 393 --write-pipe-fd 376 --max-threads 10 --max-queued-requests 10 ├─22574 /usr/libexec/ioprocess --read-pipe-fd 410 --write-pipe-fd 409 --max-threads 10 --max-queued-requests 10 ├─22587 /usr/libexec/ioprocess --read-pipe-fd 435 --write-pipe-fd 424 --max-threads 10 --max-queued-requests 10 ├─22598 /usr/libexec/ioprocess --read-pipe-fd 450 --write-pipe-fd 448 --max-threads 10 --max-queued-requests 10 ├─22609 /usr/libexec/ioprocess --read-pipe-fd 466 --write-pipe-fd 464 --max-threads 10 --max-queued-requests 10 ├─22625 /usr/libexec/ioprocess --read-pipe-fd 482 --write-pipe-fd 480 --max-threads 10 --max-queued-requests 10 ├─22631 /usr/libexec/ioprocess --read-pipe-fd 501 --write-pipe-fd 496 --max-threads 10 --max-queued-requests 10 ├─22644 /usr/libexec/ioprocess --read-pipe-fd 531 --write-pipe-fd 528 --max-threads 10 --max-queued-requests 10 ├─22653 /usr/libexec/ioprocess --read-pipe-fd 552 --write-pipe-fd 550 --max-threads 10 --max-queued-requests 10 ├─22665 /usr/libexec/ioprocess --read-pipe-fd 562 --write-pipe-fd 561 --max-threads 10 --max-queued-requests 10 ├─22674 /usr/libexec/ioprocess --read-pipe-fd 584 --write-pipe-fd 581 --max-threads 10 --max-queued-requests 10 ├─22686 /usr/libexec/ioprocess --read-pipe-fd 598 --write-pipe-fd 596 --max-threads 10 --max-queued-requests 10 ├─22697 /usr/libexec/ioprocess --read-pipe-fd 615 --write-pipe-fd 613 --max-threads 10 --max-queued-requests 10 ├─22951 /usr/libexec/ioprocess --read-pipe-fd 1406 --write-pipe-fd 1405 --max-threads 10 --max-queued-requests 10 ├─23231 /usr/libexec/ioprocess --read-pipe-fd 1321 --write-pipe-fd 1320 --max-threads 10 --max-queued-requests 10 ├─23320 /usr/libexec/ioprocess --read-pipe-fd 1414 --write-pipe-fd 1413 --max-threads 10 --max-queued-requests 10 ├─23590 /usr/libexec/ioprocess --read-pipe-fd 1329 --write-pipe-fd 1328 --max-threads 10 --max-queued-requests 10 ├─23773 /usr/libexec/ioprocess --read-pipe-fd 597 --write-pipe-fd 596 --max-threads 10 --max-queued-requests 10 ├─23955 /usr/libexec/ioprocess --read-pipe-fd 830 --write-pipe-fd 829 --max-threads 10 --max-queued-requests 10 ├─24209 /usr/libexec/ioprocess --read-pipe-fd 996 --write-pipe-fd 994 --max-threads 10 --max-queued-requests 10 ├─24456 /usr/libexec/ioprocess --read-pipe-fd 1691 --write-pipe-fd 1690 --max-threads 10 --max-queued-requests 10 ├─25186 /usr/libexec/ioprocess --read-pipe-fd 1131 --write-pipe-fd 1130 --max-threads 10 --max-queued-requests 10 ├─25193 /usr/libexec/ioprocess --read-pipe-fd 1566 --write-pipe-fd 1565 --max-threads 10 --max-queued-requests 10 ├─25353 /usr/libexec/ioprocess --read-pipe-fd 1497 --write-pipe-fd 1496 --max-threads 10 --max-queued-requests 10 ├─25595 /usr/libexec/ioprocess --read-pipe-fd 613 --write-pipe-fd 612 --max-threads 10 --max-queued-requests 10 ├─26564 /usr/libexec/ioprocess --read-pipe-fd 89 --write-pipe-fd 87 --max-threads 10 --max-queued-requests 10 ├─26594 /usr/libexec/ioprocess --read-pipe-fd 1238 --write-pipe-fd 1236 --max-threads 10 --max-queued-requests 10 ├─27001 /usr/libexec/ioprocess --read-pipe-fd 846 --write-pipe-fd 845 --max-threads 10 --max-queued-requests 10 ├─27204 /usr/libexec/ioprocess --read-pipe-fd 115 --write-pipe-fd 114 --max-threads 10 --max-queued-requests 10 ├─27322 /usr/libexec/ioprocess --read-pipe-fd 1245 --write-pipe-fd 1244 --max-threads 10 --max-queued-requests 10 ├─27675 /usr/libexec/ioprocess --read-pipe-fd 131 --write-pipe-fd 130 --max-threads 10 --max-queued-requests 10 ├─27831 /usr/libexec/ioprocess --read-pipe-fd 50 --write-pipe-fd 49 --max-threads 10 --max-queued-requests 10 ├─27952 /usr/libexec/ioprocess --read-pipe-fd 632 --write-pipe-fd 631 --max-threads 10 --max-queued-requests 10 ├─28145 /usr/libexec/ioprocess --read-pipe-fd 144 --write-pipe-fd 140 --max-threads 10 --max-queued-requests 10 ├─28312 /usr/libexec/ioprocess --read-pipe-fd 1014 --write-pipe-fd 1013 --max-threads 10 --max-queued-requests 10 ├─28348 /usr/libexec/ioprocess --read-pipe-fd 1638 --write-pipe-fd 1637 --max-threads 10 --max-queued-requests 10 ├─28606 /usr/libexec/ioprocess --read-pipe-fd 160 --write-pipe-fd 156 --max-threads 10 --max-queued-requests 10 ├─29218 /usr/libexec/ioprocess --read-pipe-fd 177 --write-pipe-fd 173 --max-threads 10 --max-queued-requests 10 ├─29776 /usr/libexec/ioprocess --read-pipe-fd 196 --write-pipe-fd 194 --max-threads 10 --max-queued-requests 10 ├─29943 /usr/libexec/ioprocess --read-pipe-fd 863 --write-pipe-fd 862 --max-threads 10 --max-queued-requests 10 ├─30023 /usr/libexec/ioprocess --read-pipe-fd 648 --write-pipe-fd 647 --max-threads 10 --max-queued-requests 10 ├─30054 /usr/libexec/ioprocess --read-pipe-fd 1432 --write-pipe-fd 1431 --max-threads 10 --max-queued-requests 10 ├─30080 /usr/libexec/ioprocess --read-pipe-fd 1147 --write-pipe-fd 1146 --max-threads 10 --max-queued-requests 10 ├─30556 /usr/libexec/ioprocess --read-pipe-fd 1347 --write-pipe-fd 1345 --max-threads 10 --max-queued-requests 10 ├─30647 /usr/libexec/ioprocess --read-pipe-fd 215 --write-pipe-fd 214 --max-threads 10 --max-queued-requests 10 ├─31237 /usr/libexec/ioprocess --read-pipe-fd 231 --write-pipe-fd 230 --max-threads 10 --max-queued-requests 10 ├─31836 /usr/libexec/ioprocess --read-pipe-fd 249 --write-pipe-fd 247 --max-threads 10 --max-queued-requests 10 ├─31909 /usr/libexec/ioprocess --read-pipe-fd 664 --write-pipe-fd 663 --max-threads 10 --max-queued-requests 10 ├─32075 /usr/libexec/ioprocess --read-pipe-fd 1032 --write-pipe-fd 1031 --max-threads 10 --max-queued-requests 10 ├─32568 /usr/libexec/ioprocess --read-pipe-fd 1261 --write-pipe-fd 1260 --max-threads 10 --max-queued-requests 10 └─32625 /usr/libexec/ioprocess --read-pipe-fd 265 --write-pipe-fd 264 --max-threads 10 --max-queued-requests 10 Jun 09 10:42:08 host12-rack04.scale.openstack.engineering.redhat.com python[2535]: DIGEST-MD5 ask_user_info() Jun 09 10:42:08 host12-rack04.scale.openstack.engineering.redhat.com python[2535]: DIGEST-MD5 make_client_response() Jun 09 10:42:08 host12-rack04.scale.openstack.engineering.redhat.com python[2535]: DIGEST-MD5 client step 3 Jun 13 09:37:39 host12-rack04.scale.openstack.engineering.redhat.com vdsm[2535]: vdsm root WARNING File: /rhev/data-center/00000001-0001-0001-0001-000000000295/8fd5280c-d61a-4e34-a304-9a687be888f6/images/d9c53eea-ef13-4d36-ac88-b07b3974d5b3/45d35e4e-203b-4e1f-9130-0b79ade080d7 already removed Jun 13 09:37:40 host12-rack04.scale.openstack.engineering.redhat.com vdsm[2535]: vdsm root WARNING File: /rhev/data-center/00000001-0001-0001-0001-000000000295/8fd5280c-d61a-4e34-a304-9a687be888f6/images/62d12416-9bdc-48cc-b8e8-5957283b32b8/ba70b71c-003c-4637-84f3-b6c096674564 already removed Jun 14 10:12:02 host12-rack04.scale.openstack.engineering.redhat.com vdsm[2535]: vdsm vds.dispatcher ERROR SSL error during reading data: unexpected eof Jun 14 10:12:02 host12-rack04.scale.openstack.engineering.redhat.com vdsm[2535]: vdsm vds.dispatcher ERROR SSL error during reading data: unexpected eof Jun 14 10:14:02 host12-rack04.scale.openstack.engineering.redhat.com su[8630]: (to vdsm) root on none Jun 15 07:56:17 host12-rack04.scale.openstack.engineering.redhat.com vdsm[2535]: vdsm IOProcess WARNING (50357) Request queue full Jun 15 07:56:17 host12-rack04.scale.openstack.engineering.redhat.com vdsm[2535]: vdsm IOProcess WARNING (50358) Request queue full