Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1339245

Summary: Creation of 300 storage domains, failed during creation of 54th storage domain.
Product: [oVirt] vdsm Reporter: Nikolai Sednev <nsednev>
Component: CoreAssignee: Fred Rolland <frolland>
Status: CLOSED CURRENTRELEASE QA Contact: guy chen <guchen>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.17.30CC: amureini, bugs, frolland, lbopf, nsednev, nsoffer, s.kieske, stirabos, tnisan, ylavi
Target Milestone: ovirt-3.6.7Flags: amureini: ovirt-3.6.z?
rule-engine: ovirt-4.0.0+
nsednev: testing_plan_complete?
ylavi: planning_ack+
rule-engine: devel_ack+
rule-engine: testing_ack+
Target Release: 4.17.30   
Hardware: x86_64   
OS: Linux   
URL: https://drive.google.com/a/redhat.com/file/d/0B85BEaDBcF88bFpYOGItN2lOcUU/view?usp=sharing
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-07-04 12:31:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
sosreport from engine
none
lsof output
none
vdsm log none

Description Nikolai Sednev 2016-05-24 13:09:40 UTC 
Description of problem:
In order to verify the https://bugzilla.redhat.com/show_bug.cgi?id=1317468, I've tried to create 300 SDs.
Creation of 300 storage domains, failed during creation of 54th storage domain.

VDSM alma04.qa.lab.tlv.redhat.com command failed: Cannot reconstruct master domain: (u"spUUID=00000001-0001-0001-0001-00000000011d, masterDom=4bf872c6-c931-4203-aac4-436f19d9e683, masterVersion=3, clusterlock params: ({'LEASETIMESEC': 60, 'IOOPTIMEOUTSEC': 10, 'LEASERETRIES': 3, 'LOCKRENEWALINTERVALSEC': 5})",)

systemctl status libvirtd -l
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/libvirtd.service.d
           └─unlimited-core.conf
   Active: active (running) since Tue 2016-05-24 08:24:38 IDT; 7h ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 1605 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           └─1605 /usr/sbin/libvirtd --listen

May 24 08:24:36 alma04.qa.lab.tlv.redhat.com systemd[1]: Starting Virtualization daemon...
May 24 08:24:37 alma04.qa.lab.tlv.redhat.com libvirtd[1605]: libvirt version: 1.2.17, package: 13.el7_2.4 (CentOS BuildSystem <http://bugs.centos.org>, 2016-03-31-16:56:26, worker1.bsys.centos.org)
May 24 08:24:37 alma04.qa.lab.tlv.redhat.com libvirtd[1605]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so not accessible
May 24 08:24:38 alma04.qa.lab.tlv.redhat.com systemd[1]: Started Virtualization daemon.
May 24 08:34:16 alma04.qa.lab.tlv.redhat.com libvirtd[1605]: Failed to acquire lock: File exists
May 24 15:53:08 alma04.qa.lab.tlv.redhat.com libvirtd[1605]: End of file while reading data: Input/output error


systemctl status vdsmd -l
● vdsmd.service - Virtual Desktop Server Manager
   Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2016-05-24 08:24:41 IDT; 7h ago
  Process: 2840 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS)
 Main PID: 2913 (vdsm)
   CGroup: /system.slice/vdsmd.service
           ├─  1043 /usr/libexec/ioprocess --read-pipe-fd 666 --write-pipe-fd 663 --max-threads 10 --max-queued-requests 10
           ├─  2288 /usr/libexec/ioprocess --read-pipe-fd 674 --write-pipe-fd 672 --max-threads 10 --max-queued-requests 10
           ├─  2876 /usr/libexec/ioprocess --read-pipe-fd 683 --write-pipe-fd 682 --max-threads 10 --max-queued-requests 10
           ├─  2913 /usr/bin/python /usr/share/vdsm/vdsm
           ├─  3125 /usr/libexec/ioprocess --read-pipe-fd 71 --write-pipe-fd 70 --max-threads 10 --max-queued-requests 10
           ├─  3826 /usr/libexec/ioprocess --read-pipe-fd 690 --write-pipe-fd 688 --max-threads 10 --max-queued-requests 10
           ├─  4827 /usr/libexec/ioprocess --read-pipe-fd 699 --write-pipe-fd 698 --max-threads 10 --max-queued-requests 10
           ├─  5897 /usr/libexec/ioprocess --read-pipe-fd 712 --write-pipe-fd 711 --max-threads 10 --max-queued-requests 10
           ├─  6853 /usr/libexec/ioprocess --read-pipe-fd 719 --write-pipe-fd 717 --max-threads 10 --max-queued-requests 10
           ├─  7857 /usr/libexec/ioprocess --read-pipe-fd 729 --write-pipe-fd 728 --max-threads 10 --max-queued-requests 10
           ├─  8755 /usr/libexec/ioprocess --read-pipe-fd 736 --write-pipe-fd 734 --max-threads 10 --max-queued-requests 10
           ├─  9831 /usr/libexec/ioprocess --read-pipe-fd 746 --write-pipe-fd 745 --max-threads 10 --max-queued-requests 10
           ├─ 10864 /usr/libexec/ioprocess --read-pipe-fd 754 --write-pipe-fd 753 --max-threads 10 --max-queued-requests 10
           ├─ 12212 /usr/libexec/ioprocess --read-pipe-fd 763 --write-pipe-fd 762 --max-threads 10 --max-queued-requests 10
           ├─ 13006 /usr/libexec/ioprocess --read-pipe-fd 771 --write-pipe-fd 770 --max-threads 10 --max-queued-requests 10
           ├─ 14050 /usr/libexec/ioprocess --read-pipe-fd 782 --write-pipe-fd 781 --max-threads 10 --max-queued-requests 10
           ├─ 15370 /usr/libexec/ioprocess --read-pipe-fd 784 --write-pipe-fd 782 --max-threads 10 --max-queued-requests 10
           ├─ 16660 /usr/libexec/ioprocess --read-pipe-fd 792 --write-pipe-fd 789 --max-threads 10 --max-queued-requests 10
           ├─ 17661 /usr/libexec/ioprocess --read-pipe-fd 803 --write-pipe-fd 802 --max-threads 10 --max-queued-requests 10
           ├─ 19145 /usr/libexec/ioprocess --read-pipe-fd 811 --write-pipe-fd 810 --max-threads 10 --max-queued-requests 10
           ├─ 20430 /usr/libexec/ioprocess --read-pipe-fd 819 --write-pipe-fd 818 --max-threads 10 --max-queued-requests 10
           ├─ 21603 /usr/libexec/ioprocess --read-pipe-fd 828 --write-pipe-fd 827 --max-threads 10 --max-queued-requests 10
           ├─ 22673 /usr/libexec/ioprocess --read-pipe-fd 837 --write-pipe-fd 836 --max-threads 10 --max-queued-requests 10
           ├─ 23788 /usr/libexec/ioprocess --read-pipe-fd 846 --write-pipe-fd 845 --max-threads 10 --max-queued-requests 10
           ├─ 25266 /usr/libexec/ioprocess --read-pipe-fd 856 --write-pipe-fd 855 --max-threads 10 --max-queued-requests 10
           ├─ 26449 /usr/libexec/ioprocess --read-pipe-fd 866 --write-pipe-fd 865 --max-threads 10 --max-queued-requests 10
           ├─ 27866 /usr/libexec/ioprocess --read-pipe-fd 875 --write-pipe-fd 874 --max-threads 10 --max-queued-requests 10
           ├─ 29236 /usr/libexec/ioprocess --read-pipe-fd 884 --write-pipe-fd 883 --max-threads 10 --max-queued-requests 10
           ├─ 30486 /usr/libexec/ioprocess --read-pipe-fd 892 --write-pipe-fd 891 --max-threads 10 --max-queued-requests 10
           ├─ 31840 /usr/libexec/ioprocess --read-pipe-fd 900 --write-pipe-fd 899 --max-threads 10 --max-queued-requests 10
           ├─ 33230 /usr/libexec/ioprocess --read-pipe-fd 908 --write-pipe-fd 907 --max-threads 10 --max-queued-requests 10
           ├─ 34428 /usr/libexec/ioprocess --read-pipe-fd 916 --write-pipe-fd 915 --max-threads 10 --max-queued-requests 10
           ├─ 36039 /usr/libexec/ioprocess --read-pipe-fd 924 --write-pipe-fd 923 --max-threads 10 --max-queued-requests 10
           ├─ 37356 /usr/libexec/ioprocess --read-pipe-fd 932 --write-pipe-fd 931 --max-threads 10 --max-queued-requests 10
           ├─ 38783 /usr/libexec/ioprocess --read-pipe-fd 941 --write-pipe-fd 940 --max-threads 10 --max-queued-requests 10
           ├─ 40028 /usr/libexec/ioprocess --read-pipe-fd 950 --write-pipe-fd 949 --max-threads 10 --max-queued-requests 10
           ├─ 41754 /usr/libexec/ioprocess --read-pipe-fd 958 --write-pipe-fd 957 --max-threads 10 --max-queued-requests 10
           ├─ 43305 /usr/libexec/ioprocess --read-pipe-fd 969 --write-pipe-fd 968 --max-threads 10 --max-queued-requests 10
           ├─ 44681 /usr/libexec/ioprocess --read-pipe-fd 971 --write-pipe-fd 969 --max-threads 10 --max-queued-requests 10
           ├─ 46296 /usr/libexec/ioprocess --read-pipe-fd 983 --write-pipe-fd 982 --max-threads 10 --max-queued-requests 10
           ├─ 47736 /usr/libexec/ioprocess --read-pipe-fd 988 --write-pipe-fd 985 --max-threads 10 --max-queued-requests 10
           ├─ 49326 /usr/libexec/ioprocess --read-pipe-fd 999 --write-pipe-fd 998 --max-threads 10 --max-queued-requests 10
           ├─ 51123 /usr/libexec/ioprocess --read-pipe-fd 1006 --write-pipe-fd 1004 --max-threads 10 --max-queued-requests 10
           ├─ 53969 /usr/libexec/ioprocess --read-pipe-fd 490 --write-pipe-fd 242 --max-threads 10 --max-queued-requests 10
           ├─ 65118 /usr/libexec/ioprocess --read-pipe-fd 93 --write-pipe-fd 92 --max-threads 10 --max-queued-requests 10
           ├─ 92547 /usr/libexec/ioprocess --read-pipe-fd 40 --write-pipe-fd 39 --max-threads 10 --max-queued-requests 10
           ├─ 92573 /usr/libexec/ioprocess --read-pipe-fd 48 --write-pipe-fd 47 --max-threads 10 --max-queued-requests 10
           ├─ 92580 /usr/libexec/ioprocess --read-pipe-fd 55 --write-pipe-fd 53 --max-threads 10 --max-queued-requests 10
           ├─ 92588 /usr/libexec/ioprocess --read-pipe-fd 65 --write-pipe-fd 63 --max-threads 10 --max-queued-requests 10
           ├─ 92595 /usr/libexec/ioprocess --read-pipe-fd 85 --write-pipe-fd 83 --max-threads 10 --max-queued-requests 10
           ├─ 92602 /usr/libexec/ioprocess --read-pipe-fd 109 --write-pipe-fd 108 --max-threads 10 --max-queued-requests 10
           ├─ 93544 /usr/libexec/ioprocess --read-pipe-fd 125 --write-pipe-fd 124 --max-threads 10 --max-queued-requests 10
           ├─ 93597 /usr/libexec/ioprocess --read-pipe-fd 128 --write-pipe-fd 126 --max-threads 10 --max-queued-requests 10
           ├─ 93870 /usr/libexec/ioprocess --read-pipe-fd 140 --write-pipe-fd 139 --max-threads 10 --max-queued-requests 10
           ├─ 93944 /usr/libexec/ioprocess --read-pipe-fd 148 --write-pipe-fd 147 --max-threads 10 --max-queued-requests 10
           ├─ 94217 /usr/libexec/ioprocess --read-pipe-fd 161 --write-pipe-fd 160 --max-threads 10 --max-queued-requests 10
           ├─ 94298 /usr/libexec/ioprocess --read-pipe-fd 161 --write-pipe-fd 157 --max-threads 10 --max-queued-requests 10
           ├─ 94611 /usr/libexec/ioprocess --read-pipe-fd 172 --write-pipe-fd 170 --max-threads 10 --max-queued-requests 10
           ├─ 94684 /usr/libexec/ioprocess --read-pipe-fd 181 --write-pipe-fd 180 --max-threads 10 --max-queued-requests 10
           ├─ 94960 /usr/libexec/ioprocess --read-pipe-fd 189 --write-pipe-fd 188 --max-threads 10 --max-queued-requests 10
           ├─ 95100 /usr/libexec/ioprocess --read-pipe-fd 198 --write-pipe-fd 197 --max-threads 10 --max-queued-requests 10
           ├─ 95379 /usr/libexec/ioprocess --read-pipe-fd 205 --write-pipe-fd 203 --max-threads 10 --max-queued-requests 10
           ├─ 95499 /usr/libexec/ioprocess --read-pipe-fd 214 --write-pipe-fd 213 --max-threads 10 --max-queued-requests 10
           ├─ 95837 /usr/libexec/ioprocess --read-pipe-fd 221 --write-pipe-fd 219 --max-threads 10 --max-queued-requests 10
           ├─ 95974 /usr/libexec/ioprocess --read-pipe-fd 230 --write-pipe-fd 229 --max-threads 10 --max-queued-requests 10
           ├─ 96295 /usr/libexec/ioprocess --read-pipe-fd 235 --write-pipe-fd 232 --max-threads 10 --max-queued-requests 10
           ├─ 96491 /usr/libexec/ioprocess --read-pipe-fd 248 --write-pipe-fd 247 --max-threads 10 --max-queued-requests 10
           ├─ 96861 /usr/libexec/ioprocess --read-pipe-fd 258 --write-pipe-fd 257 --max-threads 10 --max-queued-requests 10
           ├─ 97058 /usr/libexec/ioprocess --read-pipe-fd 265 --write-pipe-fd 263 --max-threads 10 --max-queued-requests 10
           ├─ 97435 /usr/libexec/ioprocess --read-pipe-fd 273 --write-pipe-fd 272 --max-threads 10 --max-queued-requests 10
           ├─ 97600 /usr/libexec/ioprocess --read-pipe-fd 282 --write-pipe-fd 281 --max-threads 10 --max-queued-requests 10
           ├─ 98002 /usr/libexec/ioprocess --read-pipe-fd 291 --write-pipe-fd 290 --max-threads 10 --max-queued-requests 10
           ├─ 98232 /usr/libexec/ioprocess --read-pipe-fd 297 --write-pipe-fd 295 --max-threads 10 --max-queued-requests 10
           ├─ 98658 /usr/libexec/ioprocess --read-pipe-fd 305 --write-pipe-fd 304 --max-threads 10 --max-queued-requests 10
           ├─ 98896 /usr/libexec/ioprocess --read-pipe-fd 314 --write-pipe-fd 313 --max-threads 10 --max-queued-requests 10
           ├─ 99337 /usr/libexec/ioprocess --read-pipe-fd 321 --write-pipe-fd 319 --max-threads 10 --max-queued-requests 10
           ├─ 99588 /usr/libexec/ioprocess --read-pipe-fd 330 --write-pipe-fd 329 --max-threads 10 --max-queued-requests 10
           ├─ 99985 /usr/libexec/ioprocess --read-pipe-fd 340 --write-pipe-fd 339 --max-threads 10 --max-queued-requests 10
           ├─100273 /usr/libexec/ioprocess --read-pipe-fd 345 --write-pipe-fd 342 --max-threads 10 --max-queued-requests 10
           ├─100790 /usr/libexec/ioprocess --read-pipe-fd 362 --write-pipe-fd 361 --max-threads 10 --max-queued-requests 10
           ├─101086 /usr/libexec/ioprocess --read-pipe-fd 365 --write-pipe-fd 363 --max-threads 10 --max-queued-requests 10
           ├─101554 /usr/libexec/ioprocess --read-pipe-fd 378 --write-pipe-fd 377 --max-threads 10 --max-queued-requests 10
           ├─101906 /usr/libexec/ioprocess --read-pipe-fd 383 --write-pipe-fd 380 --max-threads 10 --max-queued-requests 10
           ├─102428 /usr/libexec/ioprocess --read-pipe-fd 396 --write-pipe-fd 395 --max-threads 10 --max-queued-requests 10
           ├─102739 /usr/libexec/ioprocess --read-pipe-fd 398 --write-pipe-fd 396 --max-threads 10 --max-queued-requests 10
           ├─103272 /usr/libexec/ioprocess --read-pipe-fd 414 --write-pipe-fd 413 --max-threads 10 --max-queued-requests 10
           ├─103620 /usr/libexec/ioprocess --read-pipe-fd 414 --write-pipe-fd 413 --max-threads 10 --max-queued-requests 10
           ├─104164 /usr/libexec/ioprocess --read-pipe-fd 425 --write-pipe-fd 423 --max-threads 10 --max-queued-requests 10
           ├─104613 /usr/libexec/ioprocess --read-pipe-fd 436 --write-pipe-fd 435 --max-threads 10 --max-queued-requests 10
           ├─105232 /usr/libexec/ioprocess --read-pipe-fd 441 --write-pipe-fd 438 --max-threads 10 --max-queued-requests 10
           ├─105572 /usr/libexec/ioprocess --read-pipe-fd 450 --write-pipe-fd 449 --max-threads 10 --max-queued-requests 10
           ├─106157 /usr/libexec/ioprocess --read-pipe-fd 457 --write-pipe-fd 455 --max-threads 10 --max-queued-requests 10
           ├─106567 /usr/libexec/ioprocess --read-pipe-fd 466 --write-pipe-fd 465 --max-threads 10 --max-queued-requests 10
           ├─107201 /usr/libexec/ioprocess --read-pipe-fd 471 --write-pipe-fd 468 --max-threads 10 --max-queued-requests 10
           ├─107668 /usr/libexec/ioprocess --read-pipe-fd 481 --write-pipe-fd 480 --max-threads 10 --max-queued-requests 10
           ├─108259 /usr/libexec/ioprocess --read-pipe-fd 488 --write-pipe-fd 486 --max-threads 10 --max-queued-requests 10
           ├─108737 /usr/libexec/ioprocess --read-pipe-fd 497 --write-pipe-fd 496 --max-threads 10 --max-queued-requests 10
           ├─109396 /usr/libexec/ioprocess --read-pipe-fd 506 --write-pipe-fd 505 --max-threads 10 --max-queued-requests 10
           ├─109884 /usr/libexec/ioprocess --read-pipe-fd 514 --write-pipe-fd 513 --max-threads 10 --max-queued-requests 10
           ├─110656 /usr/libexec/ioprocess --read-pipe-fd 525 --write-pipe-fd 524 --max-threads 10 --max-queued-requests 10
           ├─111238 /usr/libexec/ioprocess --read-pipe-fd 530 --write-pipe-fd 527 --max-threads 10 --max-queued-requests 10
           ├─111900 /usr/libexec/ioprocess --read-pipe-fd 545 --write-pipe-fd 544 --max-threads 10 --max-queued-requests 10
           ├─112586 /usr/libexec/ioprocess --read-pipe-fd 544 --write-pipe-fd 541 --max-threads 10 --max-queued-requests 10
           ├─113241 /usr/libexec/ioprocess --read-pipe-fd 557 --write-pipe-fd 556 --max-threads 10 --max-queued-requests 10
           ├─113814 /usr/libexec/ioprocess --read-pipe-fd 565 --write-pipe-fd 564 --max-threads 10 --max-queued-requests 10
           ├─114548 /usr/libexec/ioprocess --read-pipe-fd 576 --write-pipe-fd 575 --max-threads 10 --max-queued-requests 10
           ├─115139 /usr/libexec/ioprocess --read-pipe-fd 581 --write-pipe-fd 578 --max-threads 10 --max-queued-requests 10
           ├─115867 /usr/libexec/ioprocess --read-pipe-fd 592 --write-pipe-fd 591 --max-threads 10 --max-queued-requests 10
           ├─116583 /usr/libexec/ioprocess --read-pipe-fd 600 --write-pipe-fd 599 --max-threads 10 --max-queued-requests 10
           ├─117561 /usr/libexec/ioprocess --read-pipe-fd 608 --write-pipe-fd 607 --max-threads 10 --max-queued-requests 10
           ├─118314 /usr/libexec/ioprocess --read-pipe-fd 616 --write-pipe-fd 615 --max-threads 10 --max-queued-requests 10
           ├─119238 /usr/libexec/ioprocess --read-pipe-fd 623 --write-pipe-fd 621 --max-threads 10 --max-queued-requests 10
           ├─120037 /usr/libexec/ioprocess --read-pipe-fd 632 --write-pipe-fd 631 --max-threads 10 --max-queued-requests 10
           ├─121020 /usr/libexec/ioprocess --read-pipe-fd 641 --write-pipe-fd 640 --max-threads 10 --max-queued-requests 10
           ├─121683 /usr/libexec/ioprocess --read-pipe-fd 650 --write-pipe-fd 649 --max-threads 10 --max-queued-requests 10
           └─122603 /usr/libexec/ioprocess --read-pipe-fd 661 --write-pipe-fd 660 --max-threads 10 --max-queued-requests 10

May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter memFree is not uint type
May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter free is not uint type
May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter free is not uint type
May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter free is not uint type
May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter free is not uint type
May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter cpuSysVdsmd is not float type
May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Required property haStatus is not provided when calling Host.getStats
May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter bootTime is not uint type
May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter rxDropped is not uint type
May 24 15:57:53 alma04.qa.lab.tlv.redhat.com vdsm[2913]: vdsm SchemaCache WARNING Parameter anonHugePages is not uint type


Version-Release number of selected component (if applicable):
Engine:
rhevm-setup-plugins-4.0.0-0.3.alpha.el7ev.noarch
rhevm-dependencies-4.0.0-0.1.alpha.git9ae0cc3.el7ev.noarch
rhevm-4.0.0-0.6.master.el7ev.noarch
rhevm-doc-3.6.0-7.el6eng.noarch
rhevm-branding-rhev-4.0.0-0.0.master.20160219183625.el7ev.noarch
postfix-2.10.1-6.el7.x86_64
ovirt-engine-extensions-api-impl-4.0.0-0.6.master.el7ev.noarch
ovirt-host-deploy-1.5.0-0.2.alpha1.git968f12d.el7ev.noarch
ovirt-engine-tools-backup-4.0.0-0.6.master.el7ev.noarch
ovirt-engine-tools-4.0.0-0.6.master.el7ev.noarch
ovirt-iso-uploader-4.0.0-0.2.master.git98e29da.el7ev.noarch
ovirt-host-deploy-java-1.5.0-0.2.alpha1.git968f12d.el7ev.noarch
ovirt-vmconsole-1.0.2-1.el7ev.noarch
ovirt-engine-setup-plugin-vmconsole-proxy-helper-4.0.0-0.6.master.el7ev.noarch
ovirt-setup-lib-1.0.1-1.el7ev.noarch
ovirt-engine-dwh-setup-4.0.0-1.el7ev.noarch
ovirt-engine-websocket-proxy-4.0.0-0.6.master.el7ev.noarch
ovirt-engine-restapi-4.0.0-0.6.master.el7ev.noarch
ovirt-engine-vmconsole-proxy-helper-4.0.0-0.6.master.el7ev.noarch
ovirt-engine-backend-4.0.0-0.6.master.el7ev.noarch
ovirt-engine-setup-base-4.0.0-0.6.master.el7ev.noarch
ovirt-log-collector-4.0.0-0.1.master.git668ba27.el7ev.noarch
ovirt-engine-sdk-python-3.6.5.0-1.el7ev.noarch
ovirt-engine-extension-aaa-jdbc-1.1.0-0.3.el7ev.noarch
ovirt-engine-setup-plugin-ovirt-engine-common-4.0.0-0.6.master.el7ev.noarch
ovirt-engine-setup-plugin-websocket-proxy-4.0.0-0.6.master.el7ev.noarch
ovirt-engine-dbscripts-4.0.0-0.6.master.el7ev.noarch
ovirt-engine-setup-4.0.0-0.6.master.el7ev.noarch
ovirt-engine-userportal-4.0.0-0.6.master.el7ev.noarch
ovirt-image-uploader-4.0.0-0.1.master.gita3e72cd.el7ev.noarch
ovirt-vmconsole-proxy-1.0.2-1.el7ev.noarch
ovirt-engine-setup-plugin-ovirt-engine-4.0.0-0.6.master.el7ev.noarch
ovirt-engine-dwh-4.0.0-1.el7ev.noarch
ovirt-engine-webadmin-portal-4.0.0-0.6.master.el7ev.noarch
ovirt-engine-4.0.0-0.6.master.el7ev.noarch
ovirt-engine-cli-3.6.2.0-1.el7ev.noarch
ovirt-engine-lib-4.0.0-0.6.master.el7ev.noarch
Linux version 3.10.0-327.22.1.el7.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC) ) #1 SMP Mon May 16 13:31:48 EDT 2016
Linux 3.10.0-327.22.1.el7.x86_64 #1 SMP Mon May 16 13:31:48 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 7.2 (Maipo)


Host:
ovirt-vmconsole-host-1.0.2-0.0.master.20160517094103.git06df50a.el7.noarch
ovirt-setup-lib-1.0.2-0.0.master.20160502125738.gitf05af9e.el7.centos.noarch
ovirt-release40-4.0.0-0.3.beta1.noarch
ovirt-vmconsole-1.0.2-0.0.master.20160517094103.git06df50a.el7.noarch
ovirt-engine-sdk-python-3.6.5.1-0.1.20160507.git5fb7e0e.el7.centos.noarch
ovirt-host-deploy-1.5.0-0.1.alpha1.el7.centos.noarch
ovirt-hosted-engine-setup-2.0.0-0.1.beta1.el7.centos.noarch
ovirt-release-host-node-4.0.0-0.3.beta1.el7.noarch
ovirt-hosted-engine-ha-2.0.0-0.1.beta1.el7.centos.noarch
ovirt-node-ng-image-update-placeholder-4.0.0-0.3.beta1.el7.noarch
libvirt-client-1.2.17-13.el7_2.4.x86_64
mom-0.5.3-1.1.el7.noarch
vdsm-4.17.999-1155.gitcf216a0.el7.centos.x86_64
sanlock-3.2.4-2.el7_2.x86_64
qemu-kvm-tools-ev-2.3.0-31.el7_2.10.1.x86_64
qemu-kvm-ev-2.3.0-31.el7_2.10.1.x86_64
qemu-kvm-common-ev-2.3.0-31.el7_2.10.1.x86_64
Linux version 3.10.0-327.18.2.el7.x86_64 (builder.centos.org) (gcc version 4.8.3 20140911 (Red Hat 4.8.3-9) (GCC) ) #1 SMP Thu May 12 11:03:55 UTC 2016
Linux 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
CentOS Linux release 7.2.1511 (Core) 

How reproducible:
100%

Steps to Reproduce:
1.Deploy 4.0HE over NFS on NGN 4.0.
2.Add one data storage domain (SD) to get hosted_storage auto-imported and to get the HE-VM visible.
3.Create 300 NFS SDs.

Actual results:
On creation of 54th SD the host failed to continue and all SDs previously active, went in to Inactive state.


Expected results:
Creation of 300 SDs should succeed.

Additional info:
Comment 1 Nikolai Sednev 2016-05-24 13:13:26 UTC 
Created attachment 1161044 [details]
sosreport from engine
Comment 2 Nikolai Sednev 2016-05-24 13:20:33 UTC 
Adding external link to host's logs https://drive.google.com/a/redhat.com/file/d/0B85BEaDBcF88bFpYOGItN2lOcUU/view?usp=sharing
Comment 3 Tal Nisan 2016-05-24 20:38:27 UTC 
Freddy, as this weeks QE contact please have a look at the logs.
Targeting for now to 4.1 till we know better what's going on here
Comment 4 Allon Mureinik 2016-05-25 06:31:16 UTC 
The relevant part of the log:

Traceback (most recent call last):
  File "/usr/share/vdsm/storage/task.py", line 875, in _run
    return fn(*args, **kargs)
  File "/usr/lib/python2.7/site-packages/vdsm/logUtils.py", line 50, in wrapper
    res = f(*args, **kwargs)
  File "/usr/share/vdsm/storage/hsm.py", line 987, in connectStoragePool
    spUUID, hostID, msdUUID, masterVersion, domainsMap)
  File "/usr/share/vdsm/storage/hsm.py", line 1052, in _connectStoragePool
    res = pool.connect(hostID, msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 664, in connect
    self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1237, in __rebuild
    self.setMasterDomain(msdUUID, masterVersion)
  File "/usr/share/vdsm/storage/sp.py", line 1445, in setMasterDomain
    domain = sdCache.produce(msdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 101, in produce
    domain.getRealDomain()
  File "/usr/share/vdsm/storage/sdc.py", line 53, in getRealDomain
    return self._cache._realProduce(self._sdUUID)
  File "/usr/share/vdsm/storage/sdc.py", line 123, in _realProduce
    self.refreshStorage()
  File "/usr/lib/python2.7/site-packages/vdsm/storage/misc.py", line 616, in helper
    return sm(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/misc.py", line 601, in __call__
    self.__lastResult = self.__func(*args, **kwargs)
  File "/usr/share/vdsm/storage/sdc.py", line 84, in refreshStorage
    multipath.rescan()
  File "/usr/share/vdsm/storage/multipath.py", line 74, in rescan
    iscsi.rescan()
  File "/usr/lib/python2.7/site-packages/vdsm/storage/misc.py", line 616, in helper
    return sm(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/misc.py", line 601, in __call__
    self.__lastResult = self.__func(*args, **kwargs)
  File "/usr/share/vdsm/storage/iscsi.py", line 435, in rescan
    rescanOp = iscsiadm.session_rescan_async()
  File "/usr/share/vdsm/storage/iscsiadm.py", line 340, in session_rescan_async
    proc = _runCmd(["-m", "session", "-R"], sync=False)
  File "/usr/share/vdsm/storage/iscsiadm.py", line 119, in _runCmd
    return misc.execCmd(cmd, printable=printCmd, sudo=True, sync=sync)
  File "/usr/lib/python2.7/site-packages/vdsm/commands.py", line 71, in execCmd
    deathSignal=deathSignal, childUmask=childUmask)
  File "/usr/lib64/python2.7/site-packages/cpopen/__init__.py", line 63, in __init__
    **kw)
  File "/usr/lib64/python2.7/subprocess.py", line 703, in __init__
    errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
  File "/usr/lib64/python2.7/subprocess.py", line 1112, in _get_handles
    p2cread, p2cwrite = self.pipe_cloexec()
  File "/usr/lib64/python2.7/subprocess.py", line 1168, in pipe_cloexec
    r, w = os.pipe()
OSError: [Errno 24] Too many open files

I wonder if we're leaking an FD somewhere.
Comment 5 Yaniv Kaul 2016-05-25 07:06:37 UTC 
Reducing severity. >50 SDs is quite rare.
Comment 6 Yaniv Kaul 2016-05-25 07:18:51 UTC 
Is this a regression?
Comment 7 Fred Rolland 2016-05-25 07:36:14 UTC 
Created attachment 1161328 [details]
lsof output
Comment 8 Fred Rolland 2016-05-25 14:55:05 UTC 
How did you create the Storage Domains ?
Comment 9 Fred Rolland 2016-05-25 15:30:24 UTC 
vdsm process is limited to 1024 fd

[root@RHEL7 ~]# cat /proc/23522/limits 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             7280                 7280                 processes 
Max open files            1024                 4096                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       7280                 7280                 signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us
Comment 10 Fred Rolland 2016-05-25 15:46:30 UTC 
I also reached the "too may open files" when creating + 55 File SD.

The ulimit of the vdsm process is 1024 (see comment #9).

It seems it has been changed a long time ago by this patch:
https://gerrit.ovirt.org/#/c/914/2
It sets the limit to the user not to the process.
Comment 11 Nikolai Sednev 2016-05-26 05:44:41 UTC 
(In reply to Yaniv Kaul from comment #6)
> Is this a regression?

Yes, it is a regression, as previously at least 100 SDs creation at a single servere was possible and worked.

(In reply to Fred Rolland from comment #8)
> How did you create the Storage Domains ?

By using Selenium and proper REST-APIs. I've seen live that first 53 were created OK, then they were attached to DC and became Active, no problem, but at the 54th they all became Inactive.
Comment 12 Fred Rolland 2016-05-26 09:12:59 UTC 
What type of SDs (File/Block) were created on the 100 SDs case ?
Comment 13 Allon Mureinik 2016-05-26 10:20:02 UTC 
Related to bug 1339777?
Comment 14 Nir Soffer 2016-05-26 11:26:08 UTC 
(In reply to Allon Mureinik from comment #13)
> Related to bug 1339777?

No, this this is leak of exactly one fd for each ioprocess, so this is 2 fds
max per storage domain. We create 2 ioprocess per domain - one for the mount
point, stopped after 1 minute of idle time, and the second for the domain, 
kept alive by the monitor.
Comment 15 Nir Soffer 2016-05-26 11:31:02 UTC 
(In reply to Nikolai Sednev from comment #0)
> Description of problem:
> In order to verify the https://bugzilla.redhat.com/show_bug.cgi?id=1317468,
> I've tried to create 300 SDs.

We don't support this amount.

> Expected results:
> Creation of 300 SDs should succeed.

It is not specified, and the system will not be useable in this state.

Nikolai, check with Yaniv what are the official limits.
Comment 16 Fred Rolland 2016-05-29 06:32:01 UTC 
Workaround :
- Edit : /lib/systemd/system/vdsmd.service
- Add following lines to ‘[Service]‘:
LimitNOFILE=4096
- Reload daemon:
systemctl daemon-reload
- Reload service:
systemctl restart vdsmd
Comment 17 Nikolai Sednev 2016-05-29 08:30:49 UTC 
(In reply to Nir Soffer from comment #15)
> (In reply to Nikolai Sednev from comment #0)
> > Description of problem:
> > In order to verify the https://bugzilla.redhat.com/show_bug.cgi?id=1317468,
> > I've tried to create 300 SDs.
> 
> We don't support this amount.

What is maximum supported amount for VGs in RHEVM?
> 
> > Expected results:
> > Creation of 300 SDs should succeed.
> 
> It is not specified, and the system will not be useable in this state.
> 
> Nikolai, check with Yaniv what are the official limits.
Anyway, system should no hang or disconnect the existing resources, instead of this it should bring warning of maximum number of VGs been reached, further addition is not supported.

Yaniv, can you please clarify this issue?
Comment 18 Yaniv Lavi 2016-06-02 12:19:30 UTC 
(In reply to Nikolai Sednev from comment #17)
> (In reply to Nir Soffer from comment #15)
> > (In reply to Nikolai Sednev from comment #0)
> > > Description of problem:
> > > In order to verify the https://bugzilla.redhat.com/show_bug.cgi?id=1317468,
> > > I've tried to create 300 SDs.
> > 
> > We don't support this amount.
> 
> What is maximum supported amount for VGs in RHEVM?

VGs ? we should aim to support more than 50. This depends on the HW in any case.

> > 
> > > Expected results:
> > > Creation of 300 SDs should succeed.
> > 
> > It is not specified, and the system will not be useable in this state.
> > 
> > Nikolai, check with Yaniv what are the official limits.
> Anyway, system should no hang or disconnect the existing resources, instead
> of this it should bring warning of maximum number of VGs been reached,
> further addition is not supported.
> 
> Yaniv, can you please clarify this issue?

We can't enforce this. It depends on the storage and hosts specs.
Comment 19 Nikolai Sednev 2016-06-02 13:16:38 UTC 
(In reply to Yaniv Dary from comment #18)
> (In reply to Nikolai Sednev from comment #17)
> > (In reply to Nir Soffer from comment #15)
> > > (In reply to Nikolai Sednev from comment #0)
> > > > Description of problem:
> > > > In order to verify the https://bugzilla.redhat.com/show_bug.cgi?id=1317468,
> > > > I've tried to create 300 SDs.
> > > 
> > > We don't support this amount.
> > 
> > What is maximum supported amount for VGs in RHEVM?
> 
> VGs ? we should aim to support more than 50. This depends on the HW in any
> case.
> 
> > > 
> > > > Expected results:
> > > > Creation of 300 SDs should succeed.
> > > 
> > > It is not specified, and the system will not be useable in this state.
> > > 
> > > Nikolai, check with Yaniv what are the official limits.
> > Anyway, system should no hang or disconnect the existing resources, instead
> > of this it should bring warning of maximum number of VGs been reached,
> > further addition is not supported.
> > 
> > Yaniv, can you please clarify this issue?
> 
> We can't enforce this. It depends on the storage and hosts specs.
VGs=Volume Groups=Storage Domains within the engine.

I've seen that on host the memory was used ~6Gig from 32Gig available, all other resources were freely available. IMHO the system crushed and that what should not happen, no active data storage domains should be disconnected. 

So what customer should expect and are there any strict limitations available and documented?
Comment 20 Yaniv Lavi 2016-06-02 14:12:18 UTC 
(In reply to Nikolai Sednev from comment #19)
> 
> So what customer should expect and are there any strict limitations
> available and documented?

We don't have a published sizing guide, but it is the number we have been requesting scale team to test and development to make sure works well.
Comment 21 Sven Kieske 2016-06-10 12:23:55 UTC 
Well I have one engine with currently around 40 Storage domains in a live 3.3.3.
deployment and plan to go much beyond that in my (now at testing stage) 3.6.z deployment.

So I really hope you can push the numbers at least to 300 :)

I think (from reading this BZ, please correct me if I'm wrong)

we have 2 issues at hand:

1. ioprocess leaking 2 fds per SD (can this be stopped?)
2. fd limit of 1024 of vdsm, which should maybe get increased (each administrator can increase it himself, as a workaround, but I bet it get's overwritten on every vdsm update?)


HTH

Sven
Comment 22 Nir Soffer 2016-06-10 13:12:12 UTC 
(In reply to Sven Kieske from comment #21)
> Well I have one engine with currently around 40 Storage domains in a live
> 3.3.3.
> deployment and plan to go much beyond that in my (now at testing stage)
> 3.6.z deployment.
> 
> So I really hope you can push the numbers at least to 300 :)

Storage domain is heavyweight, using one thread per storage domain and for
file based storage, and 1 ioprocess thread. We also perform io every 10 seconds
on each domain. With this design, 300 storage domain are not possible.

> I think (from reading this BZ, please correct me if I'm wrong)
> 
> we have 2 issues at hand:
> 
> 1. ioprocess leaking 2 fds per SD (can this be stopped?)

We don't have any fd leak, we just use too many of them for each ioprocess.

Current ioprocess uses 8 fds per instance (in vdsm). This was decreased to 
6 in upstream patches:
- https://gerrit.ovirt.org/58612
- https://gerrit.ovirt.org/58613

The lowest number of descriptors we can have is 2:
- read pipe
- write pipe

To get there, we need to rewrite ipprocess client to use an event loop
instead of one thread per ioprocess. We have the needed infrastructure
now in 4.0, so we can have this in 4.1 if we work on this.

This will allow doubling of the number of storage domains without any
change in the way storage domains and mounts are implemented.

> 2. fd limit of 1024 of vdsm, which should maybe get increased (each
> administrator can increase it himself, as a workaround, but I bet it get's
> overwritten on every vdsm update?)

This is already fixed in master and 3.6, the limit is now 4096:
- https://gerrit.ovirt.org/58123
- https://gerrit.ovirt.org/58321

There is a third issue that is not described in this bug - we are using
more then one ioprocess per storage domain.

Currently we open 1 iorpocess for storage domain, and 1 for mountpoint,
used when refreshing storage.  The ioprocess for mount points are
automatically closed after 60 seconds, since they are used once every
5 minutes during storage refresh, but this means we are creating
100 iprocesses for 50 storage domains, using 800 fds just for this.

Even worse, we actually create one ioprocess per directory in
/rhev/data-center/mnt, so if you have large amount of stale storage
domains directories there (probably removed storage domains), we create
additional usless ioprocesses. This is the root cause for this bug, and
it should be fixed by increasing the number of open files.

We are working on fixing this issue in this topic, creating one ioprocess
per mountpoint:
https://gerrit.ovirt.org/#/q/status:open+project:vdsm+branch:master+topic:oop-per-mount

These changes are invasive, no chance to get them before 4.1.

For file based storage domains, if we want to support bigger number of 
storage domains, we must separate storage domains from mounts, like 
we do in block storage. Then we can have 50-100 mounts checking accessibility
of the mounts, and unlimited of number of storage domains, where each mount
provide lot of storage domains.

Can you explain why you need 300 storage domains? and which kind of 
storage - file? block? Please open RFE for this if you have a good use
case, we may work on it for a future version.

You may also start a thread about this on the users mailing list, there may
be others that need this functionality, and we may get useful info.
Comment 23 guy chen 2016-06-15 12:44:10 UTC 
Created attachment 1168368 [details]
vdsm log
Comment 24 guy chen 2016-06-15 12:46:14 UTC 
I have run creation of 300 SD on RHEVM 3.6.7.2-0.1.el6, up to 100 all run O.k then reached Storage pool limit :

Failed to attach Storage Domain storage_scale_101 to Data Center Default. (User: admin@internal)
VDSM command failed: Too many domains in Storage pool: ()

But, 100 SD where created successfully with non of the reported symptoms, thus bug is fixed.
Further info :

systemctl status libvirtd -l: 

● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/libvirtd.service.d
           └─unlimited-core.conf
   Active: active (running) since Thu 2016-06-09 10:42:00 UTC; 6 days ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 1163 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           └─1163 /usr/sbin/libvirtd --listen

Jun 09 10:41:59 host12-rack04.scale.openstack.engineering.redhat.com systemd[1]: Starting Virtualization daemon...
Jun 09 10:41:59 host12-rack04.scale.openstack.engineering.redhat.com libvirtd[1163]: libvirt version: 1.2.17, package: 13.el7_2.5 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2016-05-26-07:48:46, x86-020.build.eng.bos.redhat.com)
Jun 09 10:41:59 host12-rack04.scale.openstack.engineering.redhat.com libvirtd[1163]: Module /usr/lib64/libvirt/connection-driver/libvirt_driver_lxc.so not accessible
Jun 09 10:42:00 host12-rack04.scale.openstack.engineering.redhat.com systemd[1]: Started Virtualization daemon.
Jun 13 09:09:13 host12-rack04.scale.openstack.engineering.redhat.com libvirtd[1163]: Received unexpected event 3
Jun 13 09:09:52 host12-rack04.scale.openstack.engineering.redhat.com libvirtd[1163]: Received unexpected event 3
Jun 13 09:54:15 host12-rack04.scale.openstack.engineering.redhat.com libvirtd[1163]: Received unexpected event 3
Jun 13 09:54:16 host12-rack04.scale.openstack.engineering.redhat.com libvirtd[1163]: Received unexpected event 3
Jun 13 10:05:43 host12-rack04.scale.openstack.engineering.redhat.com libvirtd[1163]: Received unexpected event 3


systemctl status vdsmd -l:

● vdsmd.service - Virtual Desktop Server Manager
   Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2016-06-09 10:42:06 UTC; 6 days ago
 Main PID: 2535 (vdsm)
   CGroup: /system.slice/vdsmd.service
           ├─  358 /usr/libexec/ioprocess --read-pipe-fd 880 --write-pipe-fd 879 --max-threads 10 --max-queued-requests 10
           ├─  625 /usr/libexec/ioprocess --read-pipe-fd 1513 --write-pipe-fd 1512 --max-threads 10 --max-queued-requests 10
           ├─  876 /usr/libexec/ioprocess --read-pipe-fd 1583 --write-pipe-fd 1582 --max-threads 10 --max-queued-requests 10
           ├─  942 /usr/libexec/ioprocess --read-pipe-fd 281 --write-pipe-fd 280 --max-threads 10 --max-queued-requests 10
           ├─ 1515 /usr/libexec/ioprocess --read-pipe-fd 1707 --write-pipe-fd 1706 --max-threads 10 --max-queued-requests 10
           ├─ 1777 /usr/libexec/ioprocess --read-pipe-fd 297 --write-pipe-fd 296 --max-threads 10 --max-queued-requests 10
           ├─ 1900 /usr/libexec/ioprocess --read-pipe-fd 685 --write-pipe-fd 684 --max-threads 10 --max-queued-requests 10
           ├─ 2380 /usr/libexec/ioprocess --read-pipe-fd 1167 --write-pipe-fd 1166 --max-threads 10 --max-queued-requests 10
           ├─ 2535 /usr/bin/python /usr/share/vdsm/vdsm
           ├─ 2704 /usr/libexec/ioprocess --read-pipe-fd 313 --write-pipe-fd 312 --max-threads 10 --max-queued-requests 10
           ├─ 3357 /usr/libexec/ioprocess --read-pipe-fd 321 --write-pipe-fd 320 --max-threads 10 --max-queued-requests 10
           ├─ 3483 /usr/libexec/ioprocess --read-pipe-fd 328 --write-pipe-fd 326 --max-threads 10 --max-queued-requests 10
           ├─ 3539 /usr/libexec/ioprocess --read-pipe-fd 639 --write-pipe-fd 637 --max-threads 10 --max-queued-requests 10
           ├─ 3543 /usr/libexec/ioprocess --read-pipe-fd 648 --write-pipe-fd 643 --max-threads 10 --max-queued-requests 10
           ├─ 3554 /usr/libexec/ioprocess --read-pipe-fd 669 --write-pipe-fd 664 --max-threads 10 --max-queued-requests 10
           ├─ 3563 /usr/libexec/ioprocess --read-pipe-fd 702 --write-pipe-fd 697 --max-threads 10 --max-queued-requests 10
           ├─ 3571 /usr/libexec/ioprocess --read-pipe-fd 719 --write-pipe-fd 714 --max-threads 10 --max-queued-requests 10
           ├─ 3579 /usr/libexec/ioprocess --read-pipe-fd 733 --write-pipe-fd 731 --max-threads 10 --max-queued-requests 10
           ├─ 3587 /usr/libexec/ioprocess --read-pipe-fd 749 --write-pipe-fd 747 --max-threads 10 --max-queued-requests 10
           ├─ 3596 /usr/libexec/ioprocess --read-pipe-fd 770 --write-pipe-fd 763 --max-threads 10 --max-queued-requests 10
           ├─ 3604 /usr/libexec/ioprocess --read-pipe-fd 786 --write-pipe-fd 781 --max-threads 10 --max-queued-requests 10
           ├─ 3612 /usr/libexec/ioprocess --read-pipe-fd 800 --write-pipe-fd 798 --max-threads 10 --max-queued-requests 10
           ├─ 3620 /usr/libexec/ioprocess --read-pipe-fd 816 --write-pipe-fd 814 --max-threads 10 --max-queued-requests 10
           ├─ 3628 /usr/libexec/ioprocess --read-pipe-fd 835 --write-pipe-fd 832 --max-threads 10 --max-queued-requests 10
           ├─ 3636 /usr/libexec/ioprocess --read-pipe-fd 853 --write-pipe-fd 851 --max-threads 10 --max-queued-requests 10
           ├─ 3645 /usr/libexec/ioprocess --read-pipe-fd 870 --write-pipe-fd 868 --max-threads 10 --max-queued-requests 10
           ├─ 3655 /usr/libexec/ioprocess --read-pipe-fd 1048 --write-pipe-fd 1047 --max-threads 10 --max-queued-requests 10
           ├─ 3661 /usr/libexec/ioprocess --read-pipe-fd 885 --write-pipe-fd 882 --max-threads 10 --max-queued-requests 10
           ├─ 3668 /usr/libexec/ioprocess --read-pipe-fd 903 --write-pipe-fd 901 --max-threads 10 --max-queued-requests 10
           ├─ 3672 /usr/libexec/ioprocess --read-pipe-fd 896 --write-pipe-fd 895 --max-threads 10 --max-queued-requests 10
           ├─ 3681 /usr/libexec/ioprocess --read-pipe-fd 937 --write-pipe-fd 935 --max-threads 10 --max-queued-requests 10
           ├─ 3688 /usr/libexec/ioprocess --read-pipe-fd 954 --write-pipe-fd 952 --max-threads 10 --max-queued-requests 10
           ├─ 3695 /usr/libexec/ioprocess --read-pipe-fd 969 --write-pipe-fd 966 --max-threads 10 --max-queued-requests 10
           ├─ 3702 /usr/libexec/ioprocess --read-pipe-fd 985 --write-pipe-fd 982 --max-threads 10 --max-queued-requests 10
           ├─ 3792 /usr/libexec/ioprocess --read-pipe-fd 1357 --write-pipe-fd 1356 --max-threads 10 --max-queued-requests 10
           ├─ 4091 /usr/libexec/ioprocess --read-pipe-fd 1007 --write-pipe-fd 1005 --max-threads 10 --max-queued-requests 10
           ├─ 4096 /usr/libexec/ioprocess --read-pipe-fd 1021 --write-pipe-fd 1010 --max-threads 10 --max-queued-requests 10
           ├─ 4105 /usr/libexec/ioprocess --read-pipe-fd 1032 --write-pipe-fd 1030 --max-threads 10 --max-queued-requests 10
           ├─ 4115 /usr/libexec/ioprocess --read-pipe-fd 1058 --write-pipe-fd 1056 --max-threads 10 --max-queued-requests 10
           ├─ 4123 /usr/libexec/ioprocess --read-pipe-fd 1069 --write-pipe-fd 1067 --max-threads 10 --max-queued-requests 10
           ├─ 4132 /usr/libexec/ioprocess --read-pipe-fd 1085 --write-pipe-fd 1082 --max-threads 10 --max-queued-requests 10
           ├─ 4140 /usr/libexec/ioprocess --read-pipe-fd 1118 --write-pipe-fd 1115 --max-threads 10 --max-queued-requests 10
           ├─ 4148 /usr/libexec/ioprocess --read-pipe-fd 1131 --write-pipe-fd 1129 --max-threads 10 --max-queued-requests 10
           ├─ 4156 /usr/libexec/ioprocess --read-pipe-fd 1149 --write-pipe-fd 1147 --max-threads 10 --max-queued-requests 10
           ├─ 4164 /usr/libexec/ioprocess --read-pipe-fd 1165 --write-pipe-fd 1164 --max-threads 10 --max-queued-requests 10
           ├─ 4172 /usr/libexec/ioprocess --read-pipe-fd 1198 --write-pipe-fd 1197 --max-threads 10 --max-queued-requests 10
           ├─ 4180 /usr/libexec/ioprocess --read-pipe-fd 1217 --write-pipe-fd 1214 --max-threads 10 --max-queued-requests 10
           ├─ 4188 /usr/libexec/ioprocess --read-pipe-fd 1254 --write-pipe-fd 1252 --max-threads 10 --max-queued-requests 10
           ├─ 4196 /usr/libexec/ioprocess --read-pipe-fd 1273 --write-pipe-fd 1272 --max-threads 10 --max-queued-requests 10
           ├─ 4204 /usr/libexec/ioprocess --read-pipe-fd 1291 --write-pipe-fd 1290 --max-threads 10 --max-queued-requests 10
           ├─ 4212 /usr/libexec/ioprocess --read-pipe-fd 1306 --write-pipe-fd 1303 --max-threads 10 --max-queued-requests 10
           ├─ 4220 /usr/libexec/ioprocess --read-pipe-fd 1337 --write-pipe-fd 1336 --max-threads 10 --max-queued-requests 10
           ├─ 4228 /usr/libexec/ioprocess --read-pipe-fd 1372 --write-pipe-fd 1371 --max-threads 10 --max-queued-requests 10
           ├─ 4236 /usr/libexec/ioprocess --read-pipe-fd 1387 --write-pipe-fd 1385 --max-threads 10 --max-queued-requests 10
           ├─ 4245 /usr/libexec/ioprocess --read-pipe-fd 1421 --write-pipe-fd 1419 --max-threads 10 --max-queued-requests 10
           ├─ 4253 /usr/libexec/ioprocess --read-pipe-fd 1440 --write-pipe-fd 1439 --max-threads 10 --max-queued-requests 10
           ├─ 4261 /usr/libexec/ioprocess --read-pipe-fd 1455 --write-pipe-fd 1454 --max-threads 10 --max-queued-requests 10
           ├─ 4269 /usr/libexec/ioprocess --read-pipe-fd 1468 --write-pipe-fd 1466 --max-threads 10 --max-queued-requests 10
           ├─ 4277 /usr/libexec/ioprocess --read-pipe-fd 1488 --write-pipe-fd 1487 --max-threads 10 --max-queued-requests 10
           ├─ 4285 /usr/libexec/ioprocess --read-pipe-fd 1504 --write-pipe-fd 1502 --max-threads 10 --max-queued-requests 10
           ├─ 4295 /usr/libexec/ioprocess --read-pipe-fd 1522 --write-pipe-fd 1521 --max-threads 10 --max-queued-requests 10
           ├─ 4303 /usr/libexec/ioprocess --read-pipe-fd 1534 --write-pipe-fd 1530 --max-threads 10 --max-queued-requests 10
           ├─ 4311 /usr/libexec/ioprocess --read-pipe-fd 1550 --write-pipe-fd 1549 --max-threads 10 --max-queued-requests 10
           ├─ 4319 /usr/libexec/ioprocess --read-pipe-fd 1572 --write-pipe-fd 1569 --max-threads 10 --max-queued-requests 10
           ├─ 4325 /usr/libexec/ioprocess --read-pipe-fd 348 --write-pipe-fd 347 --max-threads 10 --max-queued-requests 10
           ├─ 4333 /usr/libexec/ioprocess --read-pipe-fd 1593 --write-pipe-fd 1591 --max-threads 10 --max-queued-requests 10
           ├─ 4340 /usr/libexec/ioprocess --read-pipe-fd 1612 --write-pipe-fd 1611 --max-threads 10 --max-queued-requests 10
           ├─ 4348 /usr/libexec/ioprocess --read-pipe-fd 1625 --write-pipe-fd 1624 --max-threads 10 --max-queued-requests 10
           ├─ 4355 /usr/libexec/ioprocess --read-pipe-fd 1646 --write-pipe-fd 1645 --max-threads 10 --max-queued-requests 10
           ├─ 4362 /usr/libexec/ioprocess --read-pipe-fd 1667 --write-pipe-fd 1665 --max-threads 10 --max-queued-requests 10
           ├─ 4369 /usr/libexec/ioprocess --read-pipe-fd 1692 --write-pipe-fd 1691 --max-threads 10 --max-queued-requests 10
           ├─ 4409 /usr/libexec/ioprocess --read-pipe-fd 1363 --write-pipe-fd 1360 --max-threads 10 --max-queued-requests 10
           ├─ 4448 /usr/libexec/ioprocess --read-pipe-fd 685 --write-pipe-fd 681 --max-threads 10 --max-queued-requests 10
           ├─ 4700 /usr/libexec/ioprocess --read-pipe-fd 697 --write-pipe-fd 696 --max-threads 10 --max-queued-requests 10
           ├─ 4840 /usr/libexec/ioprocess --read-pipe-fd 1443 --write-pipe-fd 1442 --max-threads 10 --max-queued-requests 10
           ├─ 5205 /usr/libexec/ioprocess --read-pipe-fd 362 --write-pipe-fd 361 --max-threads 10 --max-queued-requests 10
           ├─ 5787 /usr/libexec/ioprocess --read-pipe-fd 1654 --write-pipe-fd 1653 --max-threads 10 --max-queued-requests 10
           ├─ 6206 /usr/libexec/ioprocess --read-pipe-fd 379 --write-pipe-fd 378 --max-threads 10 --max-queued-requests 10
           ├─ 6436 /usr/libexec/ioprocess --read-pipe-fd 1280 --write-pipe-fd 1279 --max-threads 10 --max-queued-requests 10
           ├─ 6821 /usr/libexec/ioprocess --read-pipe-fd 714 --write-pipe-fd 713 --max-threads 10 --max-queued-requests 10
           ├─ 6935 /usr/libexec/ioprocess --read-pipe-fd 913 --write-pipe-fd 912 --max-threads 10 --max-queued-requests 10
           ├─ 7024 /usr/libexec/ioprocess --read-pipe-fd 385 --write-pipe-fd 382 --max-threads 10 --max-queued-requests 10
           ├─ 7058 /usr/libexec/ioprocess --read-pipe-fd 1181 --write-pipe-fd 1180 --max-threads 10 --max-queued-requests 10
           ├─ 7145 /usr/libexec/ioprocess --read-pipe-fd 395 --write-pipe-fd 394 --max-threads 10 --max-queued-requests 10
           ├─ 7610 /usr/libexec/ioprocess --read-pipe-fd 1679 --write-pipe-fd 1677 --max-threads 10 --max-queued-requests 10
           ├─ 8019 /usr/libexec/ioprocess --read-pipe-fd 1060 --write-pipe-fd 1057 --max-threads 10 --max-queued-requests 10
           ├─ 8070 /usr/libexec/ioprocess --read-pipe-fd 1733 --write-pipe-fd 1732 --max-threads 10 --max-queued-requests 10
           ├─ 8156 /usr/libexec/ioprocess --read-pipe-fd 409 --write-pipe-fd 406 --max-threads 10 --max-queued-requests 10
           ├─ 8619 /usr/libexec/ioprocess --read-pipe-fd 1538 --write-pipe-fd 1537 --max-threads 10 --max-queued-requests 10
           ├─ 9041 /usr/libexec/ioprocess --read-pipe-fd 731 --write-pipe-fd 730 --max-threads 10 --max-queued-requests 10
           ├─ 9186 /usr/libexec/ioprocess --read-pipe-fd 430 --write-pipe-fd 429 --max-threads 10 --max-queued-requests 10
           ├─10007 /usr/libexec/ioprocess --read-pipe-fd 922 --write-pipe-fd 921 --max-threads 10 --max-queued-requests 10
           ├─10302 /usr/libexec/ioprocess --read-pipe-fd 930 --write-pipe-fd 929 --max-threads 10 --max-queued-requests 10
           ├─10452 /usr/libexec/ioprocess --read-pipe-fd 1379 --write-pipe-fd 1376 --max-threads 10 --max-queued-requests 10
           ├─10531 /usr/libexec/ioprocess --read-pipe-fd 448 --write-pipe-fd 447 --max-threads 10 --max-queued-requests 10
           ├─10551 /usr/libexec/ioprocess --read-pipe-fd 1604 --write-pipe-fd 1603 --max-threads 10 --max-queued-requests 10
           ├─11237 /usr/libexec/ioprocess --read-pipe-fd 1465 --write-pipe-fd 1464 --max-threads 10 --max-queued-requests 10
           ├─11322 /usr/libexec/ioprocess --read-pipe-fd 746 --write-pipe-fd 743 --max-threads 10 --max-queued-requests 10
           ├─11441 /usr/libexec/ioprocess --read-pipe-fd 1188 --write-pipe-fd 1186 --max-threads 10 --max-queued-requests 10
           ├─11746 /usr/libexec/ioprocess --read-pipe-fd 1198 --write-pipe-fd 1197 --max-threads 10 --max-queued-requests 10
           ├─11775 /usr/libexec/ioprocess --read-pipe-fd 464 --write-pipe-fd 463 --max-threads 10 --max-queued-requests 10
           ├─12038 /usr/libexec/ioprocess --read-pipe-fd 1294 --write-pipe-fd 1292 --max-threads 10 --max-queued-requests 10
           ├─12215 /usr/libexec/ioprocess --read-pipe-fd 1080 --write-pipe-fd 1079 --max-threads 10 --max-queued-requests 10
           ├─12783 /usr/libexec/ioprocess --read-pipe-fd 1724 --write-pipe-fd 1721 --max-threads 10 --max-queued-requests 10
           ├─12977 /usr/libexec/ioprocess --read-pipe-fd 480 --write-pipe-fd 479 --max-threads 10 --max-queued-requests 10
           ├─13567 /usr/libexec/ioprocess --read-pipe-fd 947 --write-pipe-fd 946 --max-threads 10 --max-queued-requests 10
           ├─13678 /usr/libexec/ioprocess --read-pipe-fd 763 --write-pipe-fd 762 --max-threads 10 --max-queued-requests 10
           ├─14375 /usr/libexec/ioprocess --read-pipe-fd 496 --write-pipe-fd 495 --max-threads 10 --max-queued-requests 10
           ├─15078 /usr/libexec/ioprocess --read-pipe-fd 1672 --write-pipe-fd 1671 --max-threads 10 --max-queued-requests 10
           ├─15695 /usr/libexec/ioprocess --read-pipe-fd 513 --write-pipe-fd 512 --max-threads 10 --max-queued-requests 10
           ├─16179 /usr/libexec/ioprocess --read-pipe-fd 781 --write-pipe-fd 780 --max-threads 10 --max-queued-requests 10
           ├─16235 /usr/libexec/ioprocess --read-pipe-fd 1097 --write-pipe-fd 1096 --max-threads 10 --max-queued-requests 10
           ├─16564 /usr/libexec/ioprocess --read-pipe-fd 1554 --write-pipe-fd 1553 --max-threads 10 --max-queued-requests 10
           ├─16617 /usr/libexec/ioprocess --read-pipe-fd 1396 --write-pipe-fd 1395 --max-threads 10 --max-queued-requests 10
           ├─16831 /usr/libexec/ioprocess --read-pipe-fd 1212 --write-pipe-fd 1211 --max-threads 10 --max-queued-requests 10
           ├─16897 /usr/libexec/ioprocess --read-pipe-fd 964 --write-pipe-fd 963 --max-threads 10 --max-queued-requests 10
           ├─16914 /usr/libexec/ioprocess --read-pipe-fd 523 --write-pipe-fd 521 --max-threads 10 --max-queued-requests 10
           ├─17100 /usr/libexec/ioprocess --read-pipe-fd 528 --write-pipe-fd 525 --max-threads 10 --max-queued-requests 10
           ├─17903 /usr/libexec/ioprocess --read-pipe-fd 1313 --write-pipe-fd 1312 --max-threads 10 --max-queued-requests 10
           ├─18519 /usr/libexec/ioprocess --read-pipe-fd 545 --write-pipe-fd 544 --max-threads 10 --max-queued-requests 10
           ├─18609 /usr/libexec/ioprocess --read-pipe-fd 1479 --write-pipe-fd 1476 --max-threads 10 --max-queued-requests 10
           ├─18703 /usr/libexec/ioprocess --read-pipe-fd 798 --write-pipe-fd 797 --max-threads 10 --max-queued-requests 10
           ├─18842 /usr/libexec/ioprocess --read-pipe-fd 1625 --write-pipe-fd 1624 --max-threads 10 --max-queued-requests 10
           ├─20293 /usr/libexec/ioprocess --read-pipe-fd 567 --write-pipe-fd 566 --max-threads 10 --max-queued-requests 10
           ├─20305 /usr/libexec/ioprocess --read-pipe-fd 1105 --write-pipe-fd 1103 --max-threads 10 --max-queued-requests 10
           ├─20374 /usr/libexec/ioprocess --read-pipe-fd 980 --write-pipe-fd 979 --max-threads 10 --max-queued-requests 10
           ├─20640 /usr/libexec/ioprocess --read-pipe-fd 1113 --write-pipe-fd 1112 --max-threads 10 --max-queued-requests 10
           ├─21159 /usr/libexec/ioprocess --read-pipe-fd 814 --write-pipe-fd 813 --max-threads 10 --max-queued-requests 10
           ├─21709 /usr/libexec/ioprocess --read-pipe-fd 1228 --write-pipe-fd 1227 --max-threads 10 --max-queued-requests 10
           ├─21890 /usr/libexec/ioprocess --read-pipe-fd 579 --write-pipe-fd 578 --max-threads 10 --max-queued-requests 10
           ├─22310 /usr/libexec/ioprocess --read-pipe-fd 40 --write-pipe-fd 39 --max-threads 10 --max-queued-requests 10
           ├─22319 /usr/libexec/ioprocess --read-pipe-fd 60 --write-pipe-fd 58 --max-threads 10 --max-queued-requests 10
           ├─22335 /usr/libexec/ioprocess --read-pipe-fd 68 --write-pipe-fd 67 --max-threads 10 --max-queued-requests 10
           ├─22353 /usr/libexec/ioprocess --read-pipe-fd 77 --write-pipe-fd 76 --max-threads 10 --max-queued-requests 10
           ├─22363 /usr/libexec/ioprocess --read-pipe-fd 83 --write-pipe-fd 79 --max-threads 10 --max-queued-requests 10
           ├─22374 /usr/libexec/ioprocess --read-pipe-fd 98 --write-pipe-fd 97 --max-threads 10 --max-queued-requests 10
           ├─22384 /usr/libexec/ioprocess --read-pipe-fd 112 --write-pipe-fd 111 --max-threads 10 --max-queued-requests 10
           ├─22395 /usr/libexec/ioprocess --read-pipe-fd 129 --write-pipe-fd 127 --max-threads 10 --max-queued-requests 10
           ├─22406 /usr/libexec/ioprocess --read-pipe-fd 150 --write-pipe-fd 148 --max-threads 10 --max-queued-requests 10
           ├─22421 /usr/libexec/ioprocess --read-pipe-fd 166 --write-pipe-fd 164 --max-threads 10 --max-queued-requests 10
           ├─22433 /usr/libexec/ioprocess --read-pipe-fd 183 --write-pipe-fd 181 --max-threads 10 --max-queued-requests 10
           ├─22445 /usr/libexec/ioprocess --read-pipe-fd 198 --write-pipe-fd 197 --max-threads 10 --max-queued-requests 10
           ├─22456 /usr/libexec/ioprocess --read-pipe-fd 213 --write-pipe-fd 211 --max-threads 10 --max-queued-requests 10
           ├─22466 /usr/libexec/ioprocess --read-pipe-fd 229 --write-pipe-fd 228 --max-threads 10 --max-queued-requests 10
           ├─22478 /usr/libexec/ioprocess --read-pipe-fd 244 --write-pipe-fd 243 --max-threads 10 --max-queued-requests 10
           ├─22492 /usr/libexec/ioprocess --read-pipe-fd 263 --write-pipe-fd 261 --max-threads 10 --max-queued-requests 10
           ├─22502 /usr/libexec/ioprocess --read-pipe-fd 279 --write-pipe-fd 277 --max-threads 10 --max-queued-requests 10
           ├─22518 /usr/libexec/ioprocess --read-pipe-fd 295 --write-pipe-fd 293 --max-threads 10 --max-queued-requests 10
           ├─22528 /usr/libexec/ioprocess --read-pipe-fd 311 --write-pipe-fd 309 --max-threads 10 --max-queued-requests 10
           ├─22541 /usr/libexec/ioprocess --read-pipe-fd 343 --write-pipe-fd 342 --max-threads 10 --max-queued-requests 10
           ├─22553 /usr/libexec/ioprocess --read-pipe-fd 360 --write-pipe-fd 358 --max-threads 10 --max-queued-requests 10
           ├─22564 /usr/libexec/ioprocess --read-pipe-fd 393 --write-pipe-fd 376 --max-threads 10 --max-queued-requests 10
           ├─22574 /usr/libexec/ioprocess --read-pipe-fd 410 --write-pipe-fd 409 --max-threads 10 --max-queued-requests 10
           ├─22587 /usr/libexec/ioprocess --read-pipe-fd 435 --write-pipe-fd 424 --max-threads 10 --max-queued-requests 10
           ├─22598 /usr/libexec/ioprocess --read-pipe-fd 450 --write-pipe-fd 448 --max-threads 10 --max-queued-requests 10
           ├─22609 /usr/libexec/ioprocess --read-pipe-fd 466 --write-pipe-fd 464 --max-threads 10 --max-queued-requests 10
           ├─22625 /usr/libexec/ioprocess --read-pipe-fd 482 --write-pipe-fd 480 --max-threads 10 --max-queued-requests 10
           ├─22631 /usr/libexec/ioprocess --read-pipe-fd 501 --write-pipe-fd 496 --max-threads 10 --max-queued-requests 10
           ├─22644 /usr/libexec/ioprocess --read-pipe-fd 531 --write-pipe-fd 528 --max-threads 10 --max-queued-requests 10
           ├─22653 /usr/libexec/ioprocess --read-pipe-fd 552 --write-pipe-fd 550 --max-threads 10 --max-queued-requests 10
           ├─22665 /usr/libexec/ioprocess --read-pipe-fd 562 --write-pipe-fd 561 --max-threads 10 --max-queued-requests 10
           ├─22674 /usr/libexec/ioprocess --read-pipe-fd 584 --write-pipe-fd 581 --max-threads 10 --max-queued-requests 10
           ├─22686 /usr/libexec/ioprocess --read-pipe-fd 598 --write-pipe-fd 596 --max-threads 10 --max-queued-requests 10
           ├─22697 /usr/libexec/ioprocess --read-pipe-fd 615 --write-pipe-fd 613 --max-threads 10 --max-queued-requests 10
           ├─22951 /usr/libexec/ioprocess --read-pipe-fd 1406 --write-pipe-fd 1405 --max-threads 10 --max-queued-requests 10
           ├─23231 /usr/libexec/ioprocess --read-pipe-fd 1321 --write-pipe-fd 1320 --max-threads 10 --max-queued-requests 10
           ├─23320 /usr/libexec/ioprocess --read-pipe-fd 1414 --write-pipe-fd 1413 --max-threads 10 --max-queued-requests 10
           ├─23590 /usr/libexec/ioprocess --read-pipe-fd 1329 --write-pipe-fd 1328 --max-threads 10 --max-queued-requests 10
           ├─23773 /usr/libexec/ioprocess --read-pipe-fd 597 --write-pipe-fd 596 --max-threads 10 --max-queued-requests 10
           ├─23955 /usr/libexec/ioprocess --read-pipe-fd 830 --write-pipe-fd 829 --max-threads 10 --max-queued-requests 10
           ├─24209 /usr/libexec/ioprocess --read-pipe-fd 996 --write-pipe-fd 994 --max-threads 10 --max-queued-requests 10
           ├─24456 /usr/libexec/ioprocess --read-pipe-fd 1691 --write-pipe-fd 1690 --max-threads 10 --max-queued-requests 10
           ├─25186 /usr/libexec/ioprocess --read-pipe-fd 1131 --write-pipe-fd 1130 --max-threads 10 --max-queued-requests 10
           ├─25193 /usr/libexec/ioprocess --read-pipe-fd 1566 --write-pipe-fd 1565 --max-threads 10 --max-queued-requests 10
           ├─25353 /usr/libexec/ioprocess --read-pipe-fd 1497 --write-pipe-fd 1496 --max-threads 10 --max-queued-requests 10
           ├─25595 /usr/libexec/ioprocess --read-pipe-fd 613 --write-pipe-fd 612 --max-threads 10 --max-queued-requests 10
           ├─26564 /usr/libexec/ioprocess --read-pipe-fd 89 --write-pipe-fd 87 --max-threads 10 --max-queued-requests 10
           ├─26594 /usr/libexec/ioprocess --read-pipe-fd 1238 --write-pipe-fd 1236 --max-threads 10 --max-queued-requests 10
           ├─27001 /usr/libexec/ioprocess --read-pipe-fd 846 --write-pipe-fd 845 --max-threads 10 --max-queued-requests 10
           ├─27204 /usr/libexec/ioprocess --read-pipe-fd 115 --write-pipe-fd 114 --max-threads 10 --max-queued-requests 10
           ├─27322 /usr/libexec/ioprocess --read-pipe-fd 1245 --write-pipe-fd 1244 --max-threads 10 --max-queued-requests 10
           ├─27675 /usr/libexec/ioprocess --read-pipe-fd 131 --write-pipe-fd 130 --max-threads 10 --max-queued-requests 10
           ├─27831 /usr/libexec/ioprocess --read-pipe-fd 50 --write-pipe-fd 49 --max-threads 10 --max-queued-requests 10
           ├─27952 /usr/libexec/ioprocess --read-pipe-fd 632 --write-pipe-fd 631 --max-threads 10 --max-queued-requests 10
           ├─28145 /usr/libexec/ioprocess --read-pipe-fd 144 --write-pipe-fd 140 --max-threads 10 --max-queued-requests 10
           ├─28312 /usr/libexec/ioprocess --read-pipe-fd 1014 --write-pipe-fd 1013 --max-threads 10 --max-queued-requests 10
           ├─28348 /usr/libexec/ioprocess --read-pipe-fd 1638 --write-pipe-fd 1637 --max-threads 10 --max-queued-requests 10
           ├─28606 /usr/libexec/ioprocess --read-pipe-fd 160 --write-pipe-fd 156 --max-threads 10 --max-queued-requests 10
           ├─29218 /usr/libexec/ioprocess --read-pipe-fd 177 --write-pipe-fd 173 --max-threads 10 --max-queued-requests 10
           ├─29776 /usr/libexec/ioprocess --read-pipe-fd 196 --write-pipe-fd 194 --max-threads 10 --max-queued-requests 10
           ├─29943 /usr/libexec/ioprocess --read-pipe-fd 863 --write-pipe-fd 862 --max-threads 10 --max-queued-requests 10
           ├─30023 /usr/libexec/ioprocess --read-pipe-fd 648 --write-pipe-fd 647 --max-threads 10 --max-queued-requests 10
           ├─30054 /usr/libexec/ioprocess --read-pipe-fd 1432 --write-pipe-fd 1431 --max-threads 10 --max-queued-requests 10
           ├─30080 /usr/libexec/ioprocess --read-pipe-fd 1147 --write-pipe-fd 1146 --max-threads 10 --max-queued-requests 10
           ├─30556 /usr/libexec/ioprocess --read-pipe-fd 1347 --write-pipe-fd 1345 --max-threads 10 --max-queued-requests 10
           ├─30647 /usr/libexec/ioprocess --read-pipe-fd 215 --write-pipe-fd 214 --max-threads 10 --max-queued-requests 10
           ├─31237 /usr/libexec/ioprocess --read-pipe-fd 231 --write-pipe-fd 230 --max-threads 10 --max-queued-requests 10
           ├─31836 /usr/libexec/ioprocess --read-pipe-fd 249 --write-pipe-fd 247 --max-threads 10 --max-queued-requests 10
           ├─31909 /usr/libexec/ioprocess --read-pipe-fd 664 --write-pipe-fd 663 --max-threads 10 --max-queued-requests 10
           ├─32075 /usr/libexec/ioprocess --read-pipe-fd 1032 --write-pipe-fd 1031 --max-threads 10 --max-queued-requests 10
           ├─32568 /usr/libexec/ioprocess --read-pipe-fd 1261 --write-pipe-fd 1260 --max-threads 10 --max-queued-requests 10
           └─32625 /usr/libexec/ioprocess --read-pipe-fd 265 --write-pipe-fd 264 --max-threads 10 --max-queued-requests 10

Jun 09 10:42:08 host12-rack04.scale.openstack.engineering.redhat.com python[2535]: DIGEST-MD5 ask_user_info()
Jun 09 10:42:08 host12-rack04.scale.openstack.engineering.redhat.com python[2535]: DIGEST-MD5 make_client_response()
Jun 09 10:42:08 host12-rack04.scale.openstack.engineering.redhat.com python[2535]: DIGEST-MD5 client step 3
Jun 13 09:37:39 host12-rack04.scale.openstack.engineering.redhat.com vdsm[2535]: vdsm root WARNING File: /rhev/data-center/00000001-0001-0001-0001-000000000295/8fd5280c-d61a-4e34-a304-9a687be888f6/images/d9c53eea-ef13-4d36-ac88-b07b3974d5b3/45d35e4e-203b-4e1f-9130-0b79ade080d7 already removed
Jun 13 09:37:40 host12-rack04.scale.openstack.engineering.redhat.com vdsm[2535]: vdsm root WARNING File: /rhev/data-center/00000001-0001-0001-0001-000000000295/8fd5280c-d61a-4e34-a304-9a687be888f6/images/62d12416-9bdc-48cc-b8e8-5957283b32b8/ba70b71c-003c-4637-84f3-b6c096674564 already removed
Jun 14 10:12:02 host12-rack04.scale.openstack.engineering.redhat.com vdsm[2535]: vdsm vds.dispatcher ERROR SSL error during reading data: unexpected eof
Jun 14 10:12:02 host12-rack04.scale.openstack.engineering.redhat.com vdsm[2535]: vdsm vds.dispatcher ERROR SSL error during reading data: unexpected eof
Jun 14 10:14:02 host12-rack04.scale.openstack.engineering.redhat.com su[8630]: (to vdsm) root on none
Jun 15 07:56:17 host12-rack04.scale.openstack.engineering.redhat.com vdsm[2535]: vdsm IOProcess WARNING (50357) Request queue full
Jun 15 07:56:17 host12-rack04.scale.openstack.engineering.redhat.com vdsm[2535]: vdsm IOProcess WARNING (50358) Request queue full