So lets elaborate this further.
The issue is still appearing in 0.1.45.
One queue is used in restraint to send all data out. Which can be dangerous. And now we are hitting it.
Let's say that Restraintd has to install a huge task. (hello /distibution/install) -> Log of logs appearing on the screen. Those will be saved in harness.log. Now, what happened.
Harness.log is stored for upload to lab controller.
SOUP will chunk it in small bits and transfer it to LC. If you will add a bad network to this it will take forever to put it into LC.
Meanwhile, you are trying to transfer harness.log to LC task will fire rstrnt-report-result. This will create a blocking operation between program and restraintd waiting for the result.
Restraintd will add this into the same queue as we have harness.log (which we are already transferring for 2 decades because of internets).
Description of problem: [root@ibm-p8-kvm-11-guest-07 ~]# systemctl status restraintd ● restraintd.service - The restraint harness. Loaded: loaded (/usr/lib/systemd/system/restraintd.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2020-01-17 10:40:55 EST; 24min ago Process: 1748 ExecStartPre=/usr/bin/check_beaker (code=exited, status=0/SUCCESS) Main PID: 1757 (restraintd) CGroup: /system.slice/restraintd.service ├─ 1757 /usr/bin/restraintd ├─ 1865 make run ├─ 1878 /bin/bash ./runtest.sh ├─11359 /usr/bin/rstrnt-report-result --rhts Test PASS /tmp/tmp.7nwdD9Q9TF 0 ├─11420 /bin/bash -l /usr/share/restraint/plugins/task_run.d/10_bash_login /usr/share/restraint/plugins/task_run.d/15_beakerlib /usr/share/restraint/plugins/task_run.d/20_unconf... ├─11442 /bin/sh /usr/share/restraint/plugins/run_plugins ├─11463 /bin/sh ./10_localwatchdog └─11465 rstrnt-report-result --no-plugins /10_localwatchdog WARN 0 Jan 17 10:56:33 ibm-p8-kvm-11-guest-07.virt.pnr.lab.eng.rdu2.redhat.com restraintd[1757]: gpg: Signature made Fri 17 Jan 2020 10:51:25 AM EST using DSA key ID AA7488BA Jan 17 10:56:33 ibm-p8-kvm-11-guest-07.virt.pnr.lab.eng.rdu2.redhat.com restraintd[1757]: gpg: Good signature from "bob-dsa-4096 <bob-dsa-4096>" Jan 17 10:56:33 ibm-p8-kvm-11-guest-07.virt.pnr.lab.eng.rdu2.redhat.com restraintd[1757]: :: [ 10:51:26 ] :: [ PASS ] :: Veryfing all Bob's dsa 4096 bit key signs (except detac..., got 0) Jan 17 10:56:33 ibm-p8-kvm-11-guest-07.virt.pnr.lab.eng.rdu2.redhat.com restraintd[1757]: :: [ 10:51:26 ] :: [ BEGIN ] :: Removing signatures :: actually running 'rm text.msg-si...-signed' Jan 17 10:56:33 ibm-p8-kvm-11-guest-07.virt.pnr.lab.eng.rdu2.redhat.com restraintd[1757]: :: [ 10:51:26 ] :: [ PASS ] :: Removing signatures (Expected 0, got 0) Jan 17 10:56:33 ibm-p8-kvm-11-guest-07.virt.pnr.lab.eng.rdu2.redhat.com restraintd[1757]: :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: Jan 17 10:56:33 ibm-p8-kvm-11-guest-07.virt.pnr.lab.eng.rdu2.redhat.com restraintd[1757]: :: Duration: 544s Jan 17 10:56:33 ibm-p8-kvm-11-guest-07.virt.pnr.lab.eng.rdu2.redhat.com restraintd[1757]: :: Assertions: 2157 good, 0 bad Jan 17 10:56:33 ibm-p8-kvm-11-guest-07.virt.pnr.lab.eng.rdu2.redhat.com restraintd[1757]: :: RESULT: PASS (Test) Jan 17 10:56:33 ibm-p8-kvm-11-guest-07.virt.pnr.lab.eng.rdu2.redhat.com restraintd[1757]: ** Test PASS Score:0 Hint: Some lines were ellipsized, use -l to show in full. [root@ibm-p8-kvm-11-guest-07 ~]# ps aux | grep rstrnt root 11359 0.0 0.0 157760 5888 ? Sl 10:51 0:00 /usr/bin/rstrnt-report-result --rhts Test PASS /tmp/tmp.7nwdD9Q9TF 0 root 11465 0.0 0.0 157760 5760 ? Sl 10:56 0:00 rstrnt-report-result --no-plugins /10_localwatchdog WARN 0 root 11709 0.0 0.0 111552 3200 pts/1 S+ 11:06 0:00 grep --color=auto rstrnt [root@ibm-p8-kvm-11-guest-07 ~]# date Fri Jan 17 11:06:53 EST 2020 Version-Release number of selected component (if applicable): How reproducible: Unsure Actual results: None. Restraint is just hanging there for better times Expected results: Killed/Retried/Reported Additional info: