Daniel,

It might be useful to open individual bugs for each kworker (since the solution
is separate).

Also, for the runc issue... Note that for kworkers there is an expectation that 10us per 1ms division
will be sufficient (since we know that kworkers perform periodic, short jobs) but for userspace 
sufficient execution time might be larger: lack of progress might trigger timeouts up the stack, 
causing (for example) container restarts.

Great progress!

Yeah, I thought about opening two BZs, but if the memcg happens only after the end of the exclusive pod, it will not be a problem. So I mentioned it here as a "hey, if you hit this stack while the workload is running, please file a BZ."

The side effects of throttling kworkers will be felt by the threads waiting for it, obviously. That is why it is important to try to avoid them at all, as you are trying to achieve.

Patch posted upstream:

https://lore.kernel.org/linux-mm/20201117162805.GA274911@fuller.cnet/T/#u

Test application:

#include <sys/mman.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

void main(void)
{
        int ret, fd;
        void *buf = malloc(4096);

        fd = open("/proc/sys/vm/quiet_vmstat", O_RDWR);
        if (fd == -1) {
                perror("open /proc/sys/vm/quiet_vmstat");
                exit(0);
        }

        memset(buf, 1, 4096);
        ret = mlock(buf, 4096);
        if (ret) {
                perror("mlock");
                exit(0);
        }

        ret = write(fd, buf, 1);
        if (ret == -1) {
                perror("write");
                exit(0);
        }

        while (ret != 0)
                memset(buf, 0, 10);

}

Created attachment 1730547 [details]
patch against oslat to write to /proc/sys/vm/quiet_vmstat

Created attachment 1732633 [details]
testpmd patch to quiet vmstat before pkt fwd loop

Built but untested.

(In reply to Marcelo Tosatti from comment #7)
> Created attachment 1732633 [details]
> testpmd patch to quiet vmstat before pkt fwd loop
> 
> Built but untested.

Against dpdk.git.

One question, would something like this work as a wrapper too?

isolate.sh
#!/bin/sh
cat > /proc/sys/vm/quiet_vmstat &
exec "$@"



./isolate.sh oslat ....

(In reply to Martin Sivák from comment #12)
> One question, would something like this work as a wrapper too?
> 
> isolate.sh
> #!/bin/sh
> cat > /proc/sys/vm/quiet_vmstat &
> exec "$@"
> 
> 
> 
> ./isolate.sh oslat ....

Depends on the application, but for oslat, for example, it won't work.

So its better to suggest a proper change (which won't be a write to a file in /proc/, but
a prctl interface).

Anyway, for the current interface, please do it properly inside the application.

Bryan,

Is this a defect we can publish as a known issue?  It does impact OCP and the DU use case.

/KenY

This is clearly a feature and not a bug, marking the BZ appropriately.

Also dropping NEEDINFO request since question has been answered.