Bug 1008860

I can reproduce with step1 in comment #0  (qemu-kvm-rhel7 & qemu-upstream)


Report a upstream bug to track this issue:
   https://bugs.launchpad.net/qemu/+bug/1253563

try to test with real egd daemon

* env setup: Install egd and lunch egd socket
   sudo yum install -y egd
   sudo egd.pl --debug-client --nofork localhost:8001

* test 1):
 I tried to write a C code to emulate egd client, the speed is about 140 kb/s

| char header[2];
| header[0] = 0x2;
| header[1] = 0x64;
|
| while (1) { 
|         send(sockfd, header, sizeof header, 0);
|         byte_count = recv(sockfd, buf, sizeof buf, 0);
|         count += byte_count;
| } 


*test 2) lunch qemu
 qemu ...  -chardev socket,host=localhost,port=8001,id=chr0 \
           -object rng-egd,chardev=chr0,id=rng0 \
           -device virtio-rng-pci,rng=rng0

The dd speed in guest is very slow, about 0.8 kb/s

In C code we send the new request to egd daemon immediately after the read().
When we use qemu, there is a bigger gap between two times read()

(In reply to Amos Kong from comment #4)
> try to test with real egd daemon
> 
> * env setup: Install egd and lunch egd socket
>    sudo yum install -y egd
>    sudo egd.pl --debug-client --nofork localhost:8001

correct a typo: above test used "--bottomless" option for egd.pl
 --bottomless: for testing, don't decrement entropy count

If I remove this option, both test 1 & test 2 is very very slow, almost 0 kb/s

====

  1) I will investigate why the speed of egd socket is very slow. it's the current-neck
  2) If the source speed is very fast, the bottle-neck is in the request queue implement, it also should be improved

I had posted a patch to upstream:
  http://lists.gnu.org/archive/html/qemu-devel/2013-12/msg01470.html

More discussion:
  http://marc.info/?l=qemu-devel&m=138932103503306&w=2


Using buffers and pre-reading entropy is not desirable.
The main reason is the slow source, using buffers and pre-reading is
just abuse the resource & sync read API in qemu.

> Moreover, reasonable guests won't request for a whole lot of random
> numbers in a short interval, so the theoretical performance problem
> we're seeing is just going to remain theoretical for well-behaved
> guests.

We always recommend the users to use rng-random backend. Only use
rng-egd backend when host uses a USB entropy device.


The bad speed is caused by data source, it seems not a real problem, 5 kB/s is enough for common usage.

Let's wait final decision in upstream, we might close this bug as WONTFIX.

Gerd,

This USB device is slow at providing random bits, but not if you read large chunks of data at once (say into a 256K buffer).
Just verifying. Could it be an issue with USB (I do not think so).
Amos, can you put the table of buf-size vs speed here.

Actually, Amos is using an emulator, so his timing doesn't mean much. The original issue is with the actual USB device.

> Just verifying. Could it be an issue with USB (I do not think so).

Agree.

> Amos, can you put the table of buf-size vs speed here.

Also number of requests per second please.

I'd expect the request overhead limits the data rate here.  Most likely the number of requests per second is pretty much constant, until the buffer size is so big that you hit the usb bus bandwidth limit instead.

(In reply to Gerd Hoffmann from comment #9)
> > Just verifying. Could it be an issue with USB (I do not think so).
> 
> Agree.
> 
> > Amos, can you put the table of buf-size vs speed here.
> 
> Also number of requests per second please.
> 
> I'd expect the request overhead limits the data rate here.  Most likely the
> number of requests per second is pretty much constant, until the buffer size
> is so big that you hit the usb bus bandwidth limit instead.

My solution is pre-reading, upstream is not happy with this. random data is too expensive.

My old test results doesn't relate with USB.

What I & QE will do is checking why the egd source is too slow (5 kb/s).

Thanks.

Further work:

* Amit said we can setup USB device to pass random data the /dev/random, then we can use rng-random backend. Amit, can you give us some detail? do we need to write some C code?

  # rngd --help
    Check and feed random data from hardware device to kernel entropy pool.
    -r, --rng-device=file      Kernel device used for random number input
                               (default: /dev/hwrng)

* Setup USB device to pass random data to egd daemon socket, let's if speed of egd backends is improved or not.

  I wrote a C program to read data from USB device and send the data out by a server socket. If it's support EGD protocal then we can directly treat as a EGD daemon socket.

(In reply to Amos Kong from comment #11)
> Further work:
> 
> * Amit said we can setup USB device to pass random data the /dev/random,
> then we can use rng-random backend. Amit, can you give us some detail? do we
> need to write some C code?

I hinted at this here as well: https://bugzilla.redhat.com/show_bug.cgi?id=1007664#c4

What you need to do is:

write a C program that reads entropy from the usb device.  Then, simply write the data that you just read into /dev/random.

This way, any program reading from /dev/random will get the random numbers which are generated by the entropy from the usb device.

(In reply to Amos Kong from comment #11)
> Further work:

> * Setup USB device to pass random data to egd daemon socket, let's if speed
> of egd backends is improved or not.
> 
>   I wrote a C program to read data from USB device and send the data out by
> a server socket. If it's support EGD protocal then we can directly treat as
> a EGD daemon socket.


@Gerd

I implement EGD protocol in my C program, we always get type 2 (header[0]) request from qemu, header[1] is the request size.

|randbyte = new char [255];
| while(1) {
|         char header [2];
|         read(accept_sock, header, 2);
|         printf("request size: %d\n", header[1]);
|       if ((qngStatus = QNG->RandBytes(randbyte, header[1])) != S_OK) { 
|               printf("Error: %s\n\n", QNG->StatusString());
|               return EXIT_FAILURE;
|       }       
|       send(accept_sock, randbyte, header[1], 0);
| }

# dd if=/dev/hwrng of=/dev/null count=12
12+0 records in
12+0 records out
6144 bytes (6.1 kB) copied, 56.2732 s, 0.1 kB/s

96 requests were sent, one request is fixed 64 kb.

QNG->RandBytes() is blocking read, it also takes too much time to response QEMU's request. The problem is same as we tested by egd daemon socket.

(In reply to Amos Kong from comment #13)

> I implement EGD protocol in my C program, we always get type 2 (header[0])
> request from qemu, header[1] is the request size.
> 
> |       if ((qngStatus = QNG->RandBytes(randbyte, header[1])) != S_OK) { 

What is this?  special library to access the usb rng?

> # dd if=/dev/hwrng of=/dev/null count=12

What is /dev/hwrng?  The usb device too?  Something else?

/me is a bit confused.

> QNG->RandBytes() is blocking read, it also takes too much time to response
> QEMU's request. The problem is same as we tested by egd daemon socket.

In any case I think the best way to deal with it is to feed /dev/random from the usb device as suggested by amit in comment 12.  That will (a) kill the blocking issue, (b) make the usb device work for everybody using /dev/randrom and (c) will mix the entropy from the usb device with the entropy from other sources, which is good from a security point of view (can you trust your usb device to generate really random numbers?).

And finally you can easily read big chunks at once without upsetting qemu upstream ;)

(In reply to Gerd Hoffmann from comment #14)
> (In reply to Amos Kong from comment #13)
> 
> > I implement EGD protocol in my C program, we always get type 2 (header[0])
> > request from qemu, header[1] is the request size.
> > 
> > |       if ((qngStatus = QNG->RandBytes(randbyte, header[1])) != S_OK) { 
> 
> What is this?  special library to access the usb rng?

Yes, it's libqwqng-1.3.5, we use it to access the usb rng.

> 
> > # dd if=/dev/hwrng of=/dev/null count=12
> 
> What is /dev/hwrng?  The usb device too?  Something else?

/dev/hwrng is the char device in guest, it's appear when we load virtio-rng module.
 
> /me is a bit confused.
> 
> > QNG->RandBytes() is blocking read, it also takes too much time to response
> > QEMU's request. The problem is same as we tested by egd daemon socket.
> 
> In any case I think the best way to deal with it is to feed /dev/random from
> the usb device as suggested by amit in comment 12.  That will (a) kill the
> blocking issue, (b) make the usb device work for everybody using
> /dev/randrom and (c) will mix the entropy from the usb device with the
> entropy from other sources, which is good from a security point of view

Ok, thanks

> (can you trust your usb device to generate really random numbers?).

We have a intern who is working to write some testcase to verify the data is really random.

> And finally you can easily read big chunks at once without upsetting qemu
> upstream ;)

(In reply to Amit Shah from comment #12)
> (In reply to Amos Kong from comment #11)
> > Further work:
> > 
> > * Amit said we can setup USB device to pass random data the /dev/random,
> > then we can use rng-random backend. Amit, can you give us some detail? do we
> > need to write some C code?
> 
> I hinted at this here as well:
> https://bugzilla.redhat.com/show_bug.cgi?id=1007664#c4
> 
> What you need to do is:
> 
> write a C program that reads entropy from the usb device.  Then, simply
> write the data that you just read into /dev/random.
> 
> This way, any program reading from /dev/random will get the random numbers
> which are generated by the entropy from the usb device.

I write a C program to write the data (which is read from USB rng) to /dev/random, but the speed isn't improved.


  # read data from USB and fill to buf[]
  int fd = open("/dev/random", O_WRONLY);
  while (1)
      printf("write() ==> %d\n", write(fd, buf, 64));

I did another test to verify if writing /dev/random will fill data to kernel pool.
1) test read speed of /dev/random
# dd if=/dev/random of=/dev/null 
^C0+44940 records in
10954+0 records out
5608448 bytes (5.6 MB) copied, 12.3284 s, 455 kB/s

2) test read speed of /dev/urandom
# dd if=/dev/urandom of=/dev/random
^C206122+0 records in
206121+0 records out
105533952 bytes (106 MB) copied, 8.17029 s, 12.9 MB/s

3) read data from urandom and write to random in background.
   then test the read speed of /dev/random
# dd if=/dev/urandom of=/dev/random &
# dd if=/dev/random of=/dev/null 
^C0+36065 records in
8524+0 records out
4364288 bytes (4.4 MB) copied, 14.6783 s, 297 kB/s

The read speed reduced (background process costs some system resource).
It's mean we can't add data to kernel entropy pool by write /dev/random ??

(In reply to Amos Kong from comment #16)
> (In reply to Amit Shah from comment #12)
> > (In reply to Amos Kong from comment #11)
> > > Further work:
> > > 
> > > * Amit said we can setup USB device to pass random data the /dev/random,
> > > then we can use rng-random backend. Amit, can you give us some detail? do we
> > > need to write some C code?
> > 
> > I hinted at this here as well:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1007664#c4
> > 
> > What you need to do is:
> > 
> > write a C program that reads entropy from the usb device.  Then, simply
> > write the data that you just read into /dev/random.
> > 
> > This way, any program reading from /dev/random will get the random numbers
> > which are generated by the entropy from the usb device.
> 
> I write a C program to write the data (which is read from USB rng) to
> /dev/random, but the speed isn't improved.

How did you measure?  A background process to read data from the usb device and write to /dev/random is the best bet.

In this case, you don't care about the speed, since this is a background process, and all requests for entropy only use /dev/random.

>   # read data from USB and fill to buf[]
>   int fd = open("/dev/random", O_WRONLY);
>   while (1)
>       printf("write() ==> %d\n", write(fd, buf, 64));
> 
> I did another test to verify if writing /dev/random will fill data to kernel
> pool.
> 1) test read speed of /dev/random
> # dd if=/dev/random of=/dev/null 
> ^C0+44940 records in
> 10954+0 records out
> 5608448 bytes (5.6 MB) copied, 12.3284 s, 455 kB/s
> 
> 2) test read speed of /dev/urandom
> # dd if=/dev/urandom of=/dev/random
> ^C206122+0 records in
> 206121+0 records out
> 105533952 bytes (106 MB) copied, 8.17029 s, 12.9 MB/s

Both are bogus, use this instead:

$ cat /proc/sys/kernel/random/entropy_avail

> 3) read data from urandom and write to random in background.
>    then test the read speed of /dev/random
> # dd if=/dev/urandom of=/dev/random &
> # dd if=/dev/random of=/dev/null 
> ^C0+36065 records in
> 8524+0 records out
> 4364288 bytes (4.4 MB) copied, 14.6783 s, 297 kB/s

Heh, please don't do this :-)

> It's mean we can't add data to kernel entropy pool by write /dev/random ??

Check the output of the entropy_avail file above.  That'll tell you whether the kernel entropy pool gets filled or not.

1) Adjust the wakeup threshold to 4000, we can clearly to see the effect of writing /dev/random
# echo 4000 > /proc/sys/kernel/random/read_wakeup_threshold

       The file read_wakeup_threshold contains the number of bits of entropy
       required for waking up processes that sleep waiting for entropy from
       /dev/random.  The default is 64.

2) checking entropy_avail by watch
# watch -n 0.2 cat /proc/sys/kernel/random/entropy_avail

3) execute c program (read data from USB dev and write data to /dev/random)

Result:
It's _very clear_ that the entropy_avail increases quicker when we execute C program.

Conclusion:
 So we write the data from USB device to /dev/random, then use the rng-random
 backend of QEMU. 
 Let's skip egd backend slow issue until we have other request to rely on it.

Close this bug as WONTFIX, we can re-open it when our new request is effected by  slow egd-backend.

*** Bug 915381 has been marked as a duplicate of this bug. ***