Bug 1008860 - virtio-rng-egd: get data rate <5k/sec from rng device inside guest
virtio-rng-egd: get data rate <5k/sec from rng device inside guest
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: qemu-kvm (Show other bugs)
7.0
x86_64 Linux
medium Severity medium
: rc
: ---
Assigned To: Amos Kong
Virtualization Bugs
:
: 915381 (view as bug list)
Depends On:
Blocks: VirtIORNG 916194
  Show dependency treegraph
 
Reported: 2013-09-17 04:39 EDT by Xu Han
Modified: 2015-05-24 20:07 EDT (History)
16 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2014-01-21 19:44:00 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Xu Han 2013-09-17 04:39:25 EDT
Description of problem:
get data rate <5k/sec from rng device inside guest by use rng-edg.
rng-random not hit this issue.

Version-Release number of selected component (if applicable):
qemu: qemu-kvm-rhev-1.5.3-3.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.boot guest with rng-edg (default rate | 1k/s | 100k/s)
# cat /dev/random | nc -l localhost 1024

#/usr/libexec/qemu-kvm ...
-chardev socket,host=localhost,port=1024,id=chr0 \
-object rng-egd,chardev=chr0,id=rng0 \
-device virtio-rng-pci,rng=rng0(,max-bytes=1024,period=1000 | ,max-bytes=102400,period=1000)


2.check rng output rate in guest
# dd if=/dev/hwrng of=/dev/null


Actual results:
2.default rate:
1321+1 records in
1321+0 records out
676352 bytes (676 kB) copied, 172.636 s, 3.9 kB/s

1k/s:
659+1 records in
659+0 records out
337408 bytes (337 kB) copied, 329.012 s, 1.0 kB/s

100k/s:
1510+1 records in
1510+0 records out
773120 bytes (773 kB) copied, 185.2 s, 4.2 kB/s


Expected results:


Additional info:
1.host rng output rate
# telnet localhost 1024 | dd if=/dev/stdin of=/dev/null 
42390+235935 records in
100806+1 records out
51612896 bytes (52 MB) copied, 128.431 s, 402 kB/s


2.use virtserialport instead rng-edg
2.1 boot guest with virtserialport
# cat /dev/random | nc -l localhost 1024
#/usr/libexec/qemu-kvm ...
-chardev socket,host=localhost,port=1024,id=channel0 \
-device virtio-serial-pci,id=virtio-serial0,max_ports=16 \
-device virtserialport,chardev=channel0,name=com.redhat.rhevm.vdsm,bus=virtio-serial0.0,id=port0

2.2 check virtserialport output rate
# dd if=/dev/vport2p1 of=/dev/null 
2623+782623 records in
196656+0 records out
100687872 bytes (101 MB) copied, 241.047 s, 418 kB/s
Comment 3 Amos Kong 2013-11-21 04:40:32 EST
I can reproduce with step1 in comment #0  (qemu-kvm-rhel7 & qemu-upstream)


Report a upstream bug to track this issue:
   https://bugs.launchpad.net/qemu/+bug/1253563
Comment 4 Amos Kong 2013-11-27 09:30:02 EST
try to test with real egd daemon

* env setup: Install egd and lunch egd socket
   sudo yum install -y egd
   sudo egd.pl --debug-client --nofork localhost:8001

* test 1):
 I tried to write a C code to emulate egd client, the speed is about 140 kb/s

| char header[2];
| header[0] = 0x2;
| header[1] = 0x64;
|
| while (1) { 
|         send(sockfd, header, sizeof header, 0);
|         byte_count = recv(sockfd, buf, sizeof buf, 0);
|         count += byte_count;
| } 


*test 2) lunch qemu
 qemu ...  -chardev socket,host=localhost,port=8001,id=chr0 \
           -object rng-egd,chardev=chr0,id=rng0 \
           -device virtio-rng-pci,rng=rng0

The dd speed in guest is very slow, about 0.8 kb/s

In C code we send the new request to egd daemon immediately after the read().
When we use qemu, there is a bigger gap between two times read()
Comment 5 Amos Kong 2013-11-27 09:48:25 EST
(In reply to Amos Kong from comment #4)
> try to test with real egd daemon
> 
> * env setup: Install egd and lunch egd socket
>    sudo yum install -y egd
>    sudo egd.pl --debug-client --nofork localhost:8001

correct a typo: above test used "--bottomless" option for egd.pl
 --bottomless: for testing, don't decrement entropy count

If I remove this option, both test 1 & test 2 is very very slow, almost 0 kb/s

====

  1) I will investigate why the speed of egd socket is very slow. it's the current-neck
  2) If the source speed is very fast, the bottle-neck is in the request queue implement, it also should be improved
Comment 6 Amos Kong 2014-01-09 21:41:51 EST
I had posted a patch to upstream:
  http://lists.gnu.org/archive/html/qemu-devel/2013-12/msg01470.html

More discussion:
  http://marc.info/?l=qemu-devel&m=138932103503306&w=2


Using buffers and pre-reading entropy is not desirable.
The main reason is the slow source, using buffers and pre-reading is
just abuse the resource & sync read API in qemu.

> Moreover, reasonable guests won't request for a whole lot of random
> numbers in a short interval, so the theoretical performance problem
> we're seeing is just going to remain theoretical for well-behaved
> guests.

We always recommend the users to use rng-random backend. Only use
rng-egd backend when host uses a USB entropy device.


The bad speed is caused by data source, it seems not a real problem, 5 kB/s is enough for common usage.

Let's wait final decision in upstream, we might close this bug as WONTFIX.
Comment 7 Ronen Hod 2014-01-13 03:56:49 EST
Gerd,

This USB device is slow at providing random bits, but not if you read large chunks of data at once (say into a 256K buffer).
Just verifying. Could it be an issue with USB (I do not think so).
Amos, can you put the table of buf-size vs speed here.
Comment 8 Ronen Hod 2014-01-13 04:25:40 EST
Actually, Amos is using an emulator, so his timing doesn't mean much. The original issue is with the actual USB device.
Comment 9 Gerd Hoffmann 2014-01-13 06:35:49 EST
> Just verifying. Could it be an issue with USB (I do not think so).

Agree.

> Amos, can you put the table of buf-size vs speed here.

Also number of requests per second please.

I'd expect the request overhead limits the data rate here.  Most likely the number of requests per second is pretty much constant, until the buffer size is so big that you hit the usb bus bandwidth limit instead.
Comment 10 Amos Kong 2014-01-14 00:55:16 EST
(In reply to Gerd Hoffmann from comment #9)
> > Just verifying. Could it be an issue with USB (I do not think so).
> 
> Agree.
> 
> > Amos, can you put the table of buf-size vs speed here.
> 
> Also number of requests per second please.
> 
> I'd expect the request overhead limits the data rate here.  Most likely the
> number of requests per second is pretty much constant, until the buffer size
> is so big that you hit the usb bus bandwidth limit instead.

My solution is pre-reading, upstream is not happy with this. random data is too expensive.

My old test results doesn't relate with USB.

What I & QE will do is checking why the egd source is too slow (5 kb/s).

Thanks.
Comment 11 Amos Kong 2014-01-14 01:14:48 EST
Further work:

* Amit said we can setup USB device to pass random data the /dev/random, then we can use rng-random backend. Amit, can you give us some detail? do we need to write some C code?

  # rngd --help
    Check and feed random data from hardware device to kernel entropy pool.
    -r, --rng-device=file      Kernel device used for random number input
                               (default: /dev/hwrng)

* Setup USB device to pass random data to egd daemon socket, let's if speed of egd backends is improved or not.

  I wrote a C program to read data from USB device and send the data out by a server socket. If it's support EGD protocal then we can directly treat as a EGD daemon socket.
Comment 12 Amit Shah 2014-01-14 01:25:25 EST
(In reply to Amos Kong from comment #11)
> Further work:
> 
> * Amit said we can setup USB device to pass random data the /dev/random,
> then we can use rng-random backend. Amit, can you give us some detail? do we
> need to write some C code?

I hinted at this here as well: https://bugzilla.redhat.com/show_bug.cgi?id=1007664#c4

What you need to do is:

write a C program that reads entropy from the usb device.  Then, simply write the data that you just read into /dev/random.

This way, any program reading from /dev/random will get the random numbers which are generated by the entropy from the usb device.
Comment 13 Amos Kong 2014-01-14 02:55:00 EST
(In reply to Amos Kong from comment #11)
> Further work:

> * Setup USB device to pass random data to egd daemon socket, let's if speed
> of egd backends is improved or not.
> 
>   I wrote a C program to read data from USB device and send the data out by
> a server socket. If it's support EGD protocal then we can directly treat as
> a EGD daemon socket.


@Gerd

I implement EGD protocol in my C program, we always get type 2 (header[0]) request from qemu, header[1] is the request size.

|randbyte = new char [255];
| while(1) {
|         char header [2];
|         read(accept_sock, header, 2);
|         printf("request size: %d\n", header[1]);
|       if ((qngStatus = QNG->RandBytes(randbyte, header[1])) != S_OK) { 
|               printf("Error: %s\n\n", QNG->StatusString());
|               return EXIT_FAILURE;
|       }       
|       send(accept_sock, randbyte, header[1], 0);
| }

# dd if=/dev/hwrng of=/dev/null count=12
12+0 records in
12+0 records out
6144 bytes (6.1 kB) copied, 56.2732 s, 0.1 kB/s

96 requests were sent, one request is fixed 64 kb.

QNG->RandBytes() is blocking read, it also takes too much time to response QEMU's request. The problem is same as we tested by egd daemon socket.
Comment 14 Gerd Hoffmann 2014-01-15 03:34:02 EST
(In reply to Amos Kong from comment #13)

> I implement EGD protocol in my C program, we always get type 2 (header[0])
> request from qemu, header[1] is the request size.
> 
> |       if ((qngStatus = QNG->RandBytes(randbyte, header[1])) != S_OK) { 

What is this?  special library to access the usb rng?

> # dd if=/dev/hwrng of=/dev/null count=12

What is /dev/hwrng?  The usb device too?  Something else?

/me is a bit confused.

> QNG->RandBytes() is blocking read, it also takes too much time to response
> QEMU's request. The problem is same as we tested by egd daemon socket.

In any case I think the best way to deal with it is to feed /dev/random from the usb device as suggested by amit in comment 12.  That will (a) kill the blocking issue, (b) make the usb device work for everybody using /dev/randrom and (c) will mix the entropy from the usb device with the entropy from other sources, which is good from a security point of view (can you trust your usb device to generate really random numbers?).

And finally you can easily read big chunks at once without upsetting qemu upstream ;)
Comment 15 Amos Kong 2014-01-15 03:59:09 EST
(In reply to Gerd Hoffmann from comment #14)
> (In reply to Amos Kong from comment #13)
> 
> > I implement EGD protocol in my C program, we always get type 2 (header[0])
> > request from qemu, header[1] is the request size.
> > 
> > |       if ((qngStatus = QNG->RandBytes(randbyte, header[1])) != S_OK) { 
> 
> What is this?  special library to access the usb rng?

Yes, it's libqwqng-1.3.5, we use it to access the usb rng.

> 
> > # dd if=/dev/hwrng of=/dev/null count=12
> 
> What is /dev/hwrng?  The usb device too?  Something else?

/dev/hwrng is the char device in guest, it's appear when we load virtio-rng module.
 
> /me is a bit confused.
> 
> > QNG->RandBytes() is blocking read, it also takes too much time to response
> > QEMU's request. The problem is same as we tested by egd daemon socket.
> 
> In any case I think the best way to deal with it is to feed /dev/random from
> the usb device as suggested by amit in comment 12.  That will (a) kill the
> blocking issue, (b) make the usb device work for everybody using
> /dev/randrom and (c) will mix the entropy from the usb device with the
> entropy from other sources, which is good from a security point of view

Ok, thanks

> (can you trust your usb device to generate really random numbers?).

We have a intern who is working to write some testcase to verify the data is really random.

> And finally you can easily read big chunks at once without upsetting qemu
> upstream ;)
Comment 16 Amos Kong 2014-01-15 04:34:09 EST
(In reply to Amit Shah from comment #12)
> (In reply to Amos Kong from comment #11)
> > Further work:
> > 
> > * Amit said we can setup USB device to pass random data the /dev/random,
> > then we can use rng-random backend. Amit, can you give us some detail? do we
> > need to write some C code?
> 
> I hinted at this here as well:
> https://bugzilla.redhat.com/show_bug.cgi?id=1007664#c4
> 
> What you need to do is:
> 
> write a C program that reads entropy from the usb device.  Then, simply
> write the data that you just read into /dev/random.
> 
> This way, any program reading from /dev/random will get the random numbers
> which are generated by the entropy from the usb device.

I write a C program to write the data (which is read from USB rng) to /dev/random, but the speed isn't improved.


  # read data from USB and fill to buf[]
  int fd = open("/dev/random", O_WRONLY);
  while (1)
      printf("write() ==> %d\n", write(fd, buf, 64));

I did another test to verify if writing /dev/random will fill data to kernel pool.
1) test read speed of /dev/random
# dd if=/dev/random of=/dev/null 
^C0+44940 records in
10954+0 records out
5608448 bytes (5.6 MB) copied, 12.3284 s, 455 kB/s

2) test read speed of /dev/urandom
# dd if=/dev/urandom of=/dev/random
^C206122+0 records in
206121+0 records out
105533952 bytes (106 MB) copied, 8.17029 s, 12.9 MB/s

3) read data from urandom and write to random in background.
   then test the read speed of /dev/random
# dd if=/dev/urandom of=/dev/random &
# dd if=/dev/random of=/dev/null 
^C0+36065 records in
8524+0 records out
4364288 bytes (4.4 MB) copied, 14.6783 s, 297 kB/s

The read speed reduced (background process costs some system resource).
It's mean we can't add data to kernel entropy pool by write /dev/random ??
Comment 17 Amit Shah 2014-01-15 11:08:53 EST
(In reply to Amos Kong from comment #16)
> (In reply to Amit Shah from comment #12)
> > (In reply to Amos Kong from comment #11)
> > > Further work:
> > > 
> > > * Amit said we can setup USB device to pass random data the /dev/random,
> > > then we can use rng-random backend. Amit, can you give us some detail? do we
> > > need to write some C code?
> > 
> > I hinted at this here as well:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1007664#c4
> > 
> > What you need to do is:
> > 
> > write a C program that reads entropy from the usb device.  Then, simply
> > write the data that you just read into /dev/random.
> > 
> > This way, any program reading from /dev/random will get the random numbers
> > which are generated by the entropy from the usb device.
> 
> I write a C program to write the data (which is read from USB rng) to
> /dev/random, but the speed isn't improved.

How did you measure?  A background process to read data from the usb device and write to /dev/random is the best bet.

In this case, you don't care about the speed, since this is a background process, and all requests for entropy only use /dev/random.

>   # read data from USB and fill to buf[]
>   int fd = open("/dev/random", O_WRONLY);
>   while (1)
>       printf("write() ==> %d\n", write(fd, buf, 64));
> 
> I did another test to verify if writing /dev/random will fill data to kernel
> pool.
> 1) test read speed of /dev/random
> # dd if=/dev/random of=/dev/null 
> ^C0+44940 records in
> 10954+0 records out
> 5608448 bytes (5.6 MB) copied, 12.3284 s, 455 kB/s
> 
> 2) test read speed of /dev/urandom
> # dd if=/dev/urandom of=/dev/random
> ^C206122+0 records in
> 206121+0 records out
> 105533952 bytes (106 MB) copied, 8.17029 s, 12.9 MB/s

Both are bogus, use this instead:

$ cat /proc/sys/kernel/random/entropy_avail

> 3) read data from urandom and write to random in background.
>    then test the read speed of /dev/random
> # dd if=/dev/urandom of=/dev/random &
> # dd if=/dev/random of=/dev/null 
> ^C0+36065 records in
> 8524+0 records out
> 4364288 bytes (4.4 MB) copied, 14.6783 s, 297 kB/s

Heh, please don't do this :-)

> It's mean we can't add data to kernel entropy pool by write /dev/random ??

Check the output of the entropy_avail file above.  That'll tell you whether the kernel entropy pool gets filled or not.
Comment 18 Amos Kong 2014-01-16 04:44:38 EST
1) Adjust the wakeup threshold to 4000, we can clearly to see the effect of writing /dev/random
# echo 4000 > /proc/sys/kernel/random/read_wakeup_threshold

       The file read_wakeup_threshold contains the number of bits of entropy
       required for waking up processes that sleep waiting for entropy from
       /dev/random.  The default is 64.

2) checking entropy_avail by watch
# watch -n 0.2 cat /proc/sys/kernel/random/entropy_avail

3) execute c program (read data from USB dev and write data to /dev/random)

Result:
It's _very clear_ that the entropy_avail increases quicker when we execute C program.

Conclusion:
 So we write the data from USB device to /dev/random, then use the rng-random
 backend of QEMU. 
 Let's skip egd backend slow issue until we have other request to rely on it.
Comment 19 Amos Kong 2014-01-21 19:44:00 EST
Close this bug as WONTFIX, we can re-open it when our new request is effected by  slow egd-backend.
Comment 20 Amos Kong 2014-07-11 12:16:03 EDT
*** Bug 915381 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.