Bug 1657695 - system running incredibly slow, constant 10-50% iowait
Summary: system running incredibly slow, constant 10-50% iowait
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Fedora
Classification: Fedora
Component: kernel
Version: 29
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Kernel Maintainer List
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-10 10:02 UTC by Karel Volný
Modified: 2019-02-21 21:06 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-02-21 21:06:00 UTC


Attachments (Terms of Use)

Description Karel Volný 2018-12-10 10:02:33 UTC
Description of problem:
I've tried to install Fedora (KDE spin via Everything netinstall) on an USB stick.
I did the install from a running system in qemu, and everything went okay, it took about ~1,5 hour including downloading ~1.7 GiB packages and installing them.
After booting from the stick, on the same machine on real hardware (no more qemu), everything takes ages, and from time to time the system becomes unresponsive for a few seconds. Despite that I have 16 GB of RAM and the stick has only 8 GB, so it can be all cached and still there would be plenty of RAM available for apps, there's constant iowait.

Version-Release number of selected component (if applicable):
kernel-4.19.6-300.fc29.x86_64

How reproducible:
always?

Steps to Reproduce:
1. install Fedora on USB stick
2. boot from it ...

Actual results:
an example:

[root@localhost ~]# time dnf -y install hdparm
Poslední kontrola metadat: před 1:15:08, Po 10. prosince 2018, 09:23:29 CET.
Závislosti vyřešeny.
=============================================================================================================================================================================================================================================
 balíček                                                 Arch                                                    Verze                                                         Repozitář                                                Vel.
=============================================================================================================================================================================================================================================
Installing:
 hdparm                                                  x86_64                                                  9.56-1.fc29                                                   fedora                                                   91 k

Shrnutí transakce
=============================================================================================================================================================================================================================================
Instalovat  1 balíček

Celková velikost ke stažení: 91 k
Velikost po nainstalování: 192 k
Stahování balíčků:
hdparm-9.56-1.fc29.x86_64.rpm                                                                                                                                                                                 70 kB/s |  91 kB     00:01    
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Celkem                                                                                                                                                                                                        16 kB/s |  91 kB     00:05     
Spouštím kontrolu transakce
Kontrola transakce byla úspěšná
Probíhá test transakce
Test transakce byl úspěšný.
Transakce běží
  Příprava        :                                                                                                                                                                                                                      1/1 
  Installing      : hdparm-9.56-1.fc29.x86_64                                                                                                                                                                                            1/1 
  Běžící skriptlet: hdparm-9.56-1.fc29.x86_64                                                                                                                                                                                            1/1 
  Ověřuji         : hdparm-9.56-1.fc29.x86_64                                                                                                                                                                                            1/1 

Nainstalováno:
  hdparm-9.56-1.fc29.x86_64                                                                                                                                                                                                                  

Hotovo!

real    3m46,979s
user    0m2,616s
sys     0m0,298s

Expected results:
this should have been done under 3s realtime

Additional info:
[fedora@localhost ~]$ free
              total        used        free      shared  buff/cache   available
Mem:       16043120     1063376    13091956      440392     1887788    14279272
Swap:             0           0           0
[root@localhost ~]# hdparm -t /dev/sdb

/dev/sdb:
 Timing buffered disk reads:  78 MB in  3.02 seconds =  25.81 MB/sec
[root@localhost ~]# hdparm -T /dev/sdb

/dev/sdb:
 Timing cached reads:   19746 MB in  2.00 seconds = 9896.32 MB/sec

Comment 1 Karel Volný 2018-12-10 10:08:05 UTC
p.s. using 'noatime' doesn't help a slightest bit ... even if it would, there still would be a question why the system isn't slow in qemu when accessing the same underlying device

Comment 2 Steve 2018-12-11 20:58:01 UTC
(In reply to Karel Volný from comment #0)
...
> 1. install Fedora on USB stick
...

For completeness, could you post the details of your USB configuration when booted from the USB stick?

$ lsusb -t
$ lsusb -v

Comment 3 Steve 2018-12-11 21:27:14 UTC
(In reply to Karel Volný from comment #0)
...
> [root@localhost ~]# hdparm -t /dev/sdb
> 
> /dev/sdb:
>  Timing buffered disk reads:  78 MB in  3.02 seconds =  25.81 MB/sec
...

In a test with two USB sticks:

USB 2.0 stick in a USB 2.0 port:

# hdparm -t /dev/sdc

/dev/sdc:
 Timing buffered disk reads:  90 MB in  3.01 seconds =  29.87 MB/sec

USB 3.0 stick in a USB 3.0 port:

# hdparm -t /dev/sdd

/dev/sdd:
 Timing buffered disk reads: 716 MB in  3.01 seconds = 238.18 MB/sec

# lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
    |__ Port 5: Dev 2, If 0, Class=Mass Storage, Driver=usb-storage, 5000M <<<<< USB 3.0 stick
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/12p, 480M
    |__ Port 5: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 11: Dev 4, If 0, Class=Mass Storage, Driver=usb-storage, 480M <<<<< USB 2.0 stick
    |__ Port 12: Dev 3, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M

Tested on bare metal with:

kernel-4.19.6-200.fc28.x86_64
hdparm-9.56-1.fc28.x86_64

Comment 4 Karel Volný 2018-12-17 12:21:47 UTC
system booted from hard drive:

# hdparm -t /dev/sdb

/dev/sdb:
 Timing buffered disk reads:  76 MB in  3.04 seconds =  24.98 MB/sec

# lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
    |__ Port 5: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/15p, 480M
    |__ Port 5: Dev 2, If 0, Class=Chip/SmartCard, Driver=usbfs, 12M
    |__ Port 6: Dev 9, If 0, Class=Mass Storage, Driver=usb-storage, 480M
    |__ Port 9: Dev 3, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 4: Dev 5, If 0, Class=Hub, Driver=hub/3p, 480M
            |__ Port 1: Dev 7, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
            |__ Port 2: Dev 8, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 11: Dev 4, If 0, Class=Wireless, Driver=btusb, 12M
    |__ Port 11: Dev 4, If 1, Class=Wireless, Driver=btusb, 12M
    |__ Port 12: Dev 6, If 1, Class=Video, Driver=uvcvideo, 480M
    |__ Port 12: Dev 6, If 0, Class=Video, Driver=uvcvideo, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M

# lsusb -v -s 3:9

Bus 003 Device 009: ID 0930:6544 Toshiba Corp. TransMemory-Mini / Kingston DataTraveler 2.0 Stick
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x0930 Toshiba Corp.
  idProduct          0x6544 TransMemory-Mini / Kingston DataTraveler 2.0 Stick
  bcdDevice            1.00
  iManufacturer           1 Kingston
  iProduct                2 DataTraveler 2.0
  iSerial                 3 C8600088616CCE406A196425
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0020
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              200mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk-Only
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval             255
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval             255
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  bNumConfigurations      1
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x0000
  (Bus Powered)

Comment 5 Karel Volný 2018-12-17 13:00:44 UTC
system booted from the stick:

# lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 5000M
    |__ Port 5: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/15p, 480M
    |__ Port 5: Dev 2, If 0, Class=Chip/SmartCard, Driver=usbfs, 12M
    |__ Port 6: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 480M
    |__ Port 9: Dev 4, If 0, Class=Hub, Driver=hub/4p, 480M
        |__ Port 4: Dev 6, If 0, Class=Hub, Driver=hub/3p, 480M
            |__ Port 1: Dev 8, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
            |__ Port 2: Dev 9, If 0, Class=Human Interface Device, Driver=usbhid, 1.5M
    |__ Port 11: Dev 5, If 0, Class=Wireless, Driver=btusb, 12M
    |__ Port 11: Dev 5, If 1, Class=Wireless, Driver=btusb, 12M
    |__ Port 12: Dev 7, If 1, Class=Video, Driver=uvcvideo, 480M
    |__ Port 12: Dev 7, If 0, Class=Video, Driver=uvcvideo, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/8p, 480M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/3p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/6p, 480M

# lsusb -v -s 3:3

Bus 003 Device 003: ID 0930:6544 Toshiba Corp. TransMemory-Mini / Kingston DataTraveler 2.0 Stick
Device Descriptor:
  bLength                18
  bDescriptorType         1
  bcdUSB               2.00
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  idVendor           0x0930 Toshiba Corp.
  idProduct          0x6544 TransMemory-Mini / Kingston DataTraveler 2.0 Stick
  bcdDevice            1.00
  iManufacturer           1 Kingston
  iProduct                2 DataTraveler 2.0
  iSerial                 3 C8600088616CCE406A196425
  bNumConfigurations      1
  Configuration Descriptor:
    bLength                 9
    bDescriptorType         2
    wTotalLength       0x0020
    bNumInterfaces          1
    bConfigurationValue     1
    iConfiguration          0 
    bmAttributes         0x80
      (Bus Powered)
    MaxPower              200mA
    Interface Descriptor:
      bLength                 9
      bDescriptorType         4
      bInterfaceNumber        0
      bAlternateSetting       0
      bNumEndpoints           2
      bInterfaceClass         8 Mass Storage
      bInterfaceSubClass      6 SCSI
      bInterfaceProtocol     80 Bulk-Only
      iInterface              0 
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x81  EP 1 IN
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval             255
      Endpoint Descriptor:
        bLength                 7
        bDescriptorType         5
        bEndpointAddress     0x02  EP 2 OUT
        bmAttributes            2
          Transfer Type            Bulk
          Synch Type               None
          Usage Type               Data
        wMaxPacketSize     0x0200  1x 512 bytes
        bInterval             255
Device Qualifier (for other device speed):
  bLength                10
  bDescriptorType         6
  bcdUSB               2.00
  bDeviceClass            0 
  bDeviceSubClass         0 
  bDeviceProtocol         0 
  bMaxPacketSize0        64
  bNumConfigurations      1
can't get debug descriptor: Resource temporarily unavailable
Device Status:     0x0000
  (Bus Powered)

Comment 6 Steve 2018-12-30 06:32:09 UTC
> ... the stick has only 8 GB, ...
> ... Kingston DataTraveler 2.0 Stick
>   bcdUSB               2.00

That is a slow device because it is USB 2.0 and because the low-capacity sticks tend to be slower by design -- based on looking at specs.

For some devices, vendors don't specify speeds or are vague about them. Those are the devices to avoid for Linux applications.

I suggest that you look for a USB 3.0/3.1 stick with ~64 GB capacity. This is what I used for the USB 3.0 test in Comment 3:

Kingston DataTraveler Elite G2 64 GB (USB 3.1)

> ... so it can be all cached ...

IIUC, you *installed* to the device, so Linux doesn't cache the entire disk image. In contrast, a live image is cached in memory. Have you tried writing a live image to the USB stick?

Fedora-KDE-Live-x86_64-29-1.2.iso
https://spins.fedoraproject.org/kde/download/index.html

The recommended tool is Fedora Media Writer:
https://fedoramagazine.org/make-fedora-usb-stick/

# dnf install mediawriter

# dnf info mediawriter
...
Description  : A tool to write images of Fedora media to portable drives
             : like flash drives or memory cards.

Comment 7 Steve 2018-12-30 07:48:29 UTC
(In reply to Steve from comment #3)
...
> # hdparm -t /dev/sdc
...

BTW, hdparm only does reads, but writes are much slower than reads, so a realistic benchmark should do both.

A web search for "linux disk io benchmark" found "fio" and "bonnie++", which are available as Fedora packages. The same web search found various articles on the subject.

"gnome-disks" is a graphical application that can do benchmarking. The package is "gnome-disk-utility".

Comment 8 Steve 2018-12-31 08:57:10 UTC
> ... there's constant iowait.

Does the log show any errors?

$ journalctl -b

Comment 9 Justin M. Forbes 2019-01-29 16:13:03 UTC
*********** MASS BUG UPDATE **************

We apologize for the inconvenience.  There are a large number of bugs to go through and several of them have gone stale.  Due to this, we are doing a mass bug update across all of the Fedora 29 kernel bugs.

Fedora 29 has now been rebased to 4.20.5-200.fc29.  Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel.

If you experience different issues, please open a new bug report for those.

Comment 10 Justin M. Forbes 2019-02-21 21:06:00 UTC
*********** MASS BUG UPDATE **************
This bug is being closed with INSUFFICIENT_DATA as there has not been a response in 3 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.


Note You need to log in before you can comment on or make changes to this bug.