Bug 1770599

Summary: os-prober very slow - taking up to 20 minutes
Product: [Fedora] Fedora Reporter: ghborrmann
Component: os-proberAssignee: Hedayat Vatankhah <hedayatv>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: low Docs Contact:
Priority: unspecified    
Version: 33CC: agk, bugzilla.redhat.com, gmarr, hedayatv, martinrsssf, massi.ergosum, rootkit85
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard: AcceptedFreezeException
Fixed In Version: os-prober-1.77-6.fc33 os-prober-1.77-6.fc32 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-19 18:12:42 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1782773    
Bug Blocks: 1766778    

Description ghborrmann 2019-11-10 20:28:55 UTC
Description of problem:
Execution of grub2-mkconfig takes a much longer time on Fedora 31 compared to Fedora 30.  The time appears to be mainly due to os-prober.

Version-Release number of selected component (if applicable):
1.77-3.fc31

How reproducible:
Execution is always slow, although somewhat variable in time.


Steps to Reproduce:
1.Run grub2-mkconfig


Actual results:
So far I have seen execution times from 10 to almost 20 minutes.

Expected results:
Execution time should be no more than several minutes.

Additional info:
My system:
  sda - 12 partitions, with one Windows system, one Fedora, and one Arch Linux
  sdb - 15 partitions, with five Fedora distributions
  sdc - 4 data partitions
  sdd - 2 data partitions

Comment 1 Matteo Croce 2020-06-02 20:36:06 UTC
I confirm this on FC32 also.

This is what happens while generating a grub config:


   2558 pts/0    S+     0:00  \_ /usr/bin/sh /sbin/grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
   2958 pts/0    S+     0:00  |   \_ /usr/bin/sh /etc/grub.d/30_os-prober
   2964 pts/0    S+     0:00  |       \_ /usr/bin/sh /etc/grub.d/30_os-prober
   2965 pts/0    S+     0:00  |           \_ /usr/bin/sh /bin/os-prober
   3017 pts/0    S+     0:00  |           |   \_ /usr/bin/sh /bin/os-prober
   3018 pts/0    S+     0:00  |           |       \_ /usr/bin/sh /bin/os-prober
   3104 pts/0    S+     0:00  |           |       |   \_ /usr/bin/sh /usr/libexec/os-probes/50mounted-tests /dev/sda2
   3149 pts/0    S+     0:00  |           |       |       \_ /usr/bin/sh /usr/libexec/os-probes/mounted/90linux-distro /dev/sda2 /var/lib/os-prober/mount ext2
   3151 pts/0    D+     0:00  |           |       |           \_ /usr/bin/sh /usr/libexec/os-probes/mounted/90linux-distro /dev/sda2 /var/lib/os-prober/mount ext2
   3019 pts/0    S+     0:00  |           |       \_ logger
   2966 pts/0    S+     0:00  |           \_ tr   ^
   2967 pts/0    S+     0:00  |           \_ paste -s -d
   3112 ?        Rs     3:12 grub2-mount /dev/sda2 /var/lib/os-prober/mount

# strace -p 3112
strace: Process 3112 attached
lseek(4, 64466518016, SEEK_SET)         = 64466518016
read(4, "../../../../usr/lib64/qt5/plugin"..., 32768) = 32768
lseek(4, 64466845696, SEEK_SET)         = 64466845696
read(4, "\247\240<\0\f\0\1\2.\0\0\0\7\0<\0\f\0\2\2..\0\0\250\240<\0\350\17\21\2"..., 32768) = 32768
lseek(4, 65967783936, SEEK_SET)         = 65967783936
read(4, "\0\360\17\0\0\0\0\0\0\360\17\0\0\0\0\0\240\343\3\0\0\0\0\0000\342\3\0\0\0\0\0"..., 32768) = 32768
lseek(4, 375809245184, SEEK_SET)        = 375809245184
read(4, "\344\212u\227\225\34\222A\365\347\342u\212\243D1\0\0001D\3024\367C;\334\7\261\213\247\6\366"..., 32768) = 32768
lseek(4, 375809572864, SEEK_SET)        = 375809572864
read(4, "\255>\24\336\20 \320\233H1#`\232Ae\363\250\236\5\246\244\237\20\204\5\2567\341\342\4\252\343"..., 32768) = 32768
lseek(4, 64466518016, SEEK_SET)         = 64466518016
read(4, "../../../../usr/lib64/qt5/plugin"..., 32768) = 32768

# ll /proc/3112/fd/
total 0
lrwx------. 1 root root 64  2 giu 22.10 0 -> /dev/null
lrwx------. 1 root root 64  2 giu 22.10 1 -> /dev/null
lrwx------. 1 root root 64  2 giu 22.10 2 -> /dev/null
l-wx------. 1 root root 64  2 giu 22.10 3 -> 'pipe:[57869]'
lr-x------. 1 root root 64  2 giu 22.10 4 -> /dev/sda2
lrwx------. 1 root root 64  2 giu 22.10 5 -> /dev/fuse
l-wx------. 1 root root 64  2 giu 22.10 9 -> 'pipe:[55237]'

Comment 2 Hedayat Vatankhah 2020-06-10 20:06:22 UTC
Yes, it seems that grub2-mount mounted disks are very slow on some operations. I'll try to fix it soon. I've seen some patches for other distros; it's just a matter of time for me to fix.

Comment 3 Martin 2020-06-28 02:09:00 UTC
Looks like /usr/libexec/os-probes/mounted/90linux-distro is looking at whole bunch of files in /lib*/

When I comment that line and just exit 1 instead of unknown linux, grub2-mkconfig finishes fairly quickly.

Comment 4 Hedayat Vatankhah 2020-07-02 14:10:08 UTC
Yes, that's what I referred to in my previous comment.

Comment 5 Ben Cotton 2020-08-11 15:29:55 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 33 development cycle.
Changing version to 33.

Comment 6 Hedayat Vatankhah 2020-10-10 10:24:57 UTC
This bug actually depends on #1782773, but I'll try to provide a workaround very soon.

Comment 7 Fedora Blocker Bugs Application 2020-10-10 11:27:05 UTC
Proposed as a Freeze Exception for 33-final by Fedora user hedayat using the blocker tracking app because:

 os-prober runs during boot loader installation. Currently, it takes a very long time (could by more than 5-10 minutes for each linux filesystem) on some systems (systems which have a linux distro in another partition where the new system is being installed). With this small fix, it'll run much faster.

Comment 8 Fedora Update System 2020-10-10 11:29:08 UTC
FEDORA-2020-700e58867d has been submitted as an update to Fedora 32. https://bodhi.fedoraproject.org/updates/FEDORA-2020-700e58867d

Comment 9 Fedora Update System 2020-10-10 11:29:09 UTC
FEDORA-2020-93c02b1d43 has been submitted as an update to Fedora 33. https://bodhi.fedoraproject.org/updates/FEDORA-2020-93c02b1d43

Comment 10 Fedora Update System 2020-10-10 17:47:00 UTC
FEDORA-2020-93c02b1d43 has been pushed to the Fedora 33 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-93c02b1d43`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-93c02b1d43

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 11 Fedora Update System 2020-10-10 23:01:12 UTC
FEDORA-2020-700e58867d has been pushed to the Fedora 32 testing repository.
In short time you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2020-700e58867d`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2020-700e58867d

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 12 Geoffrey Marr 2020-10-12 19:36:05 UTC
Discussed during the 2020-10-12 blocker review meeting: [0]

The decision to classify this bug as an "AcceptedFreezeException (Final)" was made as it is a noticeable issue that cannot be fixed with an update.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2020-10-12/f33-blocker-review.2020-10-12-16.00.txt

Comment 13 Fedora Update System 2020-10-19 18:12:42 UTC
FEDORA-2020-93c02b1d43 has been pushed to the Fedora 33 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 14 Fedora Update System 2020-10-25 01:19:21 UTC
FEDORA-2020-700e58867d has been pushed to the Fedora 32 stable repository.
If problem still persists, please make note of it in this bug report.