1239170 – After update stuck in cycle "fatal error: token too large, exceeds YYLMAX" followed by "alloc magic broken at ...\nAborted." and reboot

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1239170 - After update stuck in cycle "fatal error: token too large, exceeds YYLMAX" followed by "alloc magic broken at ...\nAborted." and reboot

Summary: After update stuck in cycle "fatal error: token too large, exceeds YYLMAX" fo...

Keywords:
Status:	CLOSED WONTFIX
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	grubby
Sub Component:
Version:	7.2
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Peter Jones
QA Contact:	Release Test Team
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2015-07-03 21:24 UTC by Marian Csontos
Modified:	2020-12-15 07:35 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2020-12-15 07:35:14 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)
grub files (6.49 KB, application/octet-stream) 2015-07-13 13:32 UTC, Marian Csontos	no flags	Details
View All

Description Marian Csontos 2015-07-03 21:24:36 UTC

Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 2 Marian Csontos 2015-07-03 22:44:44 UTC

Have updated a RHEL-7.2-20150625.n.0 to RHEL-7.2-20150630.n.3.

After the update machine is stuck in cycle "fatal error: token too large, exceeds YYLMAX" followed by "alloc magic broken at ...\nAborted." and reboot.

I have got first few megs from the disc and the content of /boot.

Is there something else you will need from the machine?

Comment 3 Marian Csontos 2015-07-03 22:46:35 UTC

If reproducible this might require a lot of work fixing updated systems.

Comment 4 Marian Csontos 2015-07-03 23:06:10 UTC

And the full error is:

    alloc magic is broken at 0x7fd732c0: 3832342d

Little endian: "-428" ?

There is a `-428` string in UUID:

root=UUID=2d4978b3-b450-4285-bc86-fa9fbf2d9a26

Comment 5 Peter Jones 2015-07-06 19:36:29 UTC

Well, there's no grub2 change between these two - so, we definitely need the config file from after the update, at the very least.  The config file from before the update will also help, if possible.

Comment 6 Marian Csontos 2015-07-13 13:17:14 UTC

At the end it was caused by corrupted grub.cfg which had a line like this:

    set root='UUID=2d4978b3-b450-4285-bc86-fa9fbf2d9a26'=2d4978b3-b450-4285-bc86-fa9fbf2d9a26'=2d4978b3-b450-4285-bc86-fa9fbf2d9a26...

where the `'=2d4978b3-b450-4285-bc86-fa9fbf2d9a26` part was repeated 20k times.

I have no idea how this corruption happened.

The versions of packages installed at update time:

Jan 23 15:49:32 Updated: grubby-8.28-11.el7.x86_64
Jan 23 15:49:51 Updated: 1:grub2-tools-2.02-0.16.el7.x86_64
Jan 23 15:50:38 Updated: 1:grub2-2.02-0.16.el7.x86_64

Jun 02 15:31:22 Updated: dracut-033-241.el7_1.3.x86_64

Looking into /var/log/messages there are traces of grubby killed by OOM killer:

Jul  3 22:48:59 bot-rhel7-x86-64 dracut: *** Creating image file done ***
Jul  3 22:49:30 bot-rhel7-x86-64 kernel: grubby invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Jul  3 22:49:30 bot-rhel7-x86-64 kernel: grubby cpuset=/ mems_allowed=0
Jul  3 22:49:30 bot-rhel7-x86-64 kernel: CPU: 1 PID: 8689 Comm: grubby Not tainted 3.10.0-229.4.2.el7.x86_64 #1
Jul  3 22:49:30 bot-rhel7-x86-64 kernel: Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
Jul  3 22:49:30 bot-rhel7-x86-64 kernel: ffff88007a6ea220 0000000016a310a1 ffff88000ba3b950 ffffffff816042d6
Jul  3 22:49:30 bot-rhel7-x86-64 kernel: ffff88000ba3b9e0 ffffffff815ff29f ffff8800572f2440 ffff8800572f2458
Jul  3 22:49:30 bot-rhel7-x86-64 kernel: 0000000000000202 ffff88007a6ea220 ffff88000ba3b9c8 ffffffff8111786f
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: Call Trace:
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff816042d6>] dump_stack+0x19/0x1b
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff815ff29f>] dump_header+0x8e/0x214
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff8111786f>] ? delayacct_end+0x8f/0xb0
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff8115a0ce>] oom_kill_process+0x24e/0x3b0
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff81159c36>] ? find_lock_task_mm+0x56/0xc0
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff8107bd0e>] ? has_capability_noaudit+0x1e/0x30
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff8115a8f6>] out_of_memory+0x4b6/0x4f0
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff81160ab5>] __alloc_pages_nodemask+0xa95/0xb90
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff8119f029>] alloc_pages_current+0xa9/0x170
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff81156917>] __page_cache_alloc+0x97/0xc0
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff81158c08>] filemap_fault+0x188/0x430
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff8117e27e>] __do_fault+0x7e/0x510
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff811825d5>] handle_mm_fault+0x3d5/0xd60
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff8160f5a6>] __do_page_fault+0x156/0x520
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff81189725>] ? do_mmap_pgoff+0x305/0x3c0
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff812693ba>] ? security_mmap_file+0x2a/0x90
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff81173be9>] ? vm_mmap_pgoff+0xb9/0xe0
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff8160f98a>] do_page_fault+0x1a/0x70
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff8160c725>] ? do_device_not_available+0x35/0x60
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [<ffffffff8160bbc8>] page_fault+0x28/0x30
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: Mem-Info:
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: Node 0 DMA per-cpu:
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: CPU    1: hi:    0, btch:   1 usd:   0
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: Node 0 DMA32 per-cpu:
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: CPU    0: hi:  186, btch:  31 usd: 161
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: CPU    1: hi:  186, btch:  31 usd:  88
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: active_anon:237700 inactive_anon:79541 isolated_anon:0
 active_file:1 inactive_file:0 isolated_file:0
 unevictable:0 dirty:0 writeback:0 unstable:0
 free:13257 slab_reclaimable:12090 slab_unreclaimable:11738
 mapped:1 shmem:0 pagetables:2036 bounce:0
 free_cma:0
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: Node 0 DMA free:8276kB min:348kB low:432kB high:520kB active_anon:1644kB inactive_anon:1744kB active_file:4kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15976kB managed:15892kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:376kB slab_unreclaimable:324kB kernel_stack:32kB pagetables:64kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:398 all_unreclaimable? yes
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: lowmem_reserve[]: 0 1984 1984 1984
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: Node 0 DMA32 free:44752kB min:44704kB low:55880kB high:67056kB active_anon:949156kB inactive_anon:316420kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2080756kB managed:2033384kB mlocked:0kB dirty:0kB writeback:0kB mapped:4kB shmem:0kB slab_reclaimable:47984kB slab_unreclaimable:46628kB kernel_stack:1584kB pagetables:8080kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:47 all_unreclaimable? yes
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: lowmem_reserve[]: 0 0 0 0
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: Node 0 DMA: 43*4kB (UEM) 19*8kB (UE) 23*16kB (UEM) 19*32kB (UE) 9*64kB (UE) 8*128kB (UEM) 5*256kB (UE) 2*512kB (U) 1*1024kB (U) 1*2048kB (R) 0*4096kB = 8276kB
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: Node 0 DMA32: 8619*4kB (UEM) 770*8kB (UEM) 3*16kB (U) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 44780kB
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: 1177 total pagecache pages
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: 1140 pages in swap cache
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: Swap cache stats: add 272535, delete 271395, find 972705/973576
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: Free swap  = 0kB
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: Total swap = 1048572kB
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: 524183 pages RAM
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: 0 pages HighMem/MovableOnly
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: 11864 pages reserved
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Jul  3 22:49:31 bot-rhel7-x86-64 kernel: [  323]     0   323    36893       13      75     1225             0 systemd-journal
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [  401]     0   401    29185        0      25      105         -1000 auditd
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [  423]    81   423     6708      109      19       62          -900 dbus-daemon
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [  424]     0   424    85372       64      89      161             0 rsyslogd
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [  425]     0   425     6598       38      17       52             0 systemd-logind
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [  427]     0   427   114696      226      74      322             0 NetworkManager
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [  430]    70   430     6995       25      19       47             0 avahi-daemon
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [  436]    70   436     6995        5      18       54             0 avahi-daemon
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [  450]   999   450   130711       64      50     2355             0 polkitd
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [  468]     0   468    27507        1      11       32             0 agetty
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [  469]     0   469    27507        1      10       31             0 agetty
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [  474]     0   474    34670        0      68      309             0 sshd
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [  476]     0   476    28877       16      13      124             0 bash
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [17132]     0 17132    37898        0      28      747             0 vim
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [ 1574]     0  1574    34670       35      69      273             0 sshd
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [ 1589]     0  1589    28877        1      12      144             0 bash
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [18238]     0 18238    45501      231      43      432             0 mc
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [18240]     0 18240    28878       75      12       73             0 bash
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [25281]     0 25281    37861        0      29      663             0 vim
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [25359]     0 25359   161115     5421     235    52344             0 yum
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [25484]     0 25484    31593        1      19      166             0 crond
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [25545]     0 25545    10777        1      22      135         -1000 systemd-udevd
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [25847]     0 25847    20636        0      45      212         -1000 sshd
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [ 1118]     0  1118    28283        1      12       50             0 sh
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [ 1137]     0  1137    28349        2      12       99             0 new-kernel-pkg
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [ 8688]     0  8688    28349        6      11      106             0 new-kernel-pkg
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: [ 8689]     0  8689   493793   309647     971   181391             0 grubby
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: Out of memory: Kill process 8689 (grubby) score 616 or sacrifice child
Jul  3 22:49:32 bot-rhel7-x86-64 kernel: Killed process 8689 (grubby) total-vm:1975172kB, anon-rss:1238588kB, file-rss:0kB

Comment 7 Marian Csontos 2015-07-13 13:25:53 UTC

The kernel being installed:

Jul 03 22:47:17 Installed: kernel-3.10.0-286.el7.x86_64

Comment 8 Marian Csontos 2015-07-13 13:32:50 UTC

Created attachment 1051410 [details]
grub files

The grub.cfg included is after a manual repair. I do not have the original copy.

Comment 9 Oleg Vazhnev 2015-11-12 18:06:27 UTC

I meet the same issue today.
This happened after I:

1. Disable Network Manager
systemctl mask NetworkManager
systemctl stop NetworkManager
systemctl disable NetworkManager

(after this I have this bug https://bugzilla.redhat.com/show_bug.cgi?id=1138939)

2. tried to install updates

Now I can not boot my system. It seems I have to reinstall my system, also I do not know if I can get my files via iKVM

Comment 10 Oleg Vazhnev 2015-11-12 21:45:38 UTC

During update grub file was corrupted:

http://content.screencast.com/users/OlegVazhnev/folders/Jing/media/b02408ff-a003-446c-a5dc-11241f5a8aa5/2015-11-13_0043.png

May be I can resolve the issue regenerating grub file

Comment 12 David Cantrell 2016-03-15 17:37:06 UTC

Is anyone able to reproduce this problem on RHEL 7.2 GA or a currently 7.3 nightly?

Comment 13 Marian Csontos 2016-04-06 19:16:36 UTC

This does not look like a bug which would be easy to reproduce... I have not seen it since.

Comment 14 Marian Csontos 2016-04-06 19:18:19 UTC

I guess the needinfo flag can be cleared as well.

Comment 15 karan singh 2017-07-28 19:19:27 UTC

I am hitting this while booting into my RHEL 7.4. Does anyone know how to fix it ?

Comment 16 marcindulak 2018-01-15 17:57:15 UTC

I've experienced this after an update of Ubuntu 16.04.

Searches brought me here so even if that's a different distribution I'll write that "Reinstall GRUB boot loader" available in Ubuntu's server "Rescue a broken system" helped.

In my case "fatal error: token too large, exceeds YYLMAX" happened after a finished kernel update, followed by a system hang requiring a hard reset. Maybe not all grub changes got flushed to disk, but I didn't notice anything bad in grub.cfg while in rescue mode.

Comment 17 A.Bourinsky 2019-07-13 19:11:57 UTC

can't boot system after update, each reboot failed with the same message.
OS:Fedora30, dual boot

Comment 19 Phu H. Nguyen 2020-04-08 00:41:17 UTC

I have only inspected the problem on my machine, not sure about others especially since this thread was started 5 years ago. Anyway these are what I've got:
- Fedora 31, grub2 with BLS enabled, grubby-deprecated package was installed when trying to fix another issue;
- After every kernel update, grub.cfg is corrupted with the UUID being repeated 21949 times on that 'set default_kernelopts="root=...' line, a process running 'new-kernel-pkg' script with the command '$grubby --grub2 -c $grub2Config --remove-kernel=$kernelImage' was killed, check journalctl and see that grubby was killed for OOM, running grub2-mkconfig makes grub.cfg good again; 'new-kernel-pkg' script belongs to the grubby-deprecated package;
- The UUID being repeated in grub.cfg is caused by grubby repeating the right hand side of every equal sign on that 'set' line. Grubby is used by the 'new-kernel-pkg' script which is called from /usr/lib/kernel/install.d/20-grubby.install which in turn is triggered from the 'kernel-install' script which runs after each kernel update.
So if you have BLS enabled for grub and a corrupted grub.cfg after each kernel update, you might want to check if you have the 'new-kernel-pkg' script, and along with that the grubby-deprecated package, installed, and remove it.

Comment 21 RHEL Program Management 2020-12-15 07:35:14 UTC

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

Note You need to log in before you can comment on or make changes to this bug.