Bug 2284097

Summary: mkdumprd fails when call from systemd service but not when manually invoked (composefs enabled)
Product: [Fedora] Fedora Reporter: JB Trystram <jtrystra>
Component: kexec-toolsAssignee: Lichen Liu <lichliu>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: rawhideCC: bhe, coxu, fzatlouk, lichliu, ruyang, ryncsn, travier, walters, yangrr.2009
Target Milestone: ---Flags: fedora-admin-xmlrpc: mirror+
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: AcceptedFreezeException
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-10-15 00:17:35 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2247868    
Attachments:
Description Flags
systemd execution of kdump
none
kdump log when started manually (success) none

Description JB Trystram 2024-05-31 08:11:47 UTC
Booting a Fedora CoreOS rawhide VM with kdump enabled, default config. 

kdump.service fails when building the kdumprd.
Simply doing the following makes the initrd build work:
```
sudo kdumpctkl rebuild
# then the service can be enabled
sudo systemctl start kdump
```

See attached logs : 

kdump-systemd.log > the failed log from the systemd execution. 
In order to collect the logs the following systemd drop-in was added : 
```
# /etc/systemd/system/kdump.service.d/debug.conf
[Service]
Environment="debug=1"
StandardOutput=append:/var/log/kdump-log.log
StandardError=inherit 
```

kdump-success.log > is collected with `debug=1 sudo kdumpctl restart &> /var/log/kdump-success.log`


I tried to add restart and delays with various systemd services directives without success. But I can rule out an ordering issue or 



Reproducible: Always

Steps to Reproduce:
1. boot coreOS vm
2. observe the kdump.service failure
3. run `sudo kdumpctl restart`
4. run `sudo systemctl start kdump.service
Actual Results:  
Kdump initrd is built successfully when `sudo kdumpctl restart` is called

Expected Results:  
Kdump intird should be built when started through systemd

The system is running systemd v255 (255.5-1.fc41)
The system have composeFS enabled which causes `/` to be a read-only filesystem, but `/var` and `/tmp` are mounted rw 
CoreOS is built with this PR : https://github.com/coreos/fedora-coreos-config/pull/3009


[1] https://ostreedev.github.io/ostree/composefs/#enabling-composefs-unsigned

Comment 1 JB Trystram 2024-05-31 08:15:02 UTC
Created attachment 2035857 [details]
systemd execution of kdump

This is the log produced by kdump when started from systemd
`debug=1` enabled and `set -x` added to /usr/sbin/mkdumprd

Comment 2 JB Trystram 2024-05-31 08:16:41 UTC
Created attachment 2035858 [details]
kdump log when started manually (success)

This is the log produced with `debug=1 sudo kdumpctl restart` manually started from a root shell

Comment 3 JB Trystram 2024-07-17 14:04:59 UTC
The `PrivateTmp=no` drop in stopped working with `kexec-tools-2.0.28-12`

It worked on `kexec-tools-2.0.28-10`

with `2.0.28-12` I still have the issue where generating the initramfs fail on startup, but works if `kdumpctl restart` is invoked manually.

Comment 4 JB Trystram 2024-07-17 16:43:21 UTC
I have some additionnal info on this : 

Having kdump.conf set to an SSH target do not prevent the initramfs from being built. 

The following kdump.conf works (kdump.service starts as expected on boot)

          ssh core.0.1
          sshkey /root/.ssh/id_ssh_kdump
          path /home/core/crash
          core_collector makedumpfile -F -l --message-level 1 -d 31

Comment 5 Colin Walters 2024-07-19 11:43:08 UTC
This is likely the same thing as https://issues.redhat.com/browse/RHEL-35885

I'd ask here that we actually prioritize working on bootc (Image Mode) over FCOS first, please.

Comment 6 Fedora Blocker Bugs Application 2024-09-23 18:57:00 UTC
Proposed as a Freeze Exception for 41-final by Fedora user dustymabe using the blocker tracking app because:

 F41 is the first release with composeFS enabled for CoreOS systems. Kdump doesn't work on top of composeFS. If a fix becomes available, it would be nice to allow it in as FE.

Comment 7 FrantiĊĦek Zatloukal 2024-09-30 17:36:05 UTC
Discussed during the 2024-09-30 blocker review meeting: [1]

The decision to classify this bug as a AcceptedFreezeException (Final) was made:

"This is accepted because, as far as we can make out, it's an issue for FCOS that might not be fully addressable with a post-release update."

[1] https://meetbot.fedoraproject.org/blocker-review_matrix_fedoraproject-org/2024-09-30/f41-blocker-review.2024-09-30-16.00.log.html

Comment 8 Coiby 2024-10-11 09:46:21 UTC
Hi @jtrystra,

FYI, kdump-utils-1.0.47-1.fc41 https://bodhi.fedoraproject.org/updates/FEDORA-2024-ce7991aa80 has been released to fix this bug. Can you confirm the fix? Thanks!

Comment 9 Fedora Update System 2024-10-14 12:39:18 UTC
FEDORA-2024-ce7991aa80 (kdump-utils-1.0.47-1.fc41) has been submitted as an update to Fedora 41.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-ce7991aa80

Comment 10 Fedora Update System 2024-10-15 00:17:35 UTC
FEDORA-2024-ce7991aa80 (kdump-utils-1.0.47-1.fc41) has been pushed to the Fedora 41 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 11 JB Trystram 2024-10-15 06:58:42 UTC
Hi @coxu, yes it does fix this scenario. Thanks !