Bug 2248097

Summary: [s390x] "cabal update" segfaults
Product: [Fedora] Fedora Reporter: Jens Petersen <petersen>
Component: cabal-installAssignee: Jens Petersen <petersen>
Status: POST --- QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 39CC: petersen, stefansf
Target Milestone: ---Keywords: Regression, Reopened
Target Release: ---   
Hardware: s390x   
OS: Linux   
Whiteboard:
Fixed In Version: cabal-install-3.8.1.0-7.fc40 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2024-02-10 01:25:21 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jens Petersen 2023-11-06 05:54:44 UTC
Not sure what (or why/since when) is happening, but "cabal update -w ghc-9.6"
for example is segfaulting in Fedora s390x copr chroots at least.

Even worse - not just F39+ but also F38 even.

See for example https://copr.fedorainfracloud.org/coprs/petersen/cabal-install/build/6594155/

Reproducible: Always

Steps to Reproduce:
1. run "cabal update -w ghc-9.6" in s390x fedora copr build (ie in spec file)
Actual Results:  
+ cabal update -v3 -w ghc-9.6.3
Config file path source is default config file.
Config file not found: /builddir/.config/cabal/config
Writing default configuration to /builddir/.config/cabal/config
Searching for curl in path.
Found curl at /usr/bin/curl
Searching for powershell in path.
Cannot find powershell on the path
Searching for wget in path.
Cannot find wget on the path
Selected http transport implementation: curl
File monitor 'config' changed: first run
Project settings changed, reconfiguring...
creating /builddir/build/BUILD/cabal-install-3.10.1.0/dist-newstyle
creating /builddir/build/BUILD/cabal-install-3.10.1.0/dist-newstyle/cache
/var/tmp/rpm-tmp.LV9I8Q: line 48:  7846 Segmentation fault      (core dumped) cabal update -v3 -w ghc-9.6.3

Expected Results:  
No crash

Previous builds in May were successful:
eg https://download.copr.fedorainfracloud.org/results/petersen/cabal-install/fedora-rawhide-s390x/05989927-cabal-install/

Comment 1 Jens Petersen 2023-11-06 06:00:36 UTC
Also just confirmed same segfault happening in Koji s390x rawhide (though of course cabal-install cannot normally be used there due to no network).

Segfault doesn't happen with F37 s390x though.

Comment 2 Jens Petersen 2023-11-06 07:25:56 UTC
(Just for completeness: same thing with latest cabal-3.10.2.0 just released:
https://copr.fedorainfracloud.org/coprs/petersen/cabal-install/build/6601582/
)

But I confirmed in Koji that "cabal update" also segfaults

Comment 3 Jens Petersen 2023-11-06 07:26:50 UTC
(epel9 also does not segfault of course)

Comment 4 Jens Petersen 2023-11-06 11:31:36 UTC
I managed to reproduce in a s390x fedora:39 container:

strace output

newfstatat(AT_FDCWD, "/usr/bin/ghc-pkg", {st_mode=S_IFREG|0755, st_size=352, ...}, 0) = 0
getpid()                                = 279
openat(AT_FDCWD, "/dist-newstyle/cache/compiler279-0.tmp", O_RDWR|O_CREAT|O_EXCL|O_NOCTTY|O_NONBLOCK, 0666) = 11
newfstatat(11, "", {st_mode=S_IFREG|0644, st_size=0, ...}, AT_EMPTY_PATH) = 0
ioctl(11, TCGETS, 0x3ffc88f5988)        = -1 ENOTTY (Inappropriate ioctl for device)
sched_getaffinity(0, 128, [0 1])        = 8
rt_sigprocmask(SIG_BLOCK, [INT], [], 8) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x3fbe4400000000} ---
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault (core dumped)

Comment 5 Jens Petersen 2023-11-06 12:12:43 UTC
cabal-install-3.10.1.0-1.fc39.s390x also dumps core inside fedora:39

(gdb) bt
#0  0x000000008000e008 in evacuate1 ()
#1  0x0000000083f3c9b6 in scavenge_loop1 ()
#2  0x0000000083f31cbc in scavenge_until_all_done ()
#3  0x0000000083f334fa in GarbageCollect ()
#4  0x0000000083f1dce2 in scheduleDoGC ()
#5  0x0000000083f1ee90 in schedule ()
#6  0x0000000083f1f6de in scheduleWaitThread ()
#7  0x0000000083f19610 in hs_main ()
#8  0x000000008003226e in main ()

Comment 6 Jens Petersen 2023-11-06 12:15:08 UTC
Reverting F39 to https://kojipkgs.fedoraproject.org//packages/cabal-install/3.6.2.0/3.fc39/s390x/cabal-install-3.6.2.0-3.fc39.s390x.rpm
stops `cabal update` from segfaulting and after that (~/.cabal/packages/ created) current cabal-install-3.8.1.0 no longer segfaults.

Comment 7 Jens Petersen 2023-11-06 14:21:51 UTC
Crash is also reproduced with fedora-toolbox:39 toolbox (provided there is no existing ~/.cabal/)

Comment 9 Jens Petersen 2023-11-25 02:01:53 UTC
See https://gitlab.haskell.org/ghc/ghc/-/issues/24163

Comment 10 Fedora Update System 2023-12-16 10:06:32 UTC
FEDORA-2023-d1dd535dda has been submitted as an update to Fedora 38. https://bodhi.fedoraproject.org/updates/FEDORA-2023-d1dd535dda

Comment 11 Fedora Update System 2023-12-17 01:51:58 UTC
FEDORA-2023-d1dd535dda has been pushed to the Fedora 38 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2023-d1dd535dda`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2023-d1dd535dda

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 12 Fedora Update System 2023-12-25 03:36:49 UTC
FEDORA-2023-d1dd535dda has been pushed to the Fedora 38 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 13 stefansf 2024-01-31 15:21:36 UTC
Could you also release new packages for F39 including a rebuild of cabal?

Comment 14 Jens Petersen 2024-02-01 06:09:14 UTC
Thank you

Seems I didn't push the f39 build into the updates system, ugh

Comment 15 Fedora Update System 2024-02-01 06:21:19 UTC
FEDORA-2024-88c617181a has been submitted as an update to Fedora 39. https://bodhi.fedoraproject.org/updates/FEDORA-2024-88c617181a

Comment 16 Fedora Update System 2024-02-02 02:16:56 UTC
FEDORA-2024-88c617181a has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-88c617181a`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-88c617181a

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 17 Fedora Update System 2024-02-10 01:25:21 UTC
FEDORA-2024-88c617181a (cabal-install-3.8.1.0-5.fc39 and ghc-9.4.5-137.fc39) has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 18 stefansf 2024-02-10 08:03:17 UTC
I gave cabal a try and it still fails. Is it possible that cabal-install was still compiled with the broken GHC?

$ rpm -q cabal-install
cabal-install-3.8.1.0-5.fc39.s390x
$ cabal update
Config file path source is default config file.
Config file not found: /root/.cabal/config
Writing default configuration to /root/.cabal/config
Segmentation fault (core dumped)

Comment 19 Jens Petersen 2024-02-13 16:50:00 UTC
Hm, it was built with ghc-compiler-9.4.5-137.fc39.s390x,
which is supposed to have the fixed...

https://koji.fedoraproject.org/koji/taskinfo?taskID=112716128

Just double-checking: so "/usr/bin/cabal update" segfaults for you, right?
I need to check more carefully then I guess.

Comment 20 Jens Petersen 2024-02-13 16:52:15 UTC
Is it because I need to rebuild all the deps too? ;-(
I seem to remember something like that: need to check my notes in gitlab.

Comment 21 stefansf 2024-02-13 18:45:46 UTC
(In reply to Jens Petersen from comment #19)
> Hm, it was built with ghc-compiler-9.4.5-137.fc39.s390x,
> which is supposed to have the fixed...
> 
> https://koji.fedoraproject.org/koji/taskinfo?taskID=112716128
> 
> Just double-checking: so "/usr/bin/cabal update" segfaults for you, right?

Yes, I have tested it in a fresh container, i.e., no old/potentially broken $HOME/.cabal lingering around are whatever.

Comment 22 Jens Petersen 2024-02-14 17:46:33 UTC
As I feared: https://gitlab.haskell.org/ghc/ghc/-/merge_requests/11662#note_536725

> (Just rebuilding the Fedora cabal-install package was insufficient alas.)

Sigh

Comment 23 Jens Petersen 2024-02-14 17:47:53 UTC
I wonder if the mass-rebuild was even enough to fix this for deps in F40.

Comment 24 Jens Petersen 2024-02-15 05:56:12 UTC
Okay I rebuilt cabal-install for Rawhide and next F40, which should fix the problem there.

Comment 25 stefansf 2024-02-27 16:15:58 UTC
I can confirm that cabal is working for me on rawhide and next (40).  However, it is still segfaulting on 39 for me.

Comment 26 Jens Petersen 2024-02-29 13:18:34 UTC
Okay, thank you.

Yeah, I didn't have time yet sorry - there has been quite a lot of fall out for the F40, and also some pending pandoc CVEs.
So I need more time - but let me know if you consider this is a priority, and I can try to bump it up my to-do list.

Comment 27 stefansf 2024-02-29 15:16:15 UTC
Fine for me.  Just wanted to confirm that F40/Rawhide packages are good. Once F39 packages are rebuild I will give them a try, too.