Description of problem: When trying to report a crash from ABRT, the first option is to send the information to the retrace server. It seems this procedure currently always fails. The log only says: Retrace job failed Retrace failed. Try again later and if the problem persists report this issue please. The next option is to process the crash locally and that seems to work. Version-Release number of selected component (if applicable): abrt-2.14.4-6.fc33.x86_64 libreport-2.14.0-11.fc33.x86_64 How reproducible: always Steps to Reproduce: 1. run `will_abort` 2. submit the crash to the retrace server 3. see "Job failed"
Created attachment 1718948 [details] retrace server failure screenshot
Created attachment 1718949 [details] retrace server details screenshot
Per bug 1882319 comment 11 I'm suggesting this as a Final blocker, it might violate: "All applications that can be launched using the standard graphical mechanism after a default installation of Fedora Workstation on the x86_64 architecture must start successfully and withstand a basic functionality test. " https://fedoraproject.org/wiki/Fedora_33_Final_Release_Criteria#Default_application_functionality
(In reply to Kamil Páral from comment #0) > Retrace job failed > Retrace failed. Try again later and if the problem persists report this > issue please. I see exactly the same error when I try to report a crash from F32. So this seems to be a server-related problem.
Discussed during the 2020-10-05 blocker review meeting: [0] The decision to classify this bug as a "RejectedBlocker" and an "AcceptedFreezeException" was made as the app is likely working (the bug is likely in the retrace server itself), and local traceback generation does work so we aren't stuck without being able to report crashes. Accepted as a freeze exception issue just in case a fix is needed to the package, obviously we would want that in for release. [0] https://meetbot.fedoraproject.org/fedora-blocker-review/2020-10-05/f33-blocker-review.2020-10-05-16.00.txt
The underlying problem should be fixed now and I have successfully verified the solution myself, via retracing a few synthetic crashes. If the same problem persists, please feel free to reopen the issue with more details.
Problem persists: I tried to report a gnome-shell crash today, and the retrace step failed, so I had to retrace locally. What more details do you want? (Reporting to Bugzilla failed too, because "the generated backtrace has low informational value," which is an ABRT bug because I can get a good backtrace in gdb. But that is a different bug.)
I tried sending SIGABRT to gnome-calculator and htop, and both generated the backtrace remotely just fine, but both were marked as "low information value" and none of them could be reported.
Been trying to do remote-retracing on fc32 for a while without success, and saw this item on fc33's common bugs page. Btw, there is a typo on the fc33 common bugs page: it says 1895154 instead of 1885154
(In reply to Hin-Tak Leung from comment #9) > Btw, there is a typo on the fc33 common bugs page: it says 1895154 instead of 1885154 Thanks, fixed
After bug 1893549 has been fixed, the retrace now always fails here too. Hitting "Repeat" does not help so far: --- Running report_uReport --- ('report_uReport' completed successfully) --- Skipping collect_GConf --- No matching actions found for this event. --- Skipping collect_vimrc_system --- No matching actions found for this event. --- Skipping collect_vimrc_user --- No matching actions found for this event. --- Skipping collect_xsession_errors --- No matching actions found for this event. --- Running analyze_CCpp --- Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). 'YES' Querying server settings Preparing an archive to upload Uploading 5.0 MiB Upload successful Retrace job started Retrace job failed Retrace failed. Try again later and if the problem persists report this issue please. [2020-11-03 17:39:16] [I] Analyzing crash data Do you want to generate a stack trace locally? (It may download a huge amount of data but reporting can't continue without stack trace). 'NO' ('analyze_CCpp' exited with 1)
Could you please provide a bit more detail on your crash, Christian? Which packages did the crash occur in? Could you upload the coredump (provided it's not sensitive).
Just today I saw a crash of xdg-desktop-portal. When I used retrace server, I was told it has a low information value. When I reopened the report and instead performed a local tracing, I could report a new bug just fine - bug 1894965. There is something broken on the retrace server, it doesn't seem to generate the backtrace correctly.
This time the retrace job finished successfully, so I guess it's an intermittent problem then, going on for quite(!) a while though: $ sleep 1234 & $ pkill -SEGV sleep $ fg bash: fg: job has terminated [1]+ Segmentation fault (core dumped) sleep 1234 $ abrt list Id 8347495 Component coreutils Count 1 Time 2020-11-05 18:35:14 User id 1000 (christian) Reported to ABRT Server https://retrace.fedoraproject.org/faf/reports/bthash/0fa8854a6e813ce5de3ac04e7c7db9318dfcb787 $ abrt report 8347495 Reporting problem 8347495 no actions matching this problem found for event 'collect_GConf' no actions matching this problem found for event 'collect_vimrc_system' no actions matching this problem found for event 'collect_vimrc_user' no actions matching this problem found for event 'collect_xsession_errors' ('report_uReport' completed successfully) Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). [y/N/f/e] y Querying server settings Preparing an archive to upload Uploading 16.3 KiB Upload successful Retrace job started Analyzing crash data Generating backtrace ...... Retrace job finished successfully Looking for similar problems in bugzilla Bugzilla User name: [....] ===== And, sure enough, the 2nd test shortly after didn't even get to the retrace part: $ ksh $ kill -SEGV $$ ### 688908 Function backtrace: 1 (null) + 94719070134799 2 (null) + 140434281270352 3 kill + 11 4 (null) + 94719070216393 5 (null) + 94719070215054 6 (null) + 94719070314513 7 (null) + 94719070031220 8 (null) + 94719069877289 9 (null) + 94719070660120 10 __libc_start_main + 242 11 (null) + 94719069859246 Aborted (core dumped) $ abrt list Id 0b84ec4 Component ksh Count 1 Time 2020-11-05 18:48:14 User id 1000 (christian) Reported to ABRT Server https://retrace.fedoraproject.org/faf/reports/bthash/a509aef686e612d2166f70fa256d1eab0b5440a9 $ abrt report 0b84ec4 Reporting problem 0b84ec4 no actions matching this problem found for event 'collect_GConf' no actions matching this problem found for event 'collect_vimrc_system' no actions matching this problem found for event 'collect_vimrc_user' no actions matching this problem found for event 'collect_xsession_errors' ('report_uReport' completed successfully) Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). [y/N/f/e] y Querying server settings A server-side error occurred on 'https://retrace.fedoraproject.org' Unexpected HTTP response from server: 403 Package NVR contains illegal characters Do you want to generate a stack trace locally? (It may download a huge amount of data but reporting can't continue without stack trace). [y/N/f] n ('analyze_CCpp' exited with 1) Is this maybe a variation of bug 1893549, still?
It seems we had some trouble pulling the right packages for Fedora 33 crashes in the background. The problem should be fixed now. Feel free reopen again if you stumble on the same problem again.
This just happened again: $ abrt list Id d36ed72 Component libreoffice Count 1 Time 2020-11-21 02:48:40 User id 1000 (christian) Reported to ABRT Server https://retrace.fedoraproject.org/faf/reports/bthash/5b5dd0a9e3f7209ca9c835194768a9d1af029c7e $ abrt report d36ed72 Reporting problem d36ed72 no actions matching this problem found for event 'collect_GConf' no actions matching this problem found for event 'collect_vimrc_system' no actions matching this problem found for event 'collect_vimrc_user' no actions matching this problem found for event 'collect_xsession_errors' ('report_uReport' completed successfully) Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). [y/N/f/e] y Querying server settings A server-side error occurred on 'https://retrace.fedoraproject.org' Unexpected HTTP response from server: 403 Package NVR contains illegal characters Do you want to generate a stack trace locally? (It may download a huge amount of data but reporting can't continue without stack trace). [y/N/f] Can anybody re-open this? Or is this being tracked upstream? > Handle crashes in modules > https://github.com/abrt/retrace-server/issues/400
Yours seems to be a different bug, Christian. It might be related to the upstream one, though I can't say that with certainty right now. This issue is tracking the bug which results in the "Retrace job failed" message. Could you please specify what version of the libreoffice package you have installed and which version of Fedora you're on?
This was a crash in libreoffice-core-7.0.3.1-1.fc33.x86_64 running on Fedora 33. But year, I'll continue the "Package NVR contains illegal characters" thing over in bug 1900982. Thanks.
Happened again today on a freshly installed Fedora 33 installation. gnome-shell crashed (yikes!) and the "Generating backtrace" took ~4 hours to complete, and then finally failed: --- Running report_uReport --- ('report_uReport' completed successfully) --- Skipping collect_GConf --- No matching actions found for this event. --- Skipping collect_vimrc_system --- No matching actions found for this event. --- Skipping collect_vimrc_user --- No matching actions found for this event. --- Skipping collect_xsession_errors --- No matching actions found for this event. --- Running analyze_CCpp --- Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). 'YES' Querying server settings Preparing an archive to upload You are going to upload 19.1 MiB. Continue? 'YES' Uploading 19.1 MiB Upload successful Retrace job started Analyzing crash data ................................................................................ ...................................... Generating backtrace ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ ................................................................................ .......An error occurred while connecting to 'https://retrace.fedoraproject.org' Could not connect: Connection refused Do you want to generate a stack trace locally? (It may download a huge amount of data but reporting can't continue without stack trace). 'YES' Analyzing coredump 'coredump' Cleaning cache... Cache cleaning has finished Coredump references 200 debuginfo files, 400 of them are not installed Initializing package manager Setting up repositories Looking for needed packages in repositories
Argh, and while the local processing appeared to be successful, it's the retrace server again: ============================================================================= [....] .......An error occurred while connecting to 'https://retrace.fedoraproject.org' Could not connect: Connection refused Do you want to generate a stack trace locally? (It may download a huge amount of data but reporting can't continue without stack trace). 'YES' Analyzing coredump 'coredump' Cleaning cache... Cache cleaning has finished Coredump references 200 debuginfo files, 400 of them are not installed Initializing package manager Setting up repositories Looking for needed packages in repositories Can't find packages for 200 debuginfo files Packages to download: 179 Downloading 1234.30Mb, installed size: 5281.56Mb. Continue? 'YES' Downloading (1 of 179) gnome-shell-debuginfo-3.38.2-1.fc33.x86_64.rpm: 100% [....] Generating backtrace Backtrace is generated and saved, 225233 bytes --- Running analyze_BodhiUpdates --- Looking for similar problems in bugzilla Duplicate bugzilla bug '#1842589' was found Searching for updates No updates for this package found --- Running report_Bugzilla --- Logging into Bugzilla at https://bugzilla.redhat.com Checking for duplicates Creating a new bug New bug id: 1905110 Adding External URL to bug 1905110 Adding attachments to bug 1905110 Logging out Status: NEW https://bugzilla.redhat.com/show_bug.cgi?id=1905110 --- Running post_report --- Failed to upload uReport to the server 'https://retrace.fedoraproject.org/faf' with curl: Failed to connect to retrace.fedoraproject.org port 443: Connection refused Error: curl_easy_perform: Couldn't connect to server ('post_report' exited with 1) ============================================================================= So, while I'm surely repeating myself, but: isn't this a real problem? People not being able to report bugs properly, for such a long time?
Thanks for the detailed report, Christian. We're now tracking the first issue in bug 1902312. The latter issue was caused by a brief maintenance reboot. The server should be up and running now.
Unfortunately this just happened again. It's not too often that I have to use abrt to report bugs, but when I do they always seem to fail. That wasn't the case a few years back, when abrt was working pretty reliably and bug reporting was easy (from a user's perspective). So, this happened just now: =================================================================== --- Running report_uReport --- ('report_uReport' completed successfully) --- Skipping collect_GConf --- No matching actions found for this event. --- Skipping collect_vimrc_system --- No matching actions found for this event. --- Skipping collect_vimrc_user --- No matching actions found for this event. --- Skipping collect_xsession_errors --- No matching actions found for this event. --- Running analyze_CCpp --- Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). 'YES' Querying server settings Preparing an archive to upload Uploading 7.8 MiB Upload successful Retrace job started Analyzing crash data ................................................................................ ................................................................................ .. Generating backtrace ................................................................................ .................. Retrace job failed Retrace failed. Try again later and if the problem persists report this issue please. [2021-02-06 16:44:45] [I] Analyzing crash data Do you want to generate a stack trace locally? (It may download a huge amount of data but reporting can't continue without stack trace). 'NO' ('analyze_CCpp' exited with 1) =================================================================== Maybe bug 1902312 would be more fitting, I couldn't decide. Or maybe I should just open a new bug, please let me know.
Hello, This keeps happening to me while trying to report this crash: https://retrace.fedoraproject.org/faf/reports/27174/
abrt retrace 241df89 Ma 09 feb 2021 06:21:34 +0000 Upload core dump and perform remote retracing? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. Local retracing requires downloading potentially large amount of debuginfo data [y/N] y Remote retracing Querying server settings Preparing an archive to upload You are going to upload 48,9 MiB. Continue? [y/N] y Uploading 48,9 MiB Upload successful Retrace job started Analyzing crash data ................................................................................ ...................................................................... Generating backtrace ................................................................................ ...................... Retrace job failed Retrace failed. Try again later and if the problem persists report this issue please. [2021-02-09 06:22:16] [I] Analyzing crash data
And again just now: $ abrt report cfba423 Reporting problem cfba423 no actions matching this problem found for event 'collect_GConf' no actions matching this problem found for event 'collect_vimrc_system' no actions matching this problem found for event 'collect_vimrc_user' no actions matching this problem found for event 'collect_xsession_errors' ('report_uReport' completed successfully) Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). [y/N/f/e] y Querying server settings Preparing an archive to upload You are going to upload 25.5 MiB. Continue? [y/N] y Uploading 25.5 MiB Upload successful Retrace job started Analyzing crash data ................................................................................ .......................................................................... Generating backtrace ................................................................................ ..................... Retrace job failed Retrace failed. Try again later and if the problem persists report this issue please. [2021-02-09 20:32:55] [I] Analyzing crash data
And again: --- Running report_uReport --- ('report_uReport' completed successfully) --- Skipping collect_GConf --- No matching actions found for this event. --- Skipping collect_vimrc_system --- No matching actions found for this event. --- Skipping collect_vimrc_user --- No matching actions found for this event. --- Running collect_xsession_errors --- No /home/christian/.cache/gdm/session.log --- Running analyze_CCpp --- Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). 'YES' Querying server settings Preparing an archive to upload Uploading 107.3 KiB Upload successful Retrace job started Analyzing crash data ................. Generating backtrace ................................................................................ ... Retrace job failed Retrace failed. Try again later and if the problem persists report this issue please. [2021-02-20 16:56:11] [I] Analyzing crash data Do you want to generate a stack trace locally? (It may download a huge amount of data but reporting can't continue without stack trace). 'NO' ('analyze_CCpp' exited with 1) ---- Can this report be re-opened?
Still happens (this is for geoclue, but it happens for every crash I try/have tried to report recently) --- Running report_uReport --- ('report_uReport' completed successfully) --- Skipping collect_GConf --- No matching actions found for this event. --- Skipping collect_vimrc_system --- No matching actions found for this event. --- Skipping collect_vimrc_user --- No matching actions found for this event. --- Skipping collect_xsession_errors --- No matching actions found for this event. --- Running analyze_CCpp --- Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). 'YES' Querying server settings Preparing an archive to upload Uploading 1.4 MiB Upload successful Retrace job started Analyzing crash data ............................................. Generating backtrace ................................................................................ Retrace job failed Retrace failed. Try again later and if the problem persists report this issue please. [2021-03-02 10:53:30] [I] Analyzing crash data Do you want to generate a stack trace locally? (It may download a huge amount of data but reporting can't continue without stack trace). 'NO' ('analyze_CCpp' exited with 1)
I have the same issue on Fedora 34, but in my case it fails in the middle of "Analyzing crash data"
Still not working, I wasn't successful with a single remote retracing. Reopen.
This is almost certainly not the same problem any more. All you can really tell from "Retrace job failed" is "we sent the dump to the retrace server and it failed somehow trying to analyze it". There's obviously a pretty broad range of possibilities for how that could have happened. If you're running into persistent failures at that point, it's probably best to file an issue at https://github.com/abrt/retrace-server/issues rather than filing or reopening a Bugzilla bug, as the issue is likely on the retrace server side somewhere.
FWIW, I've been following this issue the whole time, and I'm not convinced this was ever really fixed. I'm not sure if I've seen the retrace server ever *not* fail to process a backtrace in the past year or two. There is no way to distinguish why processing failed client side, so IMO it's reasonable to lump all possible occurrences of these failures into one issue report. Creating multiple reports for "it never works" doesn't seem useful....
It worked for a bit the last time it was closed; I am back here because, well, f34 is out, things break, and I am trying to file and found the retrace server is not responding... Maybe the redhat folks can re-visit the retrace server a bit before the half-yearly releases to give it a bit of love and care?
Multiple downstream bug reports would not be any use to anyone, but what would probably be more useful is reports of "it is not working for me right now with (details about the crash)" to https://github.com/abrt/retrace-server/issues .
We have a pull request in the pipeline in the upstream that should fix most of these (and related) bugs: https://github.com/abrt/retrace-server/pull/418 I'm hopeful we'll be able to merge and deploy it on retrace.fedoraproject.org early next week.
Still failing: --- Running analyze_CCpp --- Ok to upload core dump? (It may contain sensitive data). If your answer is 'No', a stack trace will be generated locally. (It may download a huge amount of data). 'YES' Querying server settings Preparing an archive to upload You are going to upload 67.0 MiB. Continue? 'YES' Uploading 67.0 MiB Upload successful Retrace job #249863983 started Analyzing crash data ......................................................................... Retrace job failed Retrace failed. Try again later and if the problem persists report this issue please.
We have deployed an upgrade for Retrace Server yesterday which should fix most of these problems. At the moment, retracing Fedora 33 and 34 on x86-64 should should proceed with little problems. (Note that very big coredumps might still take some time and perhaps fail anyway. We'll be looking into that as a separate problem.)
FWIW, I just tried to retrace a Rawhide crash, it failed. Seemed to take a bit longer to fail than it was before, but still failed.
Thanks for the heads up, Rawhide indeed seems broken on retrace.fp.org. I'm looking into it right now.
Retracing should work for Rawhide as well now. There was a bug in how we were pulling Rawhide components and RPMs. Thanks again for the heads up, Adam.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days