Bug 1601438 - goa-daemon spams log file due to expired Kerberos logins when using KCM ticket cache (sssd-kcm)
Summary: goa-daemon spams log file due to expired Kerberos logins when using KCM ticke...
Keywords:
Status: CLOSED EOL
Alias: None
Product: Fedora
Classification: Fedora
Component: gnome-online-accounts
Version: 31
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Debarshi Ray
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1722210 1774580
TreeView+ depends on / blocked
 
Reported: 2018-07-16 12:13 UTC by Alexander Korsunsky
Modified: 2023-09-07 19:14 UTC (History)
11 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1722210 1774580 (view as bug list)
Environment:
Last Closed: 2020-11-24 20:01:24 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
GNOME Gitlab 32 0 None None None 2018-12-11 15:13:09 UTC

Description Alexander Korsunsky 2018-07-16 12:13:50 UTC
Description of problem:

Gnome-online accounts creates a long list of expired "temporary logins" from kerberos logins that are not cleaned up, causing my logfile to be spammed by goa errors.


This is what my ~/.config/goa-1.0/accounts.conf looks like:

-------------------------------------------------------------
[Account account_1531739258_603]
IsTemporary=true
SessionId=8032df612956a6ef9e874c2d5b48c52e

[Account account_1531739266_604]
IsTemporary=true
SessionId=89e2991aae5283b72b2df7725b4c70b2

... Repeated 20 times ....

[Account account_1531739281_607]
IsTemporary=true
SessionId=89e2991aae5283b72b2df7725b4c70b2

[Account account_1531739701_691]
Provider=kerberos
Identity=akorsunsky
PresentationIdentity=akorsunsky
Realm=EXAMPLE.COM
SessionId=89e2991aae5283b72b2df7725b4c70b2
IsTemporary=true
TicketingEnabled=true
-------------------------------------------------------------


This causes the following errors in my journal to appear *twenty times*, *every single second*:

-------------------------------------------------------------
Jul 16 13:29:52 mymachine.example.com goa-daemon[3253]: Unsupported account type (null) for ID account_1531481542_37043 (no provider)
Jul 16 13:29:52 mymachine.example.com goa-daemon[3253]: goa_provider_get_for_provider_type: assertion 'provider_type != NULL' failed
Jul 16 13:29:52 mymachine.example.com goa-daemon[3253]: Unsupported account type (null) for identity (null) (no provider)
-------------------------------------------------------------

Please do the math on this one. Twenty times each second is 1200 times per minute, which is 72000 times per hour.
If I scroll through my logs, these messages is all I can see. Please send help.


Note that removing ~/.config/goa-1.0/accounts.conf helps temporary, but on reboot or new login it is recreated, starts filling up with these "IsTemporary=true" accounts, and then starts spamming my log again.

The issue is compounded by Bug 1379070: I don't even need GOA, I am perfectly happy without it. But I have no way of turning it off and no way of uninstalling it because it's a core dependency of all of the GNOME desktop.

For now it just keeps spamming my log files, but if we could come up with a fix or workaround, that would be really great.



Version-Release number of selected component (if applicable):
gnome-online-accounts-3.28.0-1.fc28.x86_64


How reproducible:
Every time

Steps to Reproduce:
1. Log in to my machine
2.
3.

Actual results:
Have my logs spammed


Expected results:
Not have my logs spammed


Additional info:

Comment 1 René Genz 2018-08-17 08:03:00 UTC
Thank you for pointing me here. I have the same problem with a similar environment. For more information about how goa-daemon filled /tmp, see bug 1379070#c23 and a correction in 1379070#c25.


Alexander proposed a workaround how to disable goa at 1379070#c22

Comment 2 Alexander Korsunsky 2018-12-11 15:13:09 UTC
This is still an issue in Fedora 29. In fact, when testing again I was greeted with about 300.000 messages within 10 minutes, because I forgot to clean out my  ~/.config/goa-1.0/accounts.conf .

I created an upstream bug in the hopes that somebody will actually see it: https://gitlab.gnome.org/GNOME/gnome-online-accounts/issues/32

I get that it's annoying when users think their bug is the most important one and needs to be fixed immediately - 
but you have to realize that there is no real workaround, because one can't even disable gnome-online-accounts, because something[1] and the bugs requesting it get closed with "NOTABUG". 

Honestly wondering why nobody cares.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1379070#c11

Comment 3 Alexander Korsunsky 2019-02-27 13:36:04 UTC
I was using the KCM kerberos ticket cache provided by the package sssd-kcm.

Another workaround suggested by @debarshir [1] that doesn't involve killing gnome-online-accounts is to switch back to the KEYRING ticket cache by removing the sssd-kcm package.

This is working for me.

[1] https://gitlab.gnome.org/GNOME/gnome-online-accounts/issues/32#note_419973

Comment 4 Albert 2019-04-08 01:54:40 UTC
Same issue here, getting millions of messages:

 goa-daemon[2562]: goa_provider_get_for_provider_type: assertion 'provider_type != NULL' failed
 goa-daemon[2562]: Unsupported account type (null) for identity (null) (no provider)

any progress and/or solutions ?

Comment 5 Simone Caronni 2019-10-15 04:19:39 UTC
Being hit by this on all our Fedora clients connected to FreeIPA, CPU constantly at 100% unless we remove sssd-kcm.

Comment 6 Ben Cotton 2019-10-31 18:46:05 UTC
This message is a reminder that Fedora 29 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 29 on 2019-11-26.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '29'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 29 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 7 Ben Cotton 2019-11-27 18:08:13 UTC
Fedora 29 changed to end-of-life (EOL) status on 2019-11-26. Fedora 29 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.

Comment 8 Alexander Korsunsky 2019-12-03 15:21:32 UTC
Still happens with Fedora 31, even though sssd-kcm is not even installed.

Comment 9 Debarshi Ray 2020-03-06 17:30:35 UTC
Do you still see this problem? I am asking because on many of the other reports about the same bug, people have been saying that they saw this issue go away over time.

If you are still seeing this, then I need some more information because while it seems to constantly affect some users, it just doesn't reproduce for others.

If you start from a clean state (ie., no Kerberos tickets or accounts anywhere) then what are the exact steps to trigger this?

To attain a clean state:

* Hard disable /usr/libexec/goa-daemon and /usr/libexec/goa-identity-service by removing the executable bit and killing them, so that they don't start up again.

* Remove all stanzas with "IsTemporary=true" or "Provider=kerberos". Or even better, just remove that file.

* Reboot the machine just to reset all the SSSD daemons, which offer the Kerberos KCM caches; or to reset the kernel keyring.

* Restore the executable bit on /usr/libexec/goa-daemon and /usr/libexec/goa-identity-service.

At this point, what are the exact steps that cause this bug to appear? For example, are you creating your Kerberos tickets using kinit? Or are you creating your tickets using the Online Accounts GUI in Settings? If you are using the GUI, then do you select the checkbox to save your password? Do you add more than one Kerberos account? Or just one?

I suspect that kinit was used, not the GUI. If so, when does this behaviour show up? Does one have to wait for the ticket to expire?

Also, does your Kerberos Key Distribution Centre (or KDC) allow tickets to be renewed or not?

Comment 10 Alexander Korsunsky 2020-03-11 11:39:47 UTC
(In reply to Debarshi Ray from comment #9)
> Do you still see this problem?

I had GOA disabled for the time being. I removed ~/.config/goa-1.0/accounts.conf, rebooted and reactivated it now. I'll do some testing for a few days and report back. For now, the spam is gone.


> If you start from a clean state (ie., no Kerberos tickets or accounts
> anywhere) then what are the exact steps to trigger this?

"No Kerberos tickets" is going to be difficult, because I log in through Kerberos, specifically FreeIPA.


> are you creating your Kerberos tickets using kinit?

Tickets are created automatically by SSSD on login.

> are you creating your tickets using the Online Accounts GUI in Settings? 

No, never touched it.

> Do you add more than one Kerberos account? Or just one?

Only one.

> Also, does your Kerberos Key Distribution Centre (or KDC) allow tickets to
> be renewed or not?

Yes, the FreeIPA instance allows ticket renewal and SSSD is set up to renew tickets up to 90 days.

As mentioned, I will report back in a few days on whether the bug still happens nor not. However, it still happened 3 months ago with Fedora 31.

Comment 11 Alexander Korsunsky 2020-03-18 10:59:28 UTC
A week later, my ~/.config/goa-1.0/accounts.conf did *not* blow up and I do not get those log messages anymore.

It seems that for me the bug is fixed for now.

Comment 12 René Genz 2020-10-20 09:43:08 UTC
I use a Fedora 32 computer as X2Go server (Xfce, GNOME, and KDE desktop environments are installed). It is connected by realm/SSSD to Microsoft Active Directory. Only Xfce sessions are started by the X2Go clients. On the X2Go server All 16 GiB of /tmp got filled by:
/tmp/.x2go-${name}/C-${name}-51-1601476109_stDXFCE_dp32/cmdoutput

The file contains a lot of this line:
(goa-daemon:43997): goa-daemon-WARNING **: 09:37:38.705: Unsupported account type (null) for ID account_1548857514_11127 (no provider)

and every now and then a repeated block of these:
(goa-daemon:43997): GoaBackend-CRITICAL **: 09:37:38.705: goa_provider_get_for_provider_type: assertion 'provider_type != NULL' failed

(goa-daemon:43997): goa-daemon-WARNING **: 09:37:38.705: Unsupported account type (null) for identity (null) (no provider)



Compared to bug 1379070#c23 the "non-ASCII char" error message on login is there, the line numbers are different:
$ FILE="${HOME}/.config/goa-1.0/accounts.conf"
$ fgrep '\[Account account' "${FILE}" | wc -l
890
$ ls -lh "${FILE}"
-rw-r--r-- 1 ${name} ${group} 77K 20. Okt 10:21 /usr/home/NAME/.config/goa-1.0/accounts.conf # is home directory stored on NFS share



Here is additional information about ${FILE}:
- there are entries of "[Account account_NUMBER]" without attributes
- there are a lot of entries with the attribute "IsTemporary=true"; there number goes down and up
- sometimes the entry with the line "Realm=" disappears, not just the line, the whole entry; after a few seconds the entry is back
- here is the content:
$ cat "${FILE}"
...
[Account account_1603166663_31459]

...
[Account account_1603173303_31727]
IsTemporary=true
SessionId=a7d92fedb786d01e3b6673415f749611

[Account account_1603182068_32089]
Provider=kerberos
Identity=name
PresentationIdentity=name
Realm=DOMAIN.LOCAL
SessionId=a7d92fedb786d01e3b6673415f749611
IsTemporary=true
TicketingEnabled=true


- here are the changing numbers and dis-/reappearing of REALM entry 
$ fgrep "IsTemporary=true" "${FILE}" | wc -l
767
$ fgrep "Realm=" "${FILE}" | wc -l
0
$ fgrep "Realm=" "${FILE}" | wc -l
1
$ fgrep "Realm=" "${FILE}" | wc -l
1
$ fgrep "Realm=" "${FILE}" | wc -l
1
$ fgrep "Realm=" "${FILE}" | wc -l
1
$ fgrep "Realm=" "${FILE}" | wc -l
0
$ fgrep "Realm=" "${FILE}" | wc -l
0
$ fgrep "Realm=" "${FILE}" | wc -l
0
$ fgrep "Realm=" "${FILE}" | wc -l
0
$ fgrep "Realm=" "${FILE}" | wc -l
1
$ fgrep "IsTemporary=true" "${FILE}" | wc -l
766
$ fgrep "IsTemporary=true" "${FILE}" | wc -l
767



Solution for me today was:
- free space on /tmp with (X2Go clients can connect again):
echo "" > /tmp/.x2go-${name}/C-${name}-51-1601476109_stDXFCE_dp32/cmdoutput

- activate killing of `goa-daemon` on the X2Go server on session start (leaving out error handling):
NAME="kill_goa-daemon"
FILE="/usr/local/${NAME}.bash"
cat >> "/etc/xdg/autostart/${NAME}.desktop" << EOF
[Desktop Entry]
Type=Application
Name=killall goa-daemon
Comment=killall goa-daemon, see RHBZ#1379070
Exec=${FILE}
EOF

cat >> "${FILE}" << 'EOF'
#!/bin/bash
# wait some time for everything to settle
sleep 10
echo "DEBUG: killall goa-daemon"
killall goa-daemon
EOF

chmod +x "${FILE}"

- log out of X2Go session and start a new one



I have access to a valid RHEL Workstation subscription, hence I can try to reproduce it there and file a separate bug report. If that would help, let me know, please.

Comment 13 Ben Cotton 2020-11-03 16:50:51 UTC
This message is a reminder that Fedora 31 is nearing its end of life.
Fedora will stop maintaining and issuing updates for Fedora 31 on 2020-11-24.
It is Fedora's policy to close all bug reports from releases that are no longer
maintained. At that time this bug will be closed as EOL if it remains open with a
Fedora 'version' of '31'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 31 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

Comment 14 Ben Cotton 2020-11-24 20:01:24 UTC
Fedora 31 changed to end-of-life (EOL) status on 2020-11-24. Fedora 31 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.