Bug 1947679

Summary: Encrypted/Credentials/v1@X-GSSPROXY: showing expired tickets, 12/31/69 19:00:00
Product: [Fedora] Fedora Reporter: RobbieTheK <rkudyba>
Component: gssproxyAssignee: Robbie Harwood <rharwood>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 33CC: abokovoy, gdeschner, rharwood, ssorce
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-04-09 05:43:04 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
krb5.conf, with actual domain obfuscated
none
results of strace gssproxy -i none

Description RobbieTheK 2021-04-08 21:01:02 UTC
Created attachment 1770451 [details]
krb5.conf, with actual domain obfuscated

Immediately after a user runs kinit from a client these expired tickets show up.

gssproxy-0.8.3-3.fc33.x86_64, freeipa-server-4.9.2-4,  5.10.18-200.fc33 

klist
Ticket cache: KCM:0:59081
Default principal: host/ourserver.edu

Valid starting     Expires            Service principal
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:

klist -l
Principal name                 Cache name
--------------                 ----------
host/ourdomain.edu@OL KCM:0:59081 (Expired)
admin   KCM:0:64094 (Expired)

KRB5_TRACE=/dev/stderr kinit admin
[1398701] 1617915279.516440: Getting initial credentials for admin
[1398701] 1617915279.516442: Sending unauthenticated request
[1398701] 1617915279.516443: Sending request (202 bytes) toOURSERVER.EDU
[1398701] 1617915279.516444: Initiating TCP connection to stream x.x.x.x:88
[1398701] 1617915279.516445: Sending TCP request to stream x.x.x.x:88
[1398701] 1617915279.516446: Received answer (326 bytes) from stream x.x.x.x:88
[1398701] 1617915279.516447: Terminating TCP connection to stream x.x.x.x:88
[1398701] 1617915279.516448: Response was from master KDC
[1398701] 1617915279.516449: Received error from KDC: -1765328359/Additional pre-authentication required
[1398701] 1617915279.516452: Preauthenticating using KDC method data
[1398701] 1617915279.516453: Processing preauth types: PA-PK-AS-REQ (16), PA-FX-FAST (136), PA-ETYPE-INFO2 (19), PA-PKINIT-KX (147), PA-ENC-TIMESTAMP (2), PA_AS_FRESHNESS (150), PA-FX-COOKIE (133)
[1398701] 1617915279.516454: Selected etype info: etype aes256-cts, salt "W#F>BJAgRrax0.,x", params ""
[1398701] 1617915279.516455: Received cookie: MIT
[1398701] 1617915279.516456: PKINIT client has no configured identity; giving up
[1398701] 1617915279.516457: Preauth module pkinit (147) (info) returned: 0/Success
[1398701] 1617915279.516458: PKINIT client received freshness token from KDC
[1398701] 1617915279.516459: Preauth module pkinit (150) (info) returned: 0/Success
[1398701] 1617915279.516460: PKINIT client has no configured identity; giving up
[1398701] 1617915279.516461: Preauth module pkinit (16) (real) returned: 22/Invalid argument
Password for admin:
[1398701] 1617915284.393323: AS key obtained for encrypted timestamp: aes256-cts/8BA3
[1398701] 1617915284.393325: Encrypted timestamp (for 1617915284.642429): plain 301AA011180F32303231303430383230353434345AA105020309CD7D, encrypted CEE4900E8E68B0B93F6BDFB7B1FB326AE81F6CB0FC76ED6CEAAAD52CBADA0D3E8FBC4A1CB4BB8BBE53289B63028BD857EF850B32468F2EA9
[1398701] 1617915284.393326: Preauth module encrypted_timestamp (2) (real) returned: 0/Success
[1398701] 1617915284.393327: Produced preauth for next request: PA-FX-COOKIE (133), PA-ENC-TIMESTAMP (2)
[1398701] 1617915284.393328: Sending request (297 bytes) toOURSERVER.EDU
[1398701] 1617915284.393329: Initiating TCP connection to stream x.x.x.x:88
[1398701] 1617915284.393330: Sending TCP request to stream x.x.x.x:88
[1398701] 1617915284.393331: Received answer (794 bytes) from stream x.x.x.x:88
[1398701] 1617915284.393332: Terminating TCP connection to stream x.x.x.x:88
[1398701] 1617915284.393333: Response was from master KDC
[1398701] 1617915284.393334: Processing preauth types: PA-ETYPE-INFO2 (19)
[1398701] 1617915284.393335: Selected etype info: etype aes256-cts, salt "W#F>BJAgRrax0.,x", params ""
[1398701] 1617915284.393336: Produced preauth for next request: (empty)
[1398701] 1617915284.393337: AS key determined by preauth: aes256-cts/8BA3
[1398701] 1617915284.393338: Decrypted AS reply; session key is: aes256-cts/0521
[1398701] 1617915284.393339: FAST negotiation: available
[1398701] 1617915284.393340: Initializing KCM:0:64094 with default princ admin
[1398701] 1617915284.393341: Storing admin -> krbtgt/OURDOMAIN.EDU in KCM:0:64094
[1398701] 1617915284.393342: Storing config in KCM:0:64094 for krbtgt/OURDOMAIN.EDU: fast_avail: yes
[1398701] 1617915284.393343: Storing admin -> krb5_ccache_conf_data/fast_avail/krbtgt\/OURDOMAIN.EDU\@OURDOMAIN.EDU@X-CACHECONF: in KCM:0:64094
[1398701] 1617915284.393344: Storing config in KCM:0:64094 for krbtgt/OURDOMAIN.EDU: pa_type: 2
[1398701] 1617915284.393345: Storing admin -> krb5_ccache_conf_data/pa_type/krbtgt\/OURDOMAIN.EDU\@OURDOMAIN.EDU@X-CACHECONF: in KCM:0:64094

On the server, running kinit always fixes it:

klist
Ticket cache: KCM:0:64094
Default principal: admin

Valid starting     Expires            Service principal
04/08/21 16:54:44  04/09/21 16:54:39  krbtgt/OURDOMAIN.EDU

Similar to https://bugzilla.redhat.com/show_bug.cgi?id=1724460 and https://bugzilla.redhat.com/show_bug.cgi?id=1757224

I'll attach a strace after submitting this.

Comment 1 RobbieTheK 2021-04-08 21:01:39 UTC
Created attachment 1770452 [details]
results of strace gssproxy -i

Comment 2 Alexander Bokovoy 2021-04-09 05:43:04 UTC
This is not a bug but rather an intended behavior of GSS-Proxy. It stores encrypted blobs as 'tickets' in a ccache and decodes them when GSSAPI applications try to access them.

The 'outdated' part is intentional because any entry in the ccache must have a timestamp and GSS-Proxy chose to use UNIX start time (timestamp 0) for that.

Ideally you should not use GSSAPI and non-GSSAPI against the same credential cache with GSS-Proxy. GSS-Proxy only supports GSSAPI clients; non-GSSAPI clients will not see credentials properly, i.e. they only would see these encrypted blobs.

Comment 3 RobbieTheK 2021-04-09 12:49:37 UTC
(In reply to Alexander Bokovoy from comment #2)

> Ideally you should not use GSSAPI and non-GSSAPI against the same credential
> cache with GSS-Proxy. GSS-Proxy only supports GSSAPI clients; non-GSSAPI
> clients will not see credentials properly, i.e. they only would see these
> encrypted blobs.

Since we still want NIS and Unix local users to continue to work, what's the ideal way to configure this?

Comment 4 Alexander Bokovoy 2021-04-09 13:07:00 UTC
Don't use GSS-Proxy?

Comment 5 RobbieTheK 2021-04-09 13:38:34 UTC
(In reply to Alexander Bokovoy from comment #4)
> Don't use GSS-Proxy?

OK I'll test that to see if it breaks/fixes anything. Over night, I see FreeIPA updated and now the status of gssproxy shows this:

Apr 09 02:23:31 ourdomain.edu gssproxy[1412165]: gssproxy[1412165]: Problem with /proc; program name matching won't work: 2 (No such file or directory)
Apr 09 02:23:31 ourdomain.edu gssproxy[1412165]: Problem with /proc; program name matching won't work: 2 (No such file or directory)

Is this a separate issue?

Comment 6 Alexander Bokovoy 2021-04-09 13:50:34 UTC
It is known and unrelated.

Comment 7 RobbieTheK 2021-04-09 14:23:19 UTC
(In reply to Alexander Bokovoy from comment #4)
> Don't use GSS-Proxy?

OK the outdated tickets disappear when I disable gssproxy. If I'm reading https://danwalsh.livejournal.com/65467.html correctly this also lowers security and performance.

If NIS is disabled but local users are still available (as a back up in case Kerberos stops working), will the outdated timestamps still appear? Is this just something to document for our users and admins so they also don't think it's a bug?

Comment 8 Alexander Bokovoy 2021-04-09 14:34:31 UTC
It is really a problem with non-GSSAPI tools, not the content of the ccache. Causes confusion, sure, worth documenting, may be.

Simo, what is your take on it?

Comment 9 Simo Sorce 2021-04-09 15:25:36 UTC
I am surprised to see multiple entries, that may be something to investigate in KCM.
As for documenting, the question would be where... not opposed, but also not obvious where you would put this so that it is actually useful/visible.

Comment 10 RobbieTheK 2021-04-09 15:42:08 UTC
(In reply to Simo Sorce from comment #9)
> I am surprised to see multiple entries, that may be something to investigate
> in KCM.

Can I provide you any debug/logs/strace? In the strace I attached here are some errors, but perhaps they are normal:
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffded9607d0) = -1 EINVAL (Invalid argument)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
access("/etc/system-fips", F_OK)        = -1 ENOENT (No such file or directory)
statfs("/sys/fs/selinux", {f_type=SYSFS_MAGIC, f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={val=[0, 0]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
statfs("/selinux", 0x7ffded960720)      = -1 ENOENT (No such file or directory)

> As for documenting, the question would be where... not opposed, but also not
> obvious where you would put this so that it is actually useful/visible.

Blogs, Fedora docs, README, etc.

Comment 11 Simo Sorce 2021-04-09 16:16:25 UTC
Those failures are normal on your system configuration, they are looking for optional behavior triggered only when those files exist.

Also note that GSS-Proxy has absolutely nothing to do with what User information system you use. It is orthogonal to ipa/local/nis users.

It is also generally a disabled behavior. Except for Client NFS.
So if you have NFS mounts that is what is triggering GSS-Proxy to be used.
But you have to also configure it to actually have access to some credentials to trigger the behavior you see.

I would check the configuration of gssproxy and see what's in there, it is possible it has been improperly configured.
(you do not need to reference user ccaches for NFS authentication for example, you can use separate ccaches for that case when impersonation or user keytabs are used).

Comment 12 RobbieTheK 2021-04-09 20:24:29 UTC
(In reply to Simo Sorce from comment #11)
> Those failures are normal on your system configuration, they are looking for
> optional behavior triggered only when those files exist.
> 
> Also note that GSS-Proxy has absolutely nothing to do with what User
> information system you use. It is orthogonal to ipa/local/nis users.
> 
> It is also generally a disabled behavior. Except for Client NFS.
> So if you have NFS mounts that is what is triggering GSS-Proxy to be used.
> But you have to also configure it to actually have access to some
> credentials to trigger the behavior you see.

What's generally disabled? gssproxy? I don't ever recall manually starting it. Does that not happen when FreeIPA is installed/configured?


> I would check the configuration of gssproxy and see what's in there, it is
> possible it has been improperly configured.
> (you do not need to reference user ccaches for NFS authentication for
> example, you can use separate ccaches for that case when impersonation or
> user keytabs are used).

We do have automount/autofs for users' home directories as well as CIFS/SMB for a Drobo network attached storage device, as well as exporting a share via /etc/exports

I enaabled debug for gssproxy and here is what I am seeing:

Apr  9 15:17:28 ourdomain systemd[1]: Starting GSSAPI Proxy Daemon...
Apr  9 15:17:28 ourdomain gssproxy[1433894]: [2021/04/09 19:17:28]: Debug Enabled (level: 1)
Apr  9 15:17:28 ourdomain gssproxy[1433894]: [2021/04/09 19:17:28]: Service: ipa-httpd, Keytab: /var/lib/ipa/gssproxy/http.keytab, Enctype: 18
Apr  9 15:17:29 ourdomain gssproxy[1433894]: [2021/04/09 19:17:28]: Service: ipa-api, Keytab: /var/lib/ipa/gssproxy/http.keytab, Enctype: 18
Apr  9 15:17:29 ourdomain gssproxy[1433894]: [2021/04/09 19:17:28]: Service: ipa-sweeper, Keytab: /var/lib/ipa/gssproxy/http.keytab, Enctype: 18
Apr  9 15:17:29 ourdomain gssproxy[1433894]: [2021/04/09 19:17:28]: Service: nfs-server, Keytab: /etc/krb5.keytab, Enctype: 18
Apr  9 15:17:29 ourdomain gssproxy[1433894]: [2021/04/09 19:17:29]: Service: nfs-client, Keytab: /etc/krb5.keytab, Enctype: 18
Apr  9 15:17:29 ourdomain gssproxy[1433895]: [2021/04/09 19:17:29]: Client [2021/04/09 19:17:29]: (/usr/sbin/gssproxy) [2021/04/09 19:17:29]:  connected (fd = 13)[2021/04/09 19:17:29]:  (pid = 1433895) (uid = 0) (gid = 0)[2021/04/09 19:17:29]:
Apr  9 15:17:29 ourdomain systemd[1]: Started GSSAPI Proxy Daemon.

There are 4 config files:
ls -l /etc/gssproxy/
total 16
-rw------- 1 root root 779 Apr  9 02:23 10-ipa.conf
-rw------- 1 root root 152 Mar 16 12:57 24-nfs-server.conf
-rw------- 1 root root 276 Jul 31  2020 99-nfs-client.conf
-rw------- 1 root root  11 Jul 31  2020 gssproxy.conf

# cat /etc/gssproxy/10-ipa.conf
#Installed and maintained by ipa update tools, please do not modify
[service/ipa-httpd]
  mechs = krb5
  cred_store = keytab:/var/lib/ipa/gssproxy/http.keytab
  cred_store = client_keytab:/var/lib/ipa/gssproxy/http.keytab
  allow_protocol_transition = true
  allow_client_ccache_sync = true
  cred_usage = both
  euid = apache

[service/ipa-api]
  mechs = krb5
  cred_store = keytab:/var/lib/ipa/gssproxy/http.keytab
  cred_store = client_keytab:/var/lib/ipa/gssproxy/http.keytab
  allow_constrained_delegation = true
  allow_client_ccache_sync = true
  cred_usage = initiate
  euid = ipaapi

[service/ipa-sweeper]
  mechs = krb5
  cred_store = keytab:/var/lib/ipa/gssproxy/http.keytab
  socket = /var/lib/gssproxy/ipa_ccache_sweeper.sock
  euid = ipaapi
  cred_usage = initiate


# cat /etc/gssproxy/24-nfs-server.conf
[service/nfs-server]
  mechs = krb5
  socket = /run/gssproxy.sock
  cred_store = keytab:/etc/krb5.keytab
  trusted = yes
  kernel_nfsd = yes
  euid = 0

# cat /etc/gssproxy/gssproxy.conf
[gssproxy]

from systemctl status gssproxy:
Apr 09 15:42:26 ourdomain.edu gssproxy[1433895]: [CID 14][2021/04/09 19:42:26]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "nfs-client", euid: 0,socket: (null)
Apr 09 15:42:26 ourdomain.edu gssproxy[1433895]: [CID 14][2021/04/09 19:42:26]: gp_rpc_execute: executing 8 (GSSX_INIT_SEC_CONTEXT) for service "nfs-client", euid: 0,socket: (null)

And now I no longer see the repeating tickets:

klist -A
Ticket cache: KCM:0:9354
Default principal: host/ourdomain.edu

Valid starting     Expires            Service principal
12/31/69 19:00:00  12/31/69 19:00:00  Encrypted/Credentials/v1@X-GSSPROXY:

Ticket cache: KCM:0
Default principal: admin

Valid starting     Expires            Service principal
04/09/21 15:21:11  04/10/21 15:21:09  krbtgt/OLDDSM.DSM.FORDHAM.EDU

Perhaps by adding the -d option to gssproxy and restarting it, some cache was deleted?

Comment 13 RobbieTheK 2021-04-12 16:41:35 UTC
After doing the above now seeing this when trying to access the GUI:
[auth_gssapi:error] [pid 1549967:tid 1550173] [client x.x.x.x:58472] GSS ERROR In Negotiate Auth: gss_accept_sec_context() failed: [An unsupported mechanism was requested (Unknown error)], referer: https://ourdomain.edu/ipa/ui/

gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "ipa-httpd", euid: 48,socket: (null)