Bug 1931213

Summary: gssproxy - kvno 1 not found in keytab; ticket is likely out of date
Product: [Fedora] Fedora Reporter: Wolfgang Szoecs <wolfi>
Component: gssproxyAssignee: Robbie Harwood <rharwood>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 33CC: abokovoy, gdeschner, rharwood, ssorce
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-22 14:05:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Wolfgang Szoecs 2021-02-21 16:48:27 UTC
Description of problem:

setting up krb5-NFS - gssproxy refuses mount request.
it permanently reports:

Feb 21 11:08:51 flash gssproxy[723]:    GSSX_RES_ACCEPT_SEC_CONTEXT( status: { 851968 <None> 2529638956 "Unspecified GSS failure.  Minor code may provide more information" "Request ticket server nfs/flash.chaosnet.lan kvno 1 not found in keytab; ticket is likely out of date" [  ] } context_handle: <Null> output_token: <Null> delegated_cred_handle: <Null> )

interesting enough regardless what kvno the keytab contains - gssproxy always flags 'kvno 1' as the issue.


Version-Release number of selected component (if applicable):

fedora 33 server
5.10.16-200.fc33.x86_64
gssproxy-0.8.3-3.fc33.x86_64
nfs-utils-2.5.2-1.rc4.fc33.x86_64


How reproducible:

setup a krb5 nfs4-export - and try to mount it from another machine.

Steps to Reproduce:
1. setup a correct krb5-membership (like IPA-Client!)
2. get the nfs/hostname principal into the server's keytab via 'ipa-getkeytab -p nfs/hostname -k /etc/krb5.keytab'
3. export a tree via /etc/exports like:
/dir gss/krb5(fsid=0,rw,no_root_squash,no_subtree_check,async,nohide,insecure)


Actual results:
A valid mount-attempt from any client fails.
what is a 'valid' mount-attempt ? - well - the same mount-cmd to a different server - works!
works on centos-8.3 to centos 7.9 server or vice versa (centos-7.9 server, centos 8.3 client)
EXPORTING FAILS on Fedora_33. -- Mounting another krb5-nfs4 server WORKS!

Expected results:
Exporting (and mounting it from a client) an nfs4-krb5-FS should work on fedora 33 too.

Additional info: 

(SERVER-SIDE - FEDORA-33)

[root@flash log]# klist -k
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   5 host/flash.chaosnet.lan
   5 host/flash.chaosnet.lan
   4 nfs/flash.chaosnet.lan
   4 nfs/flash.chaosnet.lan

[root@flash log]# kvno nfs/flash.chaosnet.lan
nfs/flash.chaosnet.lan: kvno = 4
[root@flash log]#

if you change gssproxy to run with '--debug-level=2' - each mount attempt from a client leaves this in /var/log/messages:

Feb 21 11:08:51 flash gssproxy[723]:    GSSX_RES_ACCEPT_SEC_CONTEXT( status: { 851968 <None> 2529638956 "Unspecified GSS failure.  Minor code may provide more information" "Request ticket server nfs/flash.chaosnet.lan kvno 1 not found in keytab; ticket is likely out of date" [  ] } context_handle: <Null> output_token: <Null> delegated_cred_handle: <Null> )

I just don't have an idea, where this idea of 'kvno 1' comes from ... it's a fresh install.

... rebooting the server doesn't change anything on this behavior.

A client mount-attempt (centos-8.3, but also centos-7.9) always just fails:

[root@hpe-lt ~]# /usr/bin/mount -v -v -v -t nfs4 flash:/ -o vers=4,sec=krb5 /mnt/flash
mount.nfs4: timeout set for Sun Feb 21 11:10:50 2021
mount.nfs4: trying text-based options 'sec=krb5,vers=4.2,addr=10.20.30.50,clientaddr=10.20.30.202'
mount.nfs4: mount(2): Permission denied
mount.nfs4: trying text-based options 'sec=krb5,vers=4,minorversion=1,addr=10.20.30.50,clientaddr=10.20.30.202'
mount.nfs4: mount(2): Permission denied
mount.nfs4: trying text-based options 'sec=krb5,vers=4,addr=10.20.30.50,clientaddr=10.20.30.202'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting flash:/

INTERESTING ENOUGH - this 'server' - has no problems to mount as a client - a krb5-NFS4-mount !!! THAT WORKS.

like this: (hpe-lt is a centos-8.3 machine)

[root@flash ~]# mkdir /mnt/hpe-lt
[root@flash ~]# mount -v -v -t nfs4 -o sec=krb5 hpe-lt:/ /mnt/hpe-lt 
mount.nfs4: timeout set for Sun Feb 21 17:48:29 2021
mount.nfs4: trying text-based options 'sec=krb5,vers=4.2,addr=10.20.30.202,clientaddr=10.20.30.50'

[root@flash ~]# df -h /mnt/hpe-lt 
Filesystem      Size  Used Avail Use% Mounted on
hpe-lt:/         72G   43G   30G  60% /mnt/hpe-lt

CURRENT suspicion is, that 'gssproxy' on fedora-33 is broken.

Comment 1 Simo Sorce 2021-02-22 14:05:12 UTC
This sound like the client has cached a ticket for key #1, if that key was removed from the server side (where gssproxy is), then no amount of changing keys will help. The solution is to clear the clients cache, and set up policies to *not* replace older keys once a new key is issued.
If the server machine was reinstalled then your only options are either to reduce the lifetime of client tickets to reduce the timeframe of this occurrences, or to save and restore keys.

Closing for now, reopen if you ahve evidence that this is not the explanation.

Comment 2 Wolfgang Szoecs 2021-02-23 12:07:07 UTC
YES - you were right :-)
After your comment - I started to focus on the questionable clients!
I've first reduced the ticket-lifetimes on the ipa-server to a low time.
(via: 'ipa krbtpolicy-mod --maxlife=3600 --maxrenew=18000')
then cleaned out again (did this before too, but no luck) the client's krb-stuff (kdestroy -A)
restarted gssproxy and rpc.gssd - and now it works ! - I can indeed now mount the fedora server from clients.