Bug 1514241 - [Regression] gssproxy-0.7.0-24.fc27 breaks NFS4 krb5i mounts
Summary: [Regression] gssproxy-0.7.0-24.fc27 breaks NFS4 krb5i mounts
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: gssproxy
Version: 27
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
Assignee: Robbie Harwood
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1507817
TreeView+ depends on / blocked
 
Reported: 2017-11-16 22:09 UTC by James
Modified: 2017-12-13 20:29 UTC (History)
10 users (show)

Fixed In Version: gssproxy-0.7.0-25.fc27
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-05 15:14:15 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)
vagrant reproducer (17.54 KB, text/plain)
2017-12-07 09:20 UTC, rob.verduijn
no flags Details

Description James 2017-11-16 22:09:23 UTC
With gssproxy-0.7.0-24.fc27 NFS4 mounts with Krb5 security fail with the server denying access.

Works OK after I downgrade to gssproxy-0.7.0-22.fc27.

Also works if I manually stop and restart gssproxy.

Comment 1 Robbie Harwood 2017-11-16 22:33:08 UTC
Setting component and blocks to match the rhel bug this is cloned from.

Comment 2 Steve Dickson 2017-11-20 15:55:24 UTC
(In reply to Robbie Harwood from comment #1)
> Setting component and blocks to match the rhel bug this is cloned from.

I don't understand how this is a nfs-utils problem when downgrading
gssproxy fixes the problem. The rpcgssd has not changed in a while... 
So how is this an nfs-utils problem?

Comment 3 Simo Sorce 2017-11-20 20:45:32 UTC
Steve,
ever heard of latent bugs triggered by another component change ?

Comment 4 Steve Dickson 2017-11-21 12:07:24 UTC
(In reply to Simo Sorce from comment #3)
> Steve,
> ever heard of latent bugs triggered by another component change ?

So something is changed in one package which breaks something in 
another package and that is a latent bug? Hmm...

Comment 5 louisgtwo 2017-11-30 02:25:09 UTC
I can confirm, downgrading to gssproxy-0.7.0-22.fc27 restores mounting nfs4 with krb5. Both server and client is fc27. Let me know how to debug this.

Comment 6 Steve Dickson 2017-11-30 21:53:36 UTC
(In reply to louisgtwo from comment #5)
> I can confirm, downgrading to gssproxy-0.7.0-22.fc27 restores mounting nfs4
> with krb5. Both server and client is fc27. Let me know how to debug this.

Just curious... What KDC are you using?

Comment 7 louisgtwo 2017-11-30 22:12:50 UTC
krb5-server-1.15.2-4.fc27.x86_64

Comment 8 Steve Dickson 2017-11-30 22:21:00 UTC
(In reply to louisgtwo from comment #7)
> krb5-server-1.15.2-4.fc27.x86_64

Thank you!

Comment 9 louisgtwo 2017-12-01 01:44:26 UTC
Just doing some testing and found out the server running gssproxy-0.7.0-24.fc27.x86_64 client running gssproxy-0.7.0-22.fc27.x86_64 and everything is fine. as soon as I upgrade the client to gssproxy-0.7.0-24.fc27.x86_64 nfs4 with krb5 stops working.

Comment 10 Robbie Harwood 2017-12-01 20:21:24 UTC
Please re-test with -25 and without KCM (yum erase sssd-kcm).  Thanks!

Comment 11 louisgtwo 2017-12-01 21:25:08 UTC
My system (fedora-workstation) does not have sssd-kcm installed. I've upgraded to -25 on both server and workstation and so far so good. NFS mounts working normally

Comment 12 James 2017-12-02 00:37:08 UTC
(In reply to Robbie Harwood from comment #10)
> Please re-test with -25 and without KCM (yum erase sssd-kcm).  Thanks!

What should GSS_USE_PROXY be in /etc/sysconfig/nfs for testing -25?

Comment 13 Robbie Harwood 2017-12-02 15:23:56 UTC
(In reply to James from comment #12)
> (In reply to Robbie Harwood from comment #10)
> > Please re-test with -25 and without KCM (yum erase sssd-kcm).  Thanks!
> 
> What should GSS_USE_PROXY be in /etc/sysconfig/nfs for testing -25?

GSS_USE_PROXY=yes always when testing gssproxy; otherwise, gssproxy isn't involved and you just end up testing something else instead.  Please remove sssd-kcm prior to testing and restart rpc-gssd after upgrading gssproxy.

Thanks louisgtwo!  Looks like this is my bug then, so taking it.

Comment 14 Robbie Harwood 2017-12-05 15:14:15 UTC
Bodhi has landed, so assuming this is fixed unless I hear otherwise.

Comment 15 Anthony Messina 2017-12-06 02:12:57 UTC
Robbie, I've tried gssproxy-0.7.0-25.fc27 (https://bodhi.fedoraproject.org/updates/FEDORA-2017-cb5743bcb0) and it doesn't seem to resolve this issue for me.

I am however using sssd-kcm.  In comment 10 you mentioned erasing sssd-kcm.  Is gssproxy/nfs/krb5 not going to work when using sssd-kcm?  If so are there plans to address this (as sssd-kcm became the default),

Comment 16 Robbie Harwood 2017-12-06 17:44:44 UTC
(In reply to Anthony Messina from comment #15)
> Robbie, I've tried gssproxy-0.7.0-25.fc27
> (https://bodhi.fedoraproject.org/updates/FEDORA-2017-cb5743bcb0) and it
> doesn't seem to resolve this issue for me.
> 
> I am however using sssd-kcm.  In comment 10 you mentioned erasing sssd-kcm. 
> Is gssproxy/nfs/krb5 not going to work when using sssd-kcm?  If so are there
> plans to address this (as sssd-kcm became the default),

gssproxy can't be responsible for KCM-specific bugs.  https://bugzilla.redhat.com/show_bug.cgi?id=1521110 is the bug I have filed for the failure I find when trying to use KCM.

Comment 17 rob.verduijn 2017-12-07 09:20:28 UTC
Created attachment 1364119 [details]
vagrant reproducer

vagrant reproducer

Comment 18 rob.verduijn 2017-12-07 09:24:52 UTC
Hello,

I've added a reproducer that proves this is still an issue on fedora27

on the fedoraclient:

using gssproxy 0.7.0-25.fc27
With GSS_USE_PROXY="yes" in /etc/sysconfig/nfs

try mounting the homes share on media with 
mount -overs=4,rw,async,noatime,timeo=14,soft,sec=krb5p,acl 192.168.122.3:/homes /media

it will fail 

change With GSS_USE_PROXY="no" in /etc/sysconfig/nfs
reboot and try again 
mount -overs=4,rw,async,noatime,timeo=14,soft,sec=krb5p,acl 192.168.122.3:/homes /media

it will succeed


the vagrant setup is kinda heavy so bring up the vms one at a time.
vagrant up ipaserver && vagrant up nfsserver && vagrant up centosclient && vagrant up fedoraclient

the centosclient is to validate that the nfs setup is working ok

accounts in this setup are : root/admin/vagrant/testuser
all passwords are : centos74

Rob Verduijn

Comment 19 rob.verduijn 2017-12-07 09:26:03 UTC
using vagrant 2.0.1-1 on centos (because it is broken on fedora)

Rob Verduijn

Comment 20 rob.verduijn 2017-12-07 09:27:57 UTC
using vagrant 2.0.1-1 on centos (because it is broken on fedora)

Rob Verduijn

Comment 21 Robbie Harwood 2017-12-08 19:32:04 UTC
Hi Rob, thanks for the very detailed reproducer.  Unforuntately I'm having some issues with vagrant at the moment.  (I'm not really sure what's up with the eth1 management you're doing, among other things.)

Does the problem continue to manifest if you restart rpc-gssd after updating gssproxy?

(The gssproxy update affects what we call the "mechglue" - it's a shim that runs in the process of applications consuming GSSAPI, like rpc-gssd does.)

Comment 22 James 2017-12-08 19:38:56 UTC
There's an element of 'stale state' to this problem. I just upgraded a machine to 
gssproxy-0.7.0-26.fc27.x86_64 without any service restarts and immediately rebooted to find I had no NFS4 mounts. I reverted back to -25 and rebooted, still nothing. I then restart rpc-gssd, gssproxy and autofs and it came back, surviving reboots.

I then upgraded to -26 again but this time emptied /var/lib/gssproxy/clients/ before rebooting. It's working now. (This is with USE_GSS_PROXY="yes".)

Comment 23 rob.verduijn 2017-12-09 12:46:32 UTC
Hello,

After running  

yum update -y --enablerepo=updates-testing gssproxy

and 
systemctl restart gssproxy.service

The command
mount -overs=4,rw,async,noatime,timeo=14,soft,sec=krb5p,acl 192.168.122.3:/homes /media

Still gives
mount.nfs: access denied by server while mounting 192.168.122.3:/homes

A reboot doesn't help either.


I've found that booting 4 vagrant systems at the same time gives quite a few issues, therefore I start them one at the time
And that if you are not using root you need to add
domain.uri = 'qemu_tcp://<fqdn of vagrant host>/system' 
to the :libvirt section of each system.

The eth1 stuff is a trick to make the vagrant guests use the ipa server as their dns service (that way you can use the example.com domain for ipa)
And I like eth1 better on the command line than 'Wired connection 1'
Also I forgo all the issues that might come from raceconditions dhcp by setting up static ip addresses on eth1 for each machine
That and my lack of knowledge on how to manipulate vagrant network settings is more or less the reason that I do all that eth1 stuff.

Rob

Comment 24 rob.verduijn 2017-12-09 12:47:45 UTC
ps if resources are a problem on your vagrant host, do not start the centosclient.
It's only function is to validate the working of the kerberized nfs exports.

Rob

Comment 25 Robbie Harwood 2017-12-12 18:43:19 UTC
Right, so what I was asking you to do was `service rpc-gssd restart` after you run the `yum update`.

Comment 26 rob.verduijn 2017-12-12 21:13:44 UTC
Hi,

Seems I misread your question.

After updateing the system again (now at gssproxy-0.7.0-26)
The command `service rpc-gssd restart` did help.

But only once.
After a reboot I could not get the mount to work again no matter how often I issued 'service rpc-gssd restart' 

Rob Verduijn

Comment 27 Robbie Harwood 2017-12-13 19:44:51 UTC
Hi Rob, I have a new version of gssproxy that may fix this issue.  Could you test gssproxy-0.7.0-29 and let me know if it works for you?  Thanks!

Comment 28 rob.verduijn 2017-12-13 20:29:22 UTC
Hello,

I pulled it from the build system since it wasn't in testing yet.
https://koji.fedoraproject.org/koji/buildinfo?buildID=1009320

That version seems to be working fine on my system.
with GSS_USE_PROXY="yes" in /etc/sysconfig/nfs

I can now mount the kerberized share again with :
mount -overs=4,rw,async,noatime,timeo=14,soft,sec=krb5p,acl 192.168.122.3:/homes /media

Thanx

Rob


Note You need to log in before you can comment on or make changes to this bug.