According to:
  :: [  BEGIN   ] :: Running 'getent group testgroup1 > /tmp/group.txt'
  :: [   PASS   ] :: Command 'getent group testgroup1 > /tmp/group.txt' (Expected 0, got 0)
  :: [  BEGIN   ] :: Running 'cat /tmp/group.txt'
  testgroup1:*:90001:testuser1

testuser1 is a member of testgroup1 on the nis client. 

He is also owner of the file:

:: [  BEGIN   ] :: chown file to another group :: actually running 'chown testuser1:testuser1 /tmp/mytestfile.user1'
:: [   PASS   ] :: chown file to another group (Expected 0, got 0)

So I don't know why followning fails:

:: [  BEGIN   ] :: chown file to another group :: actually running 'su - testuser1 -c 'chown testuser1:testgroup1 /tmp/mytestfile.user1''
su: warning: cannot change directory to /home/testuser1: No such file or directory
chown: changing ownership of `/tmp/mytestfile.user1': Operation not permitted

might be good to verify the file rights before running the second chown/chgrp with

ls -lZ /tmp/mytestfile.user1'

I haven't seen any AVC in the test output so it, most likely, isn't caused by SELinux.

Might be also worth to run. `id testuser1`

Petr,
I see that the selinux permissions are correct with respect to the file created in /tmp folder and also the id command returns the output.

:: [  BEGIN   ] :: Running 'cat /tmp/group.txt'
testgroup1:*:90001:testuser1
:: [   PASS   ] :: Command 'cat /tmp/group.txt' (Expected 0, got 0)
:: [  BEGIN   ] :: Running 'id testuser1 > /tmp/iduser.txt'
:: [   PASS   ] :: Command 'id testuser1 > /tmp/iduser.txt' (Expected 0, got 0)
:: [  BEGIN   ] :: Running 'cat /tmp/iduser.txt'
uid=56678(testuser1) gid=56678(testuser1) groups=90001(testgroup1),56678(testuser1)
:: [   PASS   ] :: Command 'cat /tmp/iduser.txt' (Expected 0, got 0)
:: [  BEGIN   ] :: touch new file as existing user :: actually running 'touch /tmp/mytestfile.user1'
:: [   PASS   ] :: touch new file as existing user (Expected 0, got 0)
:: [  BEGIN   ] :: chown file to another group :: actually running 'chown testuser1:testuser1 /tmp/mytestfile.user1'
:: [   PASS   ] :: chown file to another group (Expected 0, got 0)
:: [  BEGIN   ] :: Running 'ls -lZ /tmp/mytestfile.user1'
-rw-r--r--. testuser1 testuser1 unconfined_u:object_r:user_tmp_t:s0 /tmp/mytestfile.user1
:: [   PASS   ] :: Command 'ls -lZ /tmp/mytestfile.user1' (Expected 0, got 0)
-rw-r--r--. testuser1 testuser1 unconfined_u:object_r:user_tmp_t:s0 /tmp/mytestfile.user1
:: [   PASS   ] :: Command 'ls -lZ /tmp/mytestfile.user1' (Expected 0, got 0)
:: [  BEGIN   ] :: chown file to another group :: actually running 'su - testuser1 -c 'chown testuser1:testgroup1 /tmp/mytestfile.user1''
su: warning: cannot change directory to /home/testuser1: No such file or directory
chown: changing ownership of `/tmp/mytestfile.user1': Operation not permitted
:: [   FAIL   ] :: chown file to another group (Expected 0, got 1)

seems to me that IPA behaves correctly

(In reply to Petr Vobornik from comment #10)
> seems to me that IPA behaves correctly

What exactly is seen as a regression in coreutils?

Do we have any reproducer independent of ipa?

(In reply to Kamil Dudka from comment #11)
> (In reply to Petr Vobornik from comment #10)
> > seems to me that IPA behaves correctly
> 
> What exactly is seen as a regression in coreutils?

Identity information looks to be resolved correctly. Ownership seems to be good as well. But the chown and chgrp commands fail. 

> Do we have any reproducer independent of ipa?

I'm not aware of any reproducer except the one in comment 5.

(In reply to Petr Vobornik from comment #12)
> Identity information looks to be resolved correctly. Ownership seems to be
> good as well. But the chown and chgrp commands fail.

Are those commands run by root (or chgrp by file's owner)?

What are the working and non-working versions of the coreutils package?

> I'm not aware of any reproducer except the one in comment 5.

We need a finite sequence of commands that reproduce the bug.  Comment #5 contains only links to beaker tasks, which hardly classifies as a reproducer.

Petr/Kamil
Will update the bug with findings after having a manual run done once again.

Ok, thanks, restoring needinfo so it is clear it waits for an input from you.

What exactly is the problem with coreutils here?

Are you saying that chgrp fails with "Operation not permitted" even though it is permitted to change the group?

How are you confirming it is a bug of chgrp?

Kamil,

We haven't confirmed this is a bug of chgrp specifically yet.  I don't know where the bug actually is yet.

It appears though the issue is resolved if we clear the cache of either nscd or sssd.  Since it seems to affect both, I'm not sure where the problem is.  

Our tests first run with host being NIS client running NSCD.  To clear the cache there before testing, I tried the following:

nscd -i passwd
nscd -i group

The second set of tests run with host being an IPA client running SSSD instead of NSCD.  To clear that cache I ran

service sssd stop; rm -rf /var/lib/sss/{db,mc}/*; service sssd start

I tried several timeouts before the tests using sleep but, that didn't seem to help.

Is there a way we can get more information from the chown to find out what happened?  Would strace help here?

Thanks,
Scott

(In reply to Scott Poore from comment #17)
> Is there a way we can get more information from the chown to find out what
> happened?  Would strace help here?

If chgrp returns EPERM, it most likely means that the actual *chown* syscall returned EPERM.  You can check by strace that it is actually the case but I would be surprised if not.

Assuming the following command is being debugged:
# su - testuser1 -c 'chgrp testgroup1 /tmp/mytestfile.user1'
chgrp: changing group of ‘/tmp/mytestfile.user1’: Operation not permitted

... it would be useful to get output of the following commands before running the chgrp command:

# su - testuser1 -c 'id'
# su - testuser1 -c 'ls -lZ /tmp/mytestfile.user1'
# su - testuser1 -c 'getent passwd testuser1'
# su - testuser1 -c 'getent group testgroup1'
# su - testuser1 -c 'getent group testuser1'

Also, have you tried to run the test(s) with SELinux in permissive mode?

Kamil,

Thanks for the quick reply.  I added your suggestions to the test to gather more info.  As you can see below, the user is a part of the group but, this cannot yet be seen by id.

I'm removing the extra beakerlib output and the su warning about missing homedir but, here are the results:

# touch /tmp/mytestfile.user1

# chown testuser1:testuser1 /tmp/mytestfile.user1

# su - testuser1 -c 'id'

uid=56678(testuser1) gid=56678(testuser1) groups=56678(testuser1) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

# su - testuser1 -c 'ls -lZ /tmp/mytestfile.user1'

-rw-r--r--. testuser1 testuser1 unconfined_u:object_r:user_tmp_t:s0 /tmp/mytestfile.user1

# su - testuser1 -c 'getent passwd testuser1'

testuser1:*:56678:56678:NIS USER:/home/testuser1:/bin/sh

# su - testuser1 -c 'getent group testgroup1'

testgroup1:*:90001:testuser1

# su - testuser1 -c 'getent group testuser1'

testuser1:*:56678:

# su - testuser1 -c 'chgrp testgroup1 /tmp/mytestfile.user1'

chgrp: changing group of `/tmp/mytestfile.user1': Operation not permitted

I would think that I'm hitting a caching issue but, these tests haven't changed in years and, this is the first time we've seen this issue.

(In reply to Scott Poore from comment #19)
> I added your suggestions to the test to gather more info.

Thank you for gathering the info!

> [...] the user is a part of the group but, this cannot yet be seen by id.

That seems to be the root cause of the problem, at least from coreutils point of view.  chgrp works as intended: non-root users can set group ownership only to groups they belong to.

For some reason, the session identity does not reflect the actual state of group database (the output of 'getent group testgroup1').  I would suggest to debug the actual NSS (Name Service Switch) plug-in that serves the group database.  You can get the list of them, sorted by precedence, by the following command:

$ grep ^group /etc/nsswitch.conf

> I would think that I'm hitting a caching issue but, these tests haven't
> changed in years and, this is the first time we've seen this issue.

I am not aware of any related change in coreutils either.

The coreutils package works as expected (see comment #19 for an evidence).  If you think it does not, please specify what exactly the problem with coreutils is.  I am switching the component back to ipa for further investigation...

Kamil,

Sorry, I had started reproducing this a couple days ago to get back to you and got sidetracked.

During the first failure, the client is not yet configured as an IPA client.  It is a NIS client with the IPA master service NIS maps.

So, nsswitch.conf has this for group:

group:      files nis

since the data is coming from nis, doesn't that mean it's coming through nscd if running?

I just logged in two days after the failure and I see the expected groups with su/id:

[root@ibm-himalaya ~]# su - testuser1 -c 'id'
su: warning: cannot change directory to /home/testuser1: No such file or directory
uid=56678(testuser1) gid=56678(testuser1) groups=56678(testuser1),90001(testgroup1) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

So, now what?  strace id in the test to capture everything that it's doing?

(In reply to Scott Poore from comment #22)
> since the data is coming from nis, doesn't that mean it's coming through
> nscd if running?

Could be.  I am not the best person to ask about nscd.  I would suggest to try it out without nscd running.  If it makes the issue go away, it is likely nscd what causes the session to lack the supplementary group.

Created attachment 1181125 [details]
strace of id command

This is the command run to gather this strace info:

su - testuser1 -c 'strace -vtfo /tmp/missing_group.strace id'

This id command should have shown more than one user.   It does show looking up from nscd.   If I remember correctly, I did try with nscd off and things worked as expected.  That is why I tested with invalidating the nscd passwd and group caches.  That does resolve the problem but, that was never needed before.

I would think this may be an nscd issue but, the same thing happens with sssd as well when the host is changed from NIS client to an IPA client.

Scott,
it looks like something is trying to lookup the group before it is created and made available over NIS.

This would explain a negative cache in NSCD (or an initgroup cache in SSSD when su - is used instead of full user authentication) and the results you are seeing.
It would also explain why purging the caches fixes it, as it forces both nscd and sssd to re-read directly from the source.

Can you verify that the test scripts haven't been changed to query for the missing group before it is actually created on the server and made available via LDAP/NIS ?

Simo,

A little more digging though and it does turn out that I found other cases in the past of workarounds being added (but not always merged in so I couldn't see it in the log).

I think I've been looking at this wrong.  The problem isn't the group.  it's the user.

The group is not referenced at all before the set of tests where the failure occurs.  And then it's blocked so it has to wait until it's added on master to run.  The user is the same name as used in the previous set of tests run right before this.  But, that set cleans up and then the set failing re-creates the user.  But, since the first set did not include the group, that's what's still cached at the time.

So, I think this is just a test bug.  And from my discovery today, possibly one we've been working around for longer than I was aware.

I'm closing this one NOTABUG.

Sorry for the wild goose chase, guys.

Thanks,
Scott