Bug 993046

I have tested the same AD integration against latest BigBend ISO today and faced the same issue.
To see if RHEL 6.4 latest have the issue too, I tested against RHEL6.4 and everything is working fine with it.

After working through this issue, I am kind of positive that winbind is the culprit here. Winbindd is responsible for getting the SID from AD and converting to UID/GID and it failing to do that here.  

Check the winbindd log from the BigBend RHS node below. Check for "Could not fetch our SID - did we join?" below. I don't see similar log for Anshi or RHEL 6.4.

cat /var/log/samba/log.winbindd

[2013/08/06 05:10:35,  0] winbindd/winbindd.c:1376(main)
  winbindd version 3.6.9-156.2.el6rhs started.
  Copyright Andrew Tridgell and the Samba Team 1992-2011
[2013/08/06 05:10:35.050191,  0] winbindd/winbindd_cache.c:3168(initialize_winbindd_cache)
  initialize_winbindd_cache: clearing cache and re-creating with version number 2
[2013/08/06 05:10:35.164180,  0] winbindd/winbindd_util.c:635(init_domain_list)
  Could not fetch our SID - did we join?
[2013/08/06 05:10:35.164233,  0] winbindd/winbindd.c:1136(winbindd_register_handlers)
  unable to initialize domain list
[2013/08/06 05:11:47,  0] winbindd/winbindd.c:1376(main)
  winbindd version 3.6.9-156.2.el6rhs started.
  Copyright Andrew Tridgell and the Samba Team 1992-2011
[2013/08/06 05:11:47.214599,  0] winbindd/winbindd_cache.c:3168(initialize_winbindd_cache)
  initialize_winbindd_cache: clearing cache and re-creating with version number 2
[2013/08/06 05:21:31.380774,  0] winbindd/winbindd.c:240(winbindd_sig_term_handler)
  Got sig[15] terminate (is_parent=1)
[2013/08/06 05:26:06,  0] winbindd/winbindd.c:1376(main)
  winbindd version 3.6.9-156.2.el6rhs started.
  Copyright Andrew Tridgell and the Samba Team 1992-2011
[2013/08/06 05:26:06.267879,  0] winbindd/winbindd_cache.c:3168(initialize_winbindd_cache)
  initialize_winbindd_cache: clearing cache and re-creating with version number 2

Just two thing which comes to my mind which should be checked. Please make sure that winbind is listed in /etc/nsswitch.conf for passwd and group.

It is not recommended to use 'winbind use default domain = yes', see man smb.conf for details. Setting it to 'yes' might force you to use just 'domain users' instead of `RHSQE-DC+domain users' with chgrp.

Hei Sumit,

I tried the two things you mentioned. I have copied the commands, output and copied the contents of nsswitch.conf which has winbind. Let me know if yoy want me to try something else.

#############################################

net join -S rhsqe-dc1 -U administrator
Enter administrator's password:

Using short domain name -- RHSQE-DC
Joined 'BIGBEND-1' to dns domain 'rhsqe-dc.com'
No DNS domain configured for bigbend-1. Unable to perform DNS Update.
DNS update failed!

[root@BigBend-1 ~]# net ads testjoin
Join is OK
[root@BigBend-1 ~]# wbinfo -u
administrator
guest
krbtgt
hobbit1
[root@BigBend-1 ~]# wbinfo -g
winrmremotewmiusers__
domain computers
domain controllers
schema admins
enterprise admins
cert publishers
domain admins
domain users
domain guests
group policy creator owners
ras and ias servers
allowed rodc password replication group
denied rodc password replication group
read-only domain controllers
enterprise read-only domain controllers
cloneable domain controllers
dnsadmins
dnsupdateproxy
hobbits

[root@BigBend-1 ~]# chgrp "domain users" /mnt/samba/gfs-vol1/rhsdata01
chgrp: invalid group: `domain users'

In /etc/nsswitch.conf I have below entries

passwd:     files winbind
shadow:     files winbind
group:      files winbind

Can you attach the output of 'strace -f -s 128 chgrp "domain users" /mnt/samba/gfs-vol1/rhsdata0' ?

Does 6.4 work with the same AD server or a different one? I'm asking because you idmap configuration expects that the POSIX UIDs and GIDs are set in the AD site in rfc2307 attributes. Does the 'domain user' group on your AD server has a gidNumber attribute?

Can you check with "wbinfo -n 'domain users'" and "wbinfo -Y S-....." if winbind can translate the name to a SID and then the SID to a GID?

Created attachment 783700 [details]
strace log while running chgrp command

Hei Sumit,

I have attached the strace log for the chgrp command  in my previous comment. 

yes, the RHEL 6.4 works with the same AD domin which rhs2.1 i.e. BigBend server is joined. I have only one AD server and one domain.

Not sure if 'domain user' group on the AD server has a gidNumber. I tried to check but couldn't confirm it. But the "wbinfo -n 'domain users'" and "wbinfo -Y <GID>" gives more information about it as "wbinfo -Y <GID>" fails on rhs2.1 server but passes on RHEL 6.4.

Below commands/output are From rhs2.1
####################################
[root@BigBend-1 ~]# wbinfo -n 'domain users'
S-1-5-21-218531155-2581107591-316442423-513 SID_DOM_GROUP (2)


[root@BigBend-1 ~]# wbinfo -Y S-1-5-21-218531155-2581107591-316442423-513
failed to call wbcSidToGid: WBC_ERR_DOMAIN_NOT_FOUND
Could not convert sid S-1-5-21-218531155-2581107591-316442423-513 to gid

Below commands/output are From RHEL6.4
###################################
[root@RHEL6 ~]# wbinfo -n 'domain users'
S-1-5-21-218531155-2581107591-316442423-513 SID_DOM_GROUP (2)

[root@RHEL6 ~]# wbinfo -Y S-1-5-21-218531155-2581107591-316442423-513
16777222

So it seems winbind is not able to convert sid to gid on the RHS2.1/BigBend server

Can you set 'log level = 10' in smb.conf, stop winbind, remove winbind logs, start winbind, call "wbinfo -Y S-1-5-21-218531155-2581107591-316442423-513" and then attach the winbind log files to the ticket?

Sumit,

Today I tried on few permutations of settings on smb.conf in rhs2.1 server and I come to conclusion that if I remove "idmap config RHSQE-DC:backend = ad" from the smb.conf the "wbinfo -Y S-1-5-21-218531155-2581107591-316442423-513" and chgrp command "chgrp "RHSQE-DC+domain users" /mnt/samba/gfs-vol1/rhsdata01/" works fine.

If I don't mention  "idmap config RHSQE-DC:backend = ad" in smb.conf idmap by default uses tbd as database. I am not sure if "ad" should also work as back end database for idmap. I followed these settings (e.g. rfc2307, default domain yes and ad database) by referring the AD integration document from redhat reference architecture team i.e. https://access.redhat.com/site/articles/410303.

So the below settings in smb.conf works fine  

[global]
#--authconfig--start-line--

# Generated by authconfig on 2013/08/07 09:06:12
# DO NOT EDIT THIS SECTION (delimited by --start-line--/--end-line--)
# Any modification may be deleted or altered by authconfig in future

   workgroup = RHSQE-DC
   password server = RHSQE-DC1.RHSQE-DC.COM
   realm = RHSQE-DC.COM
   security = ads
   netbios name = rhs21ISoTest

   idmap uid = 10000-19999
   idmap gid = 10000-19999
   idmap config RHSQE-DC:default = yes
   idmap config RHSQE-DC:range = 10000000-19999999

   idmap config RHSQE-DC:schema_mode = rfc2307
   winbind nss info = rfc2307

   winbind enum users = false
   winbind enum groups = false
   winbind separator = +
   winbind use default domain = yes
   winbind nested groups = yes

   template shell = /bin/bash
   winbind offline logon = false
#--authconfig--end-line--

##########################

With the above smb.conf file , I getting the gid of "domain users" gid as 10002 and sometimes 10006 for same AD if I redo the settings on the rhs2.1 server.

[root@rhs21ISoTest ~]# wbinfo -Y S-1-5-21-218531155-2581107591-316442423-513
10002

#re-do the settings (I mean stopping samba, winbind, remove cached tbd files, do a net join to ad, start winbind and samba. I have mentioned the steps in details at the end)

[root@rhs21ISoTest ~]# wbinfo -Y S-1-5-21-218531155-2581107591-316442423-513
10006

is it a bug? I am not able to find the reason behind the gid change. Kindly let me know.

Also below are the steps I perform after any smb.conf change. Kindly let me know if it is fine.

 1. Change smb.conf, vi /etc/samba/smb.conf
 2. service smb stop;service winbind stop
 3. rm -f /var/lib/samba/*
 4. kdestroy;klist
 5. net join -S rhsqe-dc1 -U administrator
 6. service winbind start;service smb start

If you guys think that removing "idmap config RHSQE-DC:backend = ad" is not a big issue, we can remove "TestBlocker" from the bug

I would not recommend to use the tdb idmap backend. When running in a cluster node might have a different set of POSIX UIDs and GIDs. Additionally when the related tdb is deleted, new IDs are assigned, this is what you see and it is expected behavior for the tdb backend.

Please try with the AD backend again and attach the winbind logs.

Created attachment 784074 [details]
winbind logs


Below are the steps performed for collecting the logs

service smb stop;service winbind stop
cd /var/log/samba
mv log.winbindd* old/
rm -f /var/lib/samba/*
kdestroy;klist
net join -S rhsqe-dc1 -U administrator
service winbind start;service smb start
wbinfo -Y S-1-5-21-218531155-2581107591-316442423-513
tar cvzf winbindd-logs.tar.gz log.winbindd*

I can see below logs in log.winbindd for "wbinfo -Y S-1-5-21-218531155-2581107591-316442423-513"


                  sid                      : S-1-5-21-218531155-2581107591-316442423-513
[2013/08/07 14:41:02.426741,  1] ../librpc/ndr/ndr.c:284(ndr_print_function_debug)
       wbint_Sid2Gid: struct wbint_Sid2Gid
          out: struct wbint_Sid2Gid
              gid                      : *
                  gid                      : 0x0000000000000000 (0)
              result                   : NT_STATUS_NONE_MAPPED
[2013/08/07 14:41:02.426877,  5] winbindd/winbindd_sid_to_gid.c:90(winbindd_sid_to_gid_recv)
  Could not convert sid S-1-5-21-218531155-2581107591-316442423-513: NT_STATUS_NONE_MAPPED
[2013/08/07 14:41:02.426942, 10] winbindd/winbindd.c:707(wb_request_done)
  wb_request_done[4326:SID_TO_GID]: NT_STATUS_NONE_MAPPED

And also in log.winbindd-idmap

[2013/08/07 14:41:02.426352, 10] winbindd/idmap_util.c:268(idmap_sid_to_gid)
  sid [S-1-5-21-218531155-2581107591-316442423-513] is not mapped
[2013/08/07 14:41:02.426404, 10] lib/gencache.c:183(gencache_set_data_blob)
  Adding cache entry with key = IDMAP/SID2GID/S-1-5-21-218531155-2581107591-316442423-513 and timeout = Wed Aug  7 14:43:02 2013
   (120 seconds ahead)
[2013/08/07 14:41:02.426495,  1] ../librpc/ndr/ndr.c:284(ndr_print_function_debug)
       wbint_Sid2Gid: struct wbint_Sid2Gid
          out: struct wbint_Sid2Gid
              gid                      : *
                  gid                      : 0x0000000000000000 (0)
              result                   : NT_STATUS_NONE_MAPPED
[2013/08/07 14:41:02.426611,  4] winbindd/winbindd_dual.c:1317(child_handler)
  Finished processing child request 59
[2013/08/07 14:41:02.426647, 10] winbindd/winbindd_dual.c:1333(child_handler)
  Writing 3508 bytes to parent

For the logs I get the impression that the POSIX ID attributes are missing here. Can you check on AD with 'Active Directory Users and Computers' if the group has the 'Unix Attributes' tab and if a GID is set there?

If there is no 'Unix Attributes' tab, then the Unix services are not installed on the server. In this case I would recommend to use the idmap rid backend for testing. This gives consistent UIDs and GIDs on multiple servers (if the idmap configuration in smb.conf is the same) and does not require any settings on the AD side. It still would make me wonder why it works with 6.4.

I checked the AD and as you have suspected the Unix attributes were missing. Because in Windows 2012(AD server) the NIS server i.e. " Identity Management for UNIX"  is deprecated and not installed by default. 

I have installed the " Identity Management for UNIX"/"Unix attributes" by following below technet article. 

http://technet.microsoft.com/en-us/library/cc731178.aspx#BKMK_command

Also explicitly added the "domain users" to the NIS group, assigned GID, cleared the cache from the RHS node and checked the  below command to check if it can convert the SID to GID. However it failed again with a new error. 

wbinfo -Y S-1-5-21-218531155-2581107591-316442423-513
failed to call wbcSidToGid: WBC_ERR_DOMAIN_NOT_FOUND
Could not convert sid S-1-5-21-218531155-2581107591-316442423-513 to gid

I also tried rid as the idmap "rid" back-end (i.e. idmap config RHSQE-DC:backend = rid
" and works fine (see below)

[root@rhs21ISoTest ~]# wbinfo -n 'domain users'
S-1-5-21-218531155-2581107591-316442423-513 SID_DOM_GROUP (2)
[root@rhs21ISoTest ~]# wbinfo -Y S-1-5-21-218531155-2581107591-316442423-513
10000513

Regrading the RHEL 6.4, the smb.conf doesn't have idmap "ad" backend settings as the document I referred  did not suggest those entries for RHEL 6.4. The other reason is, when I started with the integration I was not much aware of these entries meaning for smb.conf, Hence never thought a single entry would make so much difference. I will also check in RHEL 6.4 with ad backend

Since the GID is now 10000 you might need to tune 'idmap config RHSQE-DC:range = 10000000-19999999' accordingly for the AD backend. But if the RID backend works for you I would suggest to use this for further testing, since it does not require and modifications in AD.

Yup, after changing it from 
"'idmap config RHSQE-DC:range = 10000000-19999999"  to "idmap config RHSQE-DC:range = 10000-19999999" it works fine now. 

[root@rhs21ISoTest ~]# wbinfo -Y S-1-5-21-218531155-2581107591-316442423-513
10000

chgrp command is working fine as expected

[root@rhs21ISoTest ~]# chgrp "domain users" /mnt/samba/gfs-vol1/rhsdata01/
[root@rhs21ISoTest ~]# echo $?
0

I also agree with using rid as beck-end as it does not require change on AD. I think we can remove the test blocker,but we need to document the preferred back-end as rid and  also the steps if somebody wants use "ad" as back-end (i.e. for win 2012 server, NIS need to be installed explicitly and "domain users"/required groups should be added to the NIS domain)

I will also test it against win2008 AD server to check how it fairs with it.

Below are the conclusions we can draw from the bug. 

For Active directory integration of RHS 2.1 with Windows 2012 server we will prefer to use rid as idmap backend "idmap config RHSQE-DC:backend = rid". The default back-end tbd is not recommended, even though it works on some configuration.

For customers already using "Identity Management for UNIX" the AD back-end would be useful. For other/new customers the RID back-end is easier to set up.

To use "ad" as idmap back-end we need to  install  the "Identity Management for UNIX"/"Unix attributes" on the AD (win server 2012) and the required  groups/users should be added to the NIS domain. Also equivalent changes about the GID range should be modified in smb.conf. I not sure how it works in Windows 2008 Active Directory. 



Following technet article can be referred for installing  "Identity Management for UNIX" on Windows AD server . 
http://technet.microsoft.com/en-us/library/cc731178.aspx#BKMK_command

Per 08/16 scrum-of-scrum meeting, not a blocker.

We have communicated our findings to the original authors of https://access.redhat.com/site/articles/410303 from the reference architecture team.

I don't think we have any action item pending on this bug. hence closing this