Bug 1665794

Summary: adcli update --add-samba-data does not work as expected
Product: Red Hat Enterprise Linux 7 Reporter: Glen Babiano <gbabiano>
Component: man-pages-overridesAssignee: Nikola Forró <nforro>
Status: CLOSED NEXTRELEASE QA Contact: David Jež <djez>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.6CC: djez, jinjli, nforro, pcech, rgiles, sbose, sgoveas, tscherf
Target Milestone: rcKeywords: ManPageChange
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: man-pages-overrides-7.8.0-1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-27 10:01:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1738573, 1745931    

Description Glen Babiano 2019-01-14 04:17:37 UTC
Description of problem:
Winbind daemon fails to start after joining to AD following the steps here: https://access.redhat.com/solutions/2221561
~~~
Attempts to start winbind fail.  Truncated error output is below:
... initialize_winbindd_cache
... clearing and re-creating with version number 2
... init_domain_list
... Could not fetch our SID - did we join?
... winbindd_register_handlers
... unable to initialize domain list
... Failed to start Samba Winbind Daemon
~~~

In order to work around the issue Cu has to run the following commands but it reports the error messages below:
# adcli join --domain EXAMPLE.COM --add-samba-data
secrets_prepare_password_change: secrets_fetch_or_upgrade_domain_info(TEST) failed
Unable to write the machine account password in the secrets database

Same results when using kerberos keytab
# adcli join --domain EXAMPLE.COM --add-samba-data -C
secrets_prepare_password_change: secrets_fetch_or_upgrade_domain_info(TEST) failed
Unable to write the machine account password in the secrets database

After running the command above, we can see that the secrets.tdb is updated and winbind service works fine.

However, the cu requirement is to run the "adcli update" command but it doesn't seem to fix the issue like the "adcli join" command above. And also it doesn't show any error messages.
# adcli update --add-samba-data
# echo $?
0

Here's what the cu has to say:
~~~
Given our existing system automation, having issue with "adcli update" be resolved would be a priority for us as it would allow us to have a single domain join mechanism for all RHEL servers, and then just run the adcli update command for fileservers.
~~~

Version-Release number of selected component (if applicable):
RHEL 7.6

How reproducible:
Everytime

Expected results:
The "adcli update --add-samba-data" should work as the "adcli join --add-samba-data" which updates the secrets.tdb file with the required SID from AD so that the service will start normally.

Additional info:

Comment 2 Sumit Bose 2019-01-14 06:26:05 UTC
Hi,

my guess is that even 'adcli join ...' does not work properly because of the error message.

The reason for this is that the 'net changesecretpw' command used by adcli to write the machine account password to Samba's internal database requires that the database has some initial values set (see https://bugzilla.redhat.com/show_bug.cgi?id=1653795 or https://bugzilla.samba.org/show_bug.cgi?id=13577)

To work around this please call:

    tdbtool /var/lib/samba/private/secrets.tdb store SECRETS/MACHINE_LAST_CHANGE_TIME/AD '2\00'
    tdbtool /var/lib/samba/private/secrets.tdb store SECRETS/MACHINE_PASSWORD/AD '2\00'

(tdbtool is in the tdb-tools package). There is a similar workaround in the Samba upstream ticket which does not use the trailing '\00'. Please make sure to always add '\00' at the end.

Please let me know if using this work around before calling 'adcli join' help to remove the error message and makes 'adcli update' work as well.

HTH

bye,
Sumit

Comment 3 Glen Babiano 2019-01-14 23:45:11 UTC
Hi Sumit,

Thanks for looking in to this. 

The Cu have tried your suggestion however, it did not make any difference. It is still showing up the error message as before. See results below:

Executing the commands returned the following:

~~~
tdbtool /var/lib/samba/private/secrets.tdb store SECRETS/MACHINE_LAST_CHANGE_TIME/AD '2\00'
Storing key:

key 35 bytes
SECRETS/MACHINE_LAST_CHANGE_TIME/AD
data 2 bytes
[000] 32 00

tdbtool /var/lib/samba/private/secrets.tdb store SECRETS/MACHINE_PASSWORD/AD '2\00'
Storing key:

key 27 bytes
SECRETS/MACHINE_PASSWORD/AD
data 2 bytes
[000] 32 00

kinit -k -t /path/to/keytab USERNAME
adcli join --domain DOMAIN.FQDN --add-samba-data --login-ccache
secrets_prepare_password_change: secrets_fetch_or_upgrade_domain_info(<DOMAIN>) failed
Unable to write the machine account password in the secrets database
~~~

Do you have any other recommendations?

Thanks, 

Glen

Comment 4 Sumit Bose 2019-01-15 08:37:24 UTC
Hi,

I'm sorry, I should have given some more details about the workaround.

The last component of the keys 'SECRETS/MACHINE_LAST_CHANGE_TIME/AD' and 'SECRETS/MACHINE_PASSWORD/AD' has to match the NetBIOS domain name (short domain name) of the domain you want to join to. In my example it is 'AD' but this might of course be different for your customer. Ther NetBIOS domain name is the name in braces in the 'secrets_prepare_password_change: secrets_fetch_or_upgrade_domain_info(<DOMAIN>) failed' message.

HTH

bye,
Sumit

Comment 5 Glen Babiano 2019-01-17 06:48:17 UTC
Hi Sumit,

Thanks for the clarification. 

1) Cu is still facing issues after following the steps below. Here's what the cu has to say:
~~~
Ran into problems again on a clean run-through:
* Kinit keytab
* realm join
* Generate Samba secrets DB
* Seed Samba Secrets DB
* Update Samba with SID entry (adcli update --add-samba-data --login-ccache)
* Confirm SID present (using tdbdump)
* Start Winbind - FAIL
"Could not fetch our SID - did we join"

The output of the tdbdump prior to starting winbind is as follows:
{
key(27) = "SECRETS/MACHINE_PASSWORD/<DOMAIN>"
data(2) = "2\00"
}
{
key(35) = "SECRETS/MACHINE_LAST_CHANGE_TIME/<DOMAIN>"
data(2) = "2\00"
}
{
key(21) = "SECRETS/SID/<SHORT HOSTNAME>"
data(68) = "<Misc data>"
}
~~~

2) The other thing that I want to verify with you is the content of the document below if it's the correct procedure or if it's possible to implement this at all, as this appears to be buggy as per feedback from customer.
~~~
https://access.redhat.com/solutions/2221561
~~~

Appreciate your feedback on this.


Thanks and regards,

Glen

Comment 6 Sumit Bose 2019-01-17 13:41:14 UTC
(In reply to Glen Babiano from comment #5)
> Hi Sumit,
> 
> Thanks for the clarification. 
> 
> 1) Cu is still facing issues after following the steps below. Here's what
> the cu has to say:
> ~~~
> Ran into problems again on a clean run-through:
> * Kinit keytab
> * realm join

By default on RHEL7 realmd uses 'net ads join' as membership software which should automatically create the secrets.tdb. Are you calling realm join with the --membership-software=adcli?

I understand that there are permission issues when using net ads join (https://bugzilla.redhat.com/show_bug.cgi?id=1663064). Please note that typically restricted account can only be used to add new host object but not modify existing ones. Please make sure that the host object does not exists in AD before trying to join.

In general I would recommend to use net ads join when joining a system where a Samba server should run because if will set all data needed by Samba/Winbind.

> * Generate Samba secrets DB
> * Seed Samba Secrets DB
> * Update Samba with SID entry (adcli update --add-samba-data --login-ccache)

You have to use the '--computer-password-lifetime=0' option to force the update (and setting in secrets.tdb) of the machine account password.

> * Confirm SID present (using tdbdump)
> * Start Winbind - FAIL
> "Could not fetch our SID - did we join"
> 
> The output of the tdbdump prior to starting winbind is as follows:
> {
> key(27) = "SECRETS/MACHINE_PASSWORD/<DOMAIN>"
> data(2) = "2\00"
> }
> {
> key(35) = "SECRETS/MACHINE_LAST_CHANGE_TIME/<DOMAIN>"
> data(2) = "2\00"
> }
> {
> key(21) = "SECRETS/SID/<SHORT HOSTNAME>"
> data(68) = "<Misc data>"
> }
> ~~~
> 
> 2) The other thing that I want to verify with you is the content of the
> document below if it's the correct procedure or if it's possible to
> implement this at all, as this appears to be buggy as per feedback from
> customer.
> ~~~
> https://access.redhat.com/solutions/2221561
> ~~~

yes, this document needs multiple updates. Afaik this is already WIP.
> 
> Appreciate your feedback on this.
> 
> 
> Thanks and regards,
> 
> Glen

About authentication issues after calling 'realm leave ....; adcli join ...' mentioned in the case. realmd will add sssd.conf setting during join and remove them during leave. adcli only joins and will not touch any configuration. So chances are that SSSD was not configured properly after running the two commands.

bye,
Sumit

Comment 7 Glen Babiano 2019-01-21 06:54:27 UTC
Hi Sumit,

Thanks for your assistance so far.

I have received the following feedback from customer:

1) "adcli join/update" command results do not match:
~~~
It should probably be mentioned that "adcli join --add-samba-data" creates 6 additional entries in the Samba secrets.tdb, compared to "adcli update --add-samba-data".  Adcli update and adcli join behaviour for the "-add-samba-data" flag should be identical.
~~~

2) realm join command defaults to "--membership-software=adcli" instead of "--membership-software=samba"?
~~~
** The realm permit issue (having to add extra settings to /etc/realmd.conf),
** The "adcli update --computer-password-lifetime=0" setting, 
** Samba database seeding requirement via tdbtool, 
** Needing to run "adcli join/update --add-samba-data" to update samba DB 
** Needing to run "adcli update --service-name=cifs" to update kerberos ticket

I presume some of the above points for the RHEL support article are managed internally via the realm join command, however shouldn't the realm join command be explicitly "realm join --membership-software=samba"?  In some systems (like here) it defaults to --membership-software=adcli which definitely does not perform these extra steps.
~~~

Can you please briefly expound on these? is this something that we need to raise as software bugs?

Thanks,

Glen

Comment 8 Sumit Bose 2019-01-21 10:40:21 UTC
(In reply to Glen Babiano from comment #7)
> Hi Sumit,
> 
> Thanks for your assistance so far.
> 
> I have received the following feedback from customer:
> 
> 1) "adcli join/update" command results do not match:
> ~~~
> It should probably be mentioned that "adcli join --add-samba-data" creates 6
> additional entries in the Samba secrets.tdb, compared to "adcli update
> --add-samba-data".  Adcli update and adcli join behaviour for the
> "-add-samba-data" flag should be identical.

It is expected that 'adcli join --add-samba-data' adds more data to a more or less empty secrets.tdb. Given the issue with the 'net changesecretpw' requires to add MACHINE_LAST_CHANGE_TIME and MACHINE_PASSWORD db-keys the initial secrets.tdb might look like:

# tdbdump /var/lib/samba/private/secrets.tdb                   
{                                                                                    
key(32) = "SECRETS/MACHINE_PASSWORD/DOM-067"                                                                                                                                                                                                 
data(2) = "2\00"                                                              
}                                                                                                         
{                                                                 
key(40) = "SECRETS/MACHINE_LAST_CHANGE_TIME/DOM-067"                            
data(2) = "2\00"                                                            
} 

(jfyi, if secrets.tdb does not exists you can easily create it by calling:

    echo "create /var/lib/samba/private/secrets.tdb" | tdbtool
)

If now 'adcli join --add-samba-data' there will be expected new db-keys:

key(35) = "SECRETS/MACHINE_DOMAIN_INFO/DOM-067"
key(32) = "SECRETS/MACHINE_PASSWORD/DOM-067"
key(37) = "SECRETS/MACHINE_PASSWORD.PREV/DOM-067"
key(40) = "SECRETS/MACHINE_LAST_CHANGE_TIME/DOM-067"
key(26) = "SECRETS/SID/HOST-8-251-244"
key(40) = "SECRETS/MACHINE_SEC_CHANNEL_TYPE/DOM-067"
key(68) = "SECRETS/SALTING_PRINCIPAL/DES/DOM-067.ABC.IDM.LAB.ENG.BRQ.REDHAT.COM"
key(19) = "SECRETS/SID/DOM-067"
key(27) = "SECRETS/PROTECT/IDS/DOM-067"

Since all those db-keys now already exist calling 'adcli update --add-samba-data' will no add additional db-keys but only update the value of existing db-keys.

> ~~~
> 
> 2) realm join command defaults to "--membership-software=adcli" instead of
> "--membership-software=samba"?
> ~~~
> ** The realm permit issue (having to add extra settings to /etc/realmd.conf),
> ** The "adcli update --computer-password-lifetime=0" setting, 
> ** Samba database seeding requirement via tdbtool, 
> ** Needing to run "adcli join/update --add-samba-data" to update samba DB 
> ** Needing to run "adcli update --service-name=cifs" to update kerberos
> ticket
> 
> I presume some of the above points for the RHEL support article are managed
> internally via the realm join command, however shouldn't the realm join
> command be explicitly "realm join --membership-software=samba"?  In some
> systems (like here) it defaults to --membership-software=adcli which
> definitely does not perform these extra steps.

As said before I agree that the document needs some update. Of course it would be always good to be as explicit as possible but please note that with a default RHEL-7 installation 'net ads join' is used to join the domain and the document mainly focused on this case. But I agree that it makes sense to add a section explaining what needs to be done when using adcli instead. 

> ~~~
> 
> Can you please briefly expound on these? is this something that we need to
> raise as software bugs?
> 
> Thanks,
> 
> Glen