RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2033405 - Samba Upgrade Breaks Shares
Summary: Samba Upgrade Breaks Shares
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: samba
Version: 8.4
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Andreas Schneider
QA Contact: sssd-qe
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-16 18:00 UTC by Jeff Scrivener
Modified: 2021-12-18 10:30 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-12-18 10:30:07 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-106041 0 None None None 2021-12-16 18:00:59 UTC

Description Jeff Scrivener 2021-12-16 18:00:00 UTC
Description of problem:
After upgrade of Samba to 4.14.5-7.el8_5 from 4.14.5-2.el8, all shares on system ceased to function.  This persists through smb service restart and reboots.  Shares are accessed using SSS backend from windows workstations.  Searching logs, etc. did not reveal any errors to work with.

dnf history undo, and going back to previous version of samba corrects the issue.

Any help that can be given to track down the issue would be appreciated.  I'll be happy to work with, and try suggestions.  For now, I am rolled back to 4.14.5-2.el8. 

Version-Release number of selected component (if applicable):
4.14.5-7.el8_5

How reproducible:
Upgrade samba components on system, no longer able to access shares defined on system.

Steps to Reproduce:
1. Upgrade Samba:

Updating Subscription Management repositories.
Transaction ID : 89
Begin time     : Thu 16 Dec 2021 03:02:02 AM CST
Begin rpmdb    : 849:f3cb6e0c795a9c0f8368a36cbd1776fdb435bd62
End time       : Thu 16 Dec 2021 03:02:14 AM CST (12 seconds)
End rpmdb      : 849:1aa00524ab38933e0c3a097e6030297666975525
User           : root <root>
Return-Code    : Success
Releasever     : 8
Command Line   : --skip-broken -y update --security
Comment        :
Packages Altered:
    Upgrade  samba-common-tools-4.14.5-7.el8_5.x86_64    @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-common-tools-4.14.5-2.el8.x86_64      @@System
    Upgrade  libwbclient-4.14.5-7.el8_5.x86_64           @rhel-8-for-x86_64-baseos-rpms
    Upgraded libwbclient-4.14.5-2.el8.x86_64             @@System
    Upgrade  samba-client-4.14.5-7.el8_5.x86_64          @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-client-4.14.5-2.el8.x86_64            @@System
    Upgrade  samba-client-libs-4.14.5-7.el8_5.x86_64     @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-client-libs-4.14.5-2.el8.x86_64       @@System
    Upgrade  libsmbclient-4.14.5-7.el8_5.x86_64          @rhel-8-for-x86_64-baseos-rpms
    Upgraded libsmbclient-4.14.5-2.el8.x86_64            @@System
    Upgrade  samba-winbind-4.14.5-7.el8_5.x86_64         @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-winbind-4.14.5-2.el8.x86_64           @@System
    Upgrade  samba-common-libs-4.14.5-7.el8_5.x86_64     @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-common-libs-4.14.5-2.el8.x86_64       @@System
    Upgrade  samba-winbind-modules-4.14.5-7.el8_5.x86_64 @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-winbind-modules-4.14.5-2.el8.x86_64   @@System
    Upgrade  samba-libs-4.14.5-7.el8_5.x86_64            @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-libs-4.14.5-2.el8.x86_64              @@System
    Upgrade  samba-common-4.14.5-7.el8_5.noarch          @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-common-4.14.5-2.el8.noarch            @@System
    Upgrade  samba-4.14.5-7.el8_5.x86_64                 @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-4.14.5-2.el8.x86_64                   @@System

2.  Testing SMB deny's access to all shares defined in /etc/samba/smb.conf

3.  Downgrade samba:

Updating Subscription Management repositories.
Transaction ID : 90
Begin time     : Thu 16 Dec 2021 11:38:13 AM CST
Begin rpmdb    : 849:1aa00524ab38933e0c3a097e6030297666975525
End time       : Thu 16 Dec 2021 11:38:19 AM CST (6 seconds)
End rpmdb      : 849:f3cb6e0c795a9c0f8368a36cbd1776fdb435bd62
User           : root <root>
Return-Code    : Success
Releasever     : 8
Command Line   : history undo 89
Comment        :
Packages Altered:
    Downgrade  samba-common-tools-4.14.5-2.el8.x86_64      @rhel-8-for-x86_64-baseos-rpms
    Downgraded samba-common-tools-4.14.5-7.el8_5.x86_64    @@System
    Downgrade  samba-common-libs-4.14.5-2.el8.x86_64       @rhel-8-for-x86_64-baseos-rpms
    Downgraded samba-common-libs-4.14.5-7.el8_5.x86_64     @@System
    Downgrade  samba-winbind-modules-4.14.5-2.el8.x86_64   @rhel-8-for-x86_64-baseos-rpms
    Downgraded samba-winbind-modules-4.14.5-7.el8_5.x86_64 @@System
    Downgrade  samba-client-libs-4.14.5-2.el8.x86_64       @rhel-8-for-x86_64-baseos-rpms
    Downgraded samba-client-libs-4.14.5-7.el8_5.x86_64     @@System
    Downgrade  samba-4.14.5-2.el8.x86_64                   @rhel-8-for-x86_64-baseos-rpms
    Downgraded samba-4.14.5-7.el8_5.x86_64                 @@System
    Downgrade  libsmbclient-4.14.5-2.el8.x86_64            @rhel-8-for-x86_64-baseos-rpms
    Downgraded libsmbclient-4.14.5-7.el8_5.x86_64          @@System
    Downgrade  samba-client-4.14.5-2.el8.x86_64            @rhel-8-for-x86_64-baseos-rpms
    Downgraded samba-client-4.14.5-7.el8_5.x86_64          @@System
    Downgrade  samba-winbind-4.14.5-2.el8.x86_64           @rhel-8-for-x86_64-baseos-rpms
    Downgraded samba-winbind-4.14.5-7.el8_5.x86_64         @@System
    Downgrade  samba-common-4.14.5-2.el8.noarch            @rhel-8-for-x86_64-baseos-rpms
    Downgraded samba-common-4.14.5-7.el8_5.noarch          @@System
    Downgrade  libwbclient-4.14.5-2.el8.x86_64             @rhel-8-for-x86_64-baseos-rpms
    Downgraded libwbclient-4.14.5-7.el8_5.x86_64           @@System
    Downgrade  samba-libs-4.14.5-2.el8.x86_64              @rhel-8-for-x86_64-baseos-rpms
    Downgraded samba-libs-4.14.5-7.el8_5.x86_64            @@System

4.  Shares work, without reboot or configuration change.

Actual results:
Shares no longer work after upgrade

Expected results:
Shares continue to work after upgrade

Additional info:

Comment 1 Dennixx 2021-12-17 10:30:56 UTC
We had exactly the same issue on RHEL7, updating 4.10.16-15.el7_9 to 4.10.16-17.el7_9.

Comment 2 Jeff Scrivener 2021-12-17 16:43:55 UTC
The more I research it, the more it appears that the newer version of Samba is not playing well with SSSD.  Samba is looking for winbind to do the security, while we have general authentication setup for SSSD.

Comment 3 Alexander Bokovoy 2021-12-17 16:55:47 UTC
Samba requires running winbindd process since 2018 or so. This is part of the Samba suite.

https://access.redhat.com/articles/4355391 explains this:

------------------------------

A file server capabilities can be enabled on a domain member running under Red Hat Enterprise Linux with the help of Samba suite. Samba suite, when running as a domain member, starts two daemons:

smbd, the main process which handles network connections, file system operations, and remote procedure calls like LSA and NETLOGON. Each connection is handled by a separate smbd child;

winbindd, a process to perform identity resolution for all configured and known domains. Active connection to a domain is handled by a separate winbindd child. winbindd processes connect to domain controllers and perform required LSA and NETLOGON operations against them. Normally, authentication of a user from a trusted domain is delegated to the domain member`s own domain controller which then forwards it further.

Winbind uses a set of identity mapping modules collectively called idmap modules in Samba terminology. Each idmap module represents a strategy to map security identifiers (SIDs) from Active Directory to corresponding POSIX IDs. Since SID namespace in Active Directory is common for all kinds of objects and POSIX ID name space is separate for users and groups, with both POSIX ID name spaces being smaller than a common SID name space, there exist multiple approaches to perform the translation. A choice of a translation method is tightly connected with a specific deployment configuration.

It is possible to configure Red Hat Enterprise Linux system to use winbindd for both system level POSIX IDs retrieval and file server operations in Samba suite. Thanks to winbindd idmap modules support, it is also possible to configure it to look up system identities via SSSD. In the latter case winbindd daemon would still be running but will not provide NSS and PAM services.

-----------------------------

So if you are not using winbindd but running Samba as a domain member, you should reconsider your configuration. In most cases this simply means to enable winbind systemd service.

Comment 4 Jeff Scrivener 2021-12-17 17:02:07 UTC
Our current configuration does have winbind running, but we're using idmap backend sss:

[global]
    realm = FQDN_DOMAIN
    workgroup = DOMAIN
    security = ads
    kerberos method = system keytab
    template homedir = /home/%U
    idmap config * : backend = tdb
    idmap config * :  range = 10000-199999
    idmap config DOMAIN : backend = sss
    idmap config DOMAIN : range = 200000-2147483647
    load printers = no
    printing = bsd
    printcap name = /dev/null
    disable spoolss = yes
    machine password timeout = 0

After the upgrade, winbind enters a failed state and cannot be started again.  Given no other configuration changes, this is suspect.  So far in my testing, the only success I've had is to move away from SSSD as my authentication agent for all realm authentication, and move to winbind.  Am I hearing that SSSD authentication, with Samba winbind is unsupported?

Comment 5 Alexander Bokovoy 2021-12-17 17:09:52 UTC
No, that's not correct.

The configuration you have should work with both SSSD and winbindd running. We do test this with IPA and trust to AD, for example.

http://freeipa-org-pr-ci.s3-website.eu-central-1.amazonaws.com/jobs/a7006178-5e00-11ec-a932-fa163e684033/report.html is an example of a successful run of such configuration -- sorry, I don't have RHEL test run public, this is Fedora Rawhide with essentially the same configs.

For IPA case this is automated with 'ipa-client-samba' tool. How it looks internally is described in https://freeipa.readthedocs.io/en/latest/designs/adtrust/samba-domain-member.html. For direct AD integration it is basically the same and your configuration should work too.

What exactly does winbindd report on start? Please add 'log level = 10' to see the debug info.

Comment 6 Jeff Scrivener 2021-12-17 17:51:10 UTC
I've done a lot of fiddling with it, and have it in quite the mess presently  I'll revert this to snapshot and grab some clean before and after logs later this afternoon.  Thanks for taking time to look into this with us. :)

Comment 7 Jeff Scrivener 2021-12-17 19:27:26 UTC
Alexander,

Thanks for pointing me to look at my debug logs (I should have done that before I posted the bug report).  Turns out I had a config issue that did not manifest itself as a failure until the upgrade.  Given the change to correct CVE-2020-25717, it's not surprising that the config error broke things.

My logging in winbind indicated the following error:
winbind Could not fetch our SID - did we join?

Turns out winbind was NOT running, nor do I think it ever was.  I reverted to snapshot, and ultimately a few different versions of smb.conf, sssd.conf, realmd.conf from backup.  I found that in my only working config for 4.14.5-2, I had this commented out in smb.conf:

"# security = ads"

If I uncommented that and attempted to start winbind, it would fall down.  If I commented it, winbind would start, but it had lots of errors, including "did we join?"  None-the-less, the Samba shares would work without winbind on the older version, but not with the newer.

Ultimately what I did to fix the issue:

I backed up my smb.conf, sssd.conf, realmd.conf files, then removed the system from the domain, then re-added it:

# realm leave
# net ads join -U domain_admin_user
# adcli join domain.fqdn -U domain_admin_user

I did both net ads join and adcli join to get both the winbind and sssd components registered.  I then copied my backed up sssd.conf to /etc/sssd.

# cp sssd.conf /etc/sssd/sssd.conf
# systemctl restart winbind sssd smb

After verifying that all services would start clean, and stay up, combined with the ability to access my shares, I then took the plunge and upgraded again:

Packages Altered:
    Upgrade  samba-common-tools-4.14.5-7.el8_5.x86_64    @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-common-tools-4.14.5-2.el8.x86_64      @@System
    Upgrade  libwbclient-4.14.5-7.el8_5.x86_64           @rhel-8-for-x86_64-baseos-rpms
    Upgraded libwbclient-4.14.5-2.el8.x86_64             @@System
    Upgrade  samba-client-4.14.5-7.el8_5.x86_64          @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-client-4.14.5-2.el8.x86_64            @@System
    Upgrade  samba-client-libs-4.14.5-7.el8_5.x86_64     @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-client-libs-4.14.5-2.el8.x86_64       @@System
    Upgrade  libsmbclient-4.14.5-7.el8_5.x86_64          @rhel-8-for-x86_64-baseos-rpms
    Upgraded libsmbclient-4.14.5-2.el8.x86_64            @@System
    Upgrade  samba-winbind-4.14.5-7.el8_5.x86_64         @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-winbind-4.14.5-2.el8.x86_64           @@System
    Upgrade  samba-common-libs-4.14.5-7.el8_5.x86_64     @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-common-libs-4.14.5-2.el8.x86_64       @@System
    Upgrade  samba-winbind-modules-4.14.5-7.el8_5.x86_64 @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-winbind-modules-4.14.5-2.el8.x86_64   @@System
    Upgrade  samba-libs-4.14.5-7.el8_5.x86_64            @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-libs-4.14.5-2.el8.x86_64              @@System
    Upgrade  samba-common-4.14.5-7.el8_5.noarch          @rhel-8-for-x86_64-baseos-rpms
    Upgraded samba-common-4.14.5-2.el8.noarch            @@System

After the upgrade I restarted all three services I tested again, and everything worked.  After, I did a reboot... but then I could no longer log in with a domain account OR access the shares???!!

Logged in as a non-domain account and checked, and both the sssd and winbind services were disabled for some reason.  I enabled them, did another reboot, and now we're cooking with fire!

Below are my sssd, samba.conf (global) and realmd.conf files, sanitized of course.  Hopefully this will help someone else out if they are having an issue.  From my perspective, this was and is a non-bug and can be closed. :)

Thanks,
Jeff

======================================================

# cat /etc/sssd/sssd.conf

[sssd]
domains = domain.fqdn
config_file_version = 2
services = nss, pam
default_domain_suffix = domain.fqdn

[nss]
homedir_substring = /home

[pam]

[domain/domain.fqdn]
ad_domain = domain.fqdn
krb5_realm = DOMAIN.FQDN
realmd_tags = manages-system joined-with-samba
cache_credentials = True
id_provider = ad
krb5_store_password_if_offline = True
default_shell = /bin/bash
ldap_id_mapping = False
use_fully_qualified_names = True
fallback_homedir = /home/%u
access_provider = simple
full_name_format = %1$s
simple_allow_groups = admin_group_here

======================================================

# ​cat /etc/realmd.conf
[providers]
   ​samba = no

[active-directory]
   ​default-client = sssd
   ​# default-client = winbind

======================================================

[global]
   ​realm = DOMAIN.FQDN
   ​workgroup = domain
   ​security = ads
   ​kerberos method = system keytab
   ​template homedir = /home/%U
   ​idmap config * : backend = tbd
   ​idmap config * :  range = 10000-199999
   ​idmap config domain : backend = sss
   ​idmap config domain : range = 200000-2147483647
   ​load printers = no
   ​printing = bsd
   ​printcap name = /dev/null
   ​disable spoolss = yes
   ​machine password timeout = 0
#   log level = 10

Comment 8 Alexander Bokovoy 2021-12-18 10:30:07 UTC
As per comment #7 closing as NOTABUG


Note You need to log in before you can comment on or make changes to this bug.