RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1657349 - Winbind doesn't get restarted on package upgrade
Summary: Winbind doesn't get restarted on package upgrade
Keywords:
Status: CLOSED DUPLICATE of bug 1878205
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: samba
Version: 7.6
Hardware: All
OS: Linux
unspecified
urgent
Target Milestone: rc
: ---
Assignee: Andreas Schneider
QA Contact: sssd-qe
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-07 17:51 UTC by joel
Modified: 2024-06-13 22:01 UTC (History)
14 users (show)

Fixed In Version: samba-4.10.4-10.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-10-20 09:26:23 UTC
Target Upstream Version:
Embargoed:
pm-rhel: mirror+


Attachments (Terms of Use)
yum logs (20.18 KB, text/plain)
2018-12-18 21:59 UTC, joel
no flags Details

Description joel 2018-12-07 17:51:04 UTC
Description of problem:
All winbind authentication fails after updating to RHEL 7.6 immediately after winbind itself is updated.

Version-Release number of selected component (if applicable):


How reproducible:
customer has multiple servers to update and it has occurred on everyone so far

Steps to Reproduce:
1.start a looped ssh command
2.start the updating process
3.when winbind is updated the ssh connections will start to fail

Actual results:
ssh start to fail

Expected results:
winbind not to loose it's file links after update

Additional info:
From two lsof's before and after update:
$ grep libreplace-samba4 samba_bug_before.txt | grep winbindd
COMMAND    PID  TID      USER  FD        TYPE             DEVICE    SIZE/OFF  NODE      NAME
winbindd   2023          root  mem       REG              253,0     11088     101033174 /usr/lib64/samba/libreplace-samba4.so
winbindd   2024          root  mem       REG              253,0     11088     101033174 /usr/lib64/samba/libreplace-samba4.so
winbindd   2034          root  mem       REG              253,0     11088     101033174 /usr/lib64/samba/libreplace-samba4.so
winbindd   7268          root  mem       REG              253,0     11088     101033174 /usr/lib64/samba/libreplace-samba4.so
winbindd   9198          root  mem       REG              253,0     11088     101033174 /usr/lib64/samba/libreplace-samba4.so


$ grep libreplace-samba4 samba_bug_after.txt | grep winbindd
COMMAND   PID   TID      USER  FD        TYPE             DEVICE  SIZE/OFF    NODE      NAME
winbindd   1458          root  DEL       REG              253,0               101033174 /usr/lib64/samba/libreplace-samba4.so;5c0978f1
winbindd   1465          root  DEL       REG              253,0               101033174 /usr/lib64/samba/libreplace-samba4.so;5c0978f1
winbindd   1468          root  DEL       REG              253,0               101033174 /usr/lib64/samba/libreplace-samba4.so;5c0978f1
winbindd  20873          root  DEL       REG              253,0               101033174 /usr/lib64/samba/libreplace-samba4.so;5c0978f1
winbindd  20875          root  DEL       REG              253,0               101033174 /usr/lib64/samba/libreplace-samba4.so;5c0978f1

Comment 2 Andreas Schneider 2018-12-12 17:10:55 UTC
This sounds more like that when the new rpm got installed, the post script of the RPM did not restart winbind through systemd. Restarting winbind manually with systemd fixed the issue. Is that correct?

If yes, we need to look at yum logs and systemd logs why systemd was not able to restart winbindd.

Comment 3 joel 2018-12-18 21:56:22 UTC
Hello,

Yes rebooting the system fixed the issue.

I have attached the yum logs, customer does not have "systemd" logs. Can you tell me how to set that up debugging to capture that?

Comment 4 joel 2018-12-18 21:59:09 UTC
Created attachment 1515424 [details]
yum logs

Comment 5 Andreas Schneider 2019-01-04 07:47:21 UTC
The log doesn't show any error.

Comment 7 joel 2019-01-11 22:01:04 UTC
Andreas,

After winbind is updated it's files seem to loose their nodes(not sure if that is the correct way to say it)

root@RDRE1-DALEX01:~ # ls -l /usr/lib64/libwbclient.so.0
lrwxrwxrwx. 1 root root 19 Dec  6 14:35 /usr/lib64/libwbclient.so.0 -> libwbclient.so.0.14

root@RDRE1-DALEX01:~ # ls -l /usr/lib64/libwbclient.so.0.14 
lrwxrwxrwx. 1 root root 40 Dec  6 14:35 /usr/lib64/libwbclient.so.0.14 -> /etc/alternatives/libwbclient.so.0.14-64

root@RDRE1-DALEX01:~ # ls -l /etc/alternatives/libwbclient.so.0.14-64
lrwxrwxrwx. 1 root root 45 Dec  6 14:35 /etc/alternatives/libwbclient.so.0.14-64 -> /usr/lib64/samba/wbclient/libwbclient.so.0.14

root@RDRE1-DALEX01:~ # ls -l /usr/lib64/samba/wbclient/libwbclient.so.0.14 
-rwxr-xr-x. 1 root root 57472 Aug  9 09:41 /usr/lib64/samba/wbclient/libwbclient.so.0.14

Comment 8 Andreas Schneider 2019-01-15 14:55:26 UTC
This is strange. Does reinstalling the libwbclient package fix it?

Comment 9 joel 2019-01-15 18:12:51 UTC
At this point the  cu found that restarting winbind solved his issue:

1) All our RHEL7.5 machines use winbind to talk to AD (and allow people to login with windows credentials)
2) When I do a "yum update" to 7.6, right after it installs the new winbind packages, SSH just times out and so does the console login. They hang for 30+ seconds and then reset.
3) So after the binbind packages are installed, no one can access the machine.

IF however, I ssh in with a couple shells, and run "yum update", ssh and console logins still break, but the existing logins are still valid. If at that time I restart winbind, SSH and Console logins work again.

cu has implemented his own work around of adding stop|start winbind commands to his scripts, but doesn't feel this is the best solution as he has to cut all existing connections.

Comment 10 Andreas Schneider 2019-02-13 15:25:00 UTC
Moving to sssd per Sumits request.

Comment 12 Andreas Schneider 2019-11-25 11:00:57 UTC
Found the issue in our spec file.

Comment 15 Andreas Schneider 2019-12-04 14:14:55 UTC
1. Winbind isn't restarted on package upgrade that means that the system is in a non-functional state after the upgrade and users are not able to login.

2. This is just a package change that winbind gets probably restarted, there is not code change and no risk that anything breaks!

3. -

Comment 18 David Gardner 2020-04-24 15:59:15 UTC
Fails for us when upgrading from 7.7->7.8 packages in CR repo, causing all authentication to fail until rebooted or the winbind service is restarted.

Please update the upgrade scripts as this is a major bug which will break all our hosts (which use winbind for AD authentication and user info) when the 7.8 update is released.

Comment 19 David Gardner 2020-04-27 10:26:45 UTC
I don't think this is just because winbind isn't getting restarted, as it *is* getting restarted!
I believe instead that this is a problem with it not being restarted after other packages which contain libraries on which it is dependent are upgraded.

For example, the output of `lsof` on the running winbind process using :

  `lsof -p $(pidof -s /usr/sbin/winbindd) | grep lib | grep DEL`

..contains various libraries.  Picking one at random as `/usr/lib64/libsamba-passdb.so*` we can find that this is in the package samba-client-libs, which has also been upgraded.  However, the winbindd processes started a couple of minutes *before* the new libsamba-passdb.so* files were created, so it's not getting kicked when these upgrade.

On Debian-based distros there is a `needrestart` command which is used to check which services need restarting after package upgrades (or possibly before, judging by the files which are contained in the packages?) and then the restart is scheduled for once all the dependent upgrades are complete.  How can we achieve the same on RHEL/CentOS during samba upgrades?

Comment 20 David Gardner 2020-04-27 10:29:53 UTC
Ignore the first line of my previous comment -- it isn't getting restarted but the same subsequent query applies.
(I thought it was, judging by the process start time, but I had rebooted the test host just before performing the upgrade, oops! :)

Comment 21 David Gardner 2020-04-27 12:50:02 UTC
`needrestart` handles this correctly when installed and set to automatically restart services when updates are installed.  Is this the intended fix and that winbind and other daemons are *not* supposed to restart themselves automatically when they upgrade?  If so this should be documented as required.

(Can't see how `needs-restarting` can do the same at present, as needrestart as a yum plugin to support this)

Comment 22 David Gardner 2020-04-28 10:31:37 UTC
After a flurry of upgrades, it turns out that smbd also doesn't restart cleanly when upgraded, and needs a kick...

Comment 23 joel 2020-11-12 21:58:24 UTC
Hello,

Has there been any progress on this issue?

Comment 26 Andreas Schneider 2021-10-20 09:26:23 UTC

*** This bug has been marked as a duplicate of bug 1878205 ***


Note You need to log in before you can comment on or make changes to this bug.