Bug 1942443

Summary: Login using password failed after upgrade to Fedora 34
Product: [Fedora] Fedora Reporter: Benny Amorsen <benny+bugzilla>
Component: authselectAssignee: Pavel Březina <pbrezina>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 34CC: alciregi, awilliam, bberg, bcotton, caillon+fedoraproject, droidbittin, fedora, fhirsch4, gmarr, gnome-sig, jcolding, jhrozek, joss.sparkes, kparal, lruzicka, mattias.eriksson, mcatanza, mclasen, m.schwartau, mwolf, pasik, pbrezina, peter.hutterer, philip.wyett, renault, rhughes, robatino, rstrode, tiyicot958
Target Milestone: ---Keywords: CommonBugs, Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard: AcceptedFreezeException RejectedBlocker https://fedoraproject.org/wiki/Common_F34_bugs#login-authselect-fingerprint-error
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-13 08:35:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1829024, 1829025    
Attachments:
Description Flags
journalctl -b
none
List of installed packaged
none
/etc/authselect/nsswitch.conf
none
Output from removal of fprintd-pam none

Description Benny Amorsen 2021-03-24 11:44:03 UTC
Created attachment 1765904 [details]
journalctl -b

Description of problem:
At the GDM prompt, selecting my user results in "Sorry, fingerprint authentication didn't work. Please try again."

Providing the correct password results in a slight delay, then the same message repeats.

Version-Release number of selected component (if applicable):
gdm-40~rc-1.fc34.x86_64

How reproducible:
Happens every time (but I have not tried a restart)

Steps to Reproduce:
1. Have Fedora 33 on a device with a fingerprint reader, but no fingerprints added.
2. Upgrade to Fedora 34 using dnf system-upgrade
3. Try to log in with a password

Actual results:
"Sorry, fingerprint authentication didn't work. Please try again." (even before providing a password, but also after providing one).

Expected results:
Successful login.

Additional info:

Strangely, switching to a different virtual console works. Login does not try to use the fingerprint reader, and the password is accepted. This is how I am able to write this bug report.

I have never set up fingerprints; I am not a huge fan of single-factor biometric  authentication.

Comment 1 Kamil Páral 2021-03-24 12:10:46 UTC
This sounds very serious, marking as a proposed blocker, in case we can debug/reproduce this.

Comment 2 Benny Amorsen 2021-03-24 12:49:52 UTC
Created attachment 1765925 [details]
List of installed packaged

Comment 3 Benny Amorsen 2021-03-24 12:51:32 UTC
Removing fprintd-pam (which autoremoves libfprint and fprintd) resolves the problem.

Comment 4 Benny Amorsen 2021-03-24 12:55:29 UTC
When removing fprintd:

[error] [/etc/authselect/nsswitch.conf] has unexpected content!
[error] Unexpected changes to the configuration were detected.
[error] Refusing to activate profile unless those changes are removed or overwrite is requested.
Unable to disable feature [17]: File exists

I will attach nsswitch.conf

Comment 5 Benny Amorsen 2021-03-24 12:57:22 UTC
Created attachment 1765927 [details]
/etc/authselect/nsswitch.conf

Comment 6 Benny Amorsen 2021-03-24 13:00:00 UTC
Created attachment 1765928 [details]
Output from removal of fprintd-pam

That one happens reliably if I undo and redo the removal of fprintd-pam.

Comment 7 mattias.eriksson 2021-03-24 13:44:19 UTC
I see simmilar issues, but with the lock screen. After I lock the screen I either see a blank screen with no photo, name and such. I can input but it will not succeed. In other cases I see the proper lock screen, but both password and fingerprint fails. To login after the screen is locked, I have to click the switch user, select the same user and login.

Comment 8 Meinert Schwartau 2021-03-27 07:42:55 UTC
Had the same issue using a Lenovo Thinkpad P15 Gen1. Before I had Fedora silverblue 33 installed and no fingerprint sensor configured (the option just does not appear in fedora 33 silverblue even though it is available in workstation for my laptop). After upgrading to fedora silverblue 34 I was unable to login because of the mentioned fingerprint error message which reappeared in some form of a loop every few seconds and deleted the entered password. Entering the password fast enough and pressing entered before it was deleted by the "Sorry, fingerprint authentication didn't work. Please try again." did not work out too.

Working fedora silverblue 33 Deployment:
Version: 33.20210326.0 (2021-03-26T00:52:21Z)
                BaseCommit: da512b217d2b1dac0dfd23ba8e433b7f00fce3ab2d5e65f3dac68ab646e22bcc
              GPGSignature: Valid signature by 963A2BEB02009608FE67EA4249FD77499570FF31
           LayeredPackages: akmod-nvidia fedora-workstation-repositories podman-compose
                            xorg-x11-drv-nvidia zsh
             LocalPackages: rpmfusion-nonfree-release-33-1.noarch
                            rpmfusion-free-release-33-1.noarch
                    Pinned: yes

Fedora 34 Deployment where I was unable to login:
ostree://fedora:fedora/34/x86_64/silverblue
                   Version: 34.20210326.n.0 (2021-03-26T08:08:45Z)
                BaseCommit: 61e157e8e9dc56962b2881e51c4578d082d9c10eee51f58ba1ad1bf4a731f8ce
              GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39
                      Diff: 1245 upgraded, 13 downgraded, 61 removed, 42 added
           LayeredPackages: akmod-nvidia fedora-workstation-repositories podman-compose
                            xorg-x11-drv-nvidia zsh
             LocalPackages: rpmfusion-nonfree-release-34-0.2.noarch
                            rpmfusion-free-release-34-0.3.noarch

Comment 9 Meinert Schwartau 2021-03-27 09:30:38 UTC
Tried to remove fprintd-pam from the fedora silverblue 34 base image as mentioned above:

ostree://fedora:fedora/34/x86_64/silverblue
                   Version: 34.20210326.n.0 (2021-03-26T08:08:45Z)
                BaseCommit: 61e157e8e9dc56962b2881e51c4578d082d9c10eee51f58ba1ad1bf4a731f8ce
              GPGSignature: Valid signature by 8C5BA6990BDB26E19F2A1A801161AE6945719A39
                      Diff: 1245 upgraded, 13 downgraded, 61 removed, 41 added
       RemovedBasePackages: fprintd-pam 1.90.9-2.fc34
           LayeredPackages: akmod-nvidia fedora-workstation-repositories podman-compose
                            xorg-x11-drv-nvidia zsh
             LocalPackages: rpmfusion-nonfree-release-34-0.2.noarch
                            rpmfusion-free-release-34-0.3.noarch


But I still got the "fingerprint" error and was unable to login after reboot.

Comment 10 Meinert Schwartau 2021-03-27 16:57:08 UTC
I was able to login now:
- Switched to Screen Keyboard 
- With Screen keyboard enabled the problem disappeared 
- Entered password
- Created fingerprint

Now having a fingerprint registered I am able to login without problems.

Comment 11 Adam Williamson 2021-03-29 16:25:59 UTC
Can affected folks say what versions of authselect and authselect-libs are installed? Thanks!

Comment 12 Meinert Schwartau 2021-03-29 17:26:18 UTC
On my system I have the following versions installed:

> rpm -qa | grep authsel
authselect-libs-1.2.2-6.fc34.x86_64
authselect-1.2.2-6.fc34.x86_64

Comment 13 Benjamin Berg 2021-03-29 18:03:20 UTC
Could you try running "authselect apply-changes"?

https://bugzilla.redhat.com/show_bug.cgi?id=1933520#c46 suggests that we are dealing with local modifications to /etc/nsswitch.conf (or some other files), which prevent the pam configuration to be updated and fixed.

Comment 14 Geoffrey Marr 2021-03-29 18:32:19 UTC
Discussed during the 2021-03-29 blocker review meeting: [0]

The decision to delay the classification of this as a blocker bug was made as this seems serious, but benzea believes it's the same as #1933520 and should have been fixed already. If it is not, we need to figure out under what circumstances that fix isn't working before we can decide if the impact is wide enough to be a blocker.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2021-03-29/f34-blocker-review.2021-03-29-16.00.txt

Comment 15 Benny Amorsen 2021-03-29 18:51:15 UTC
authselect apply-changes
[error] [/etc/authselect/nsswitch.conf] has unexpected content!
[error] Unexpected changes to the configuration were detected.
[error] Refusing to activate profile unless those changes are removed or overwrite is requested.
Some unexpected changes to the configuration were detected. Use 'select' command instead.

Comment 16 Benny Amorsen 2021-03-29 18:52:41 UTC
I have the same versions of authselect as Meinert Schwartau.

rpm -qa | grep authsel
authselect-libs-1.2.2-6.fc34.x86_64
authselect-1.2.2-6.fc34.x86_64
authselect-compat-1.2.2-6.fc34.x86_64

Comment 17 Meinert Schwartau 2021-03-29 19:05:27 UTC
When I ran the command I got the following output:

> sudo authselect apply-changes
No existing configuration detected.

Comment 18 Meinert Schwartau 2021-03-31 06:44:29 UTC
As I said, after registering a fingerprint for my user the problem disappeared. But I added a new user so that I have two users now. I just found out that if I click "switch user":
- and click on the user for whom I added a fingerprint I see the message "(or place finger on reader)" and am able to login without hassle
- and click on the newly created user I get the "Sorry, fingerprint ..." error again. 

The newly created user was a admin user with password provided.

Comment 19 Pavel Březina 2021-03-31 14:03:07 UTC
> [error] [/etc/authselect/nsswitch.conf] has unexpected content!
> [error] Unexpected changes to the configuration were detected.
> [error] Refusing to activate profile unless those changes are removed or overwrite is requested.

This means that nsswitch.conf was modified outside authselect therefore authselect will not touch it so it does not overwrite user changes. The file that you attached looks like default glibc. Is there perhaps any /etc/nsswitch.conf.bak or .rpmold?

> No existing configuration detected.

This means that you never opt-in into authselect configuration.

To recover from  both issues and restore system default use:
  authselect select sssd with-fingerprint with-silent-lastlog --force

fprintd should expect possible authselect failure and produce a graceful error message that would be more helpful in fprintd context instead of the authselect error.

Comment 20 Meinert Schwartau 2021-03-31 16:45:15 UTC
(In reply to Pavel Březina from comment #19)
> > [error] [/etc/authselect/nsswitch.conf] has unexpected content!
> > [error] Unexpected changes to the configuration were detected.
> > [error] Refusing to activate profile unless those changes are removed or overwrite is requested.
> 
> This means that nsswitch.conf was modified outside authselect therefore
> authselect will not touch it so it does not overwrite user changes. The file
> that you attached looks like default glibc. Is there perhaps any
> /etc/nsswitch.conf.bak or .rpmold?
> 
> > No existing configuration detected.
> 
> This means that you never opt-in into authselect configuration.
> 
> To recover from  both issues and restore system default use:
>   authselect select sssd with-fingerprint with-silent-lastlog --force
> 
> fprintd should expect possible authselect failure and produce a graceful
> error message that would be more helpful in fprintd context instead of the
> authselect error.

That fixed it :-)

Comment 21 Adam Williamson 2021-04-05 16:23:11 UTC
Pavel, so what's the plan here? I think you talked about forcing a fix in the RPM scriptlets in cases where we know the prior fix doesn't get applied? Thanks!

Comment 22 Geoffrey Marr 2021-04-05 23:46:21 UTC
Discussed during the 2021-04-05 blocker review meeting: [0]

The decision to classify this bug as a "RejectedBlocker (Final)" and an "AcceptedFreezeException (Final)" was made as the consensus was that it's just too narrow in scope to see as a release blocker (we believe it affects systems that started as Fedora 27 or earlier, and systems where nsswitch configuration was changed outside of authselect). It is accepted as a freeze exception.

[0] https://meetbot.fedoraproject.org/fedora-blocker-review/2021-04-05/f34-blocker-review.2021-04-05-16.02.txt

Comment 23 Pavel Březina 2021-04-06 08:31:54 UTC
(In reply to Adam Williamson from comment #21)
> Pavel, so what's the plan here? I think you talked about forcing a fix in

AFAIK the issue is fixed by authselect-1.2.3. If Benjamin agrees, this BZ can be closed as duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1943406

> the RPM scriptlets in cases where we know the prior fix doesn't get applied?
> Thanks!

This BZ also describes a problem when users gets authselect error message when they removed fprintd. This error message is correct and authselect failure is expected in given cases. We talked this through with Benjamin and he should have some patch ready to provide a better error message for fprintd context.

Comment 24 Benjamin Berg 2021-04-06 08:52:03 UTC
So, I was hoping to have a solution for https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/3853

Either way. I think the sensible thing here is to force-update the specific line in /etc/pam.d/fingerprint-auth from the RPM postinstall script. We'll still have a userbase that is excluded from authselect, which is really not great, but at least it is an easy way to paper over this issue.

Comment 25 Benny Amorsen 2021-04-06 09:07:38 UTC
(we believe it affects systems that started as Fedora 27 or earlier, and systems where nsswitch configuration was changed outside of authselect).

Neither of these apply to my laptop. It was installed with Fedora 32 and nsswitch configuration was not manually changed from the default.

I still do not understand why authselect refused to touch my nsswitch.conf.

Comment 26 Michael Catanzaro 2021-04-06 11:42:42 UTC
(In reply to Benny Amorsen from comment #25)
> (we believe it affects systems that started as Fedora 27 or earlier,

OK, that's our bug then.

Proposal: RPM scriptlets should just always use 'authselect -f'. If authselect is enabled, then user changes to nsswitch.conf outside authselect are invalid and should be corrected on upgrade. It may not necessarily be what the user necessarily wants to happen, but it should avoid gratuitously broken systems. (Disadvantage: if we run 'authselect -f' automatically, it kind of defeats the point of having the -f flag at all.)

Comment 27 Pavel Březina 2021-04-06 12:18:18 UTC
(In reply to Benjamin Berg from comment #24)
> So, I was hoping to have a solution for
> https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/3853
> 
> Either way. I think the sensible thing here is to force-update the specific
> line in /etc/pam.d/fingerprint-auth from the RPM postinstall script. We'll
> still have a userbase that is excluded from authselect, which is really not
> great, but at least it is an easy way to paper over this issue.

Wouldn't it be better to patch the default files in pam package?
(In reply to Benny Amorsen from comment #25)
> (we believe it affects systems that started as Fedora 27 or earlier, and
> systems where nsswitch configuration was changed outside of authselect).
> 
> Neither of these apply to my laptop. It was installed with Fedora 32 and
> nsswitch configuration was not manually changed from the default.
> 
> I still do not understand why authselect refused to touch my nsswitch.conf.

Is there any nsswitch.conf.bak|rpmnew|rpmold or something like that?

(In reply to Michael Catanzaro from comment #26)
> (In reply to Benny Amorsen from comment #25)
> > (we believe it affects systems that started as Fedora 27 or earlier,
> 
> OK, that's our bug then.
> 
> Proposal: RPM scriptlets should just always use 'authselect -f'. If
> authselect is enabled, then user changes to nsswitch.conf outside authselect
> are invalid and should be corrected on upgrade. It may not necessarily be
> what the user necessarily wants to happen, but it should avoid gratuitously
> broken systems. (Disadvantage: if we run 'authselect -f' automatically, it
> kind of defeats the point of having the -f flag at all.)

Ideally, rpm scriptlets should not touch PAM at all. It is too easy to break it and too difficult to do it right. Packages should rather document what needs to be done.

Hopefully, we can enforce authselect configuration in the future which would solve things like these.

Comment 28 Benny Amorsen 2021-04-06 12:23:30 UTC
(In reply to Pavel Březina from comment #27)

> Is there any nsswitch.conf.bak|rpmnew|rpmold or something like that?

Yes, there is an nsswitch.conf.bak. I am not sure why that is relevant.

The configuration in nsswitch.conf has come either from the default in the RPM or from authselect. Surely authselect should be able to handle either case?

Comment 29 Benjamin Berg 2021-04-07 12:29:11 UTC
So, I have an ancient installation here (currently F31, probably upgraded for longer than F27).

2019-09-30T14:59:40Z SUBDEBUG Upgraded: glibc-2.29-22.fc30.x86_64
...
2019-09-30T19:33:34Z SUBDEBUG Upgrade: glibc-2.30-5.fc31.x86_64
2019-09-30T19:33:34Z SUBDEBUG Upgrade: glibc-2.30-5.fc31.x86_64
2019-09-30T19:33:34Z INFO warning: /etc/nsswitch.conf created as /etc/nsswitch.conf.rpmnew

[root@ben-x1 log]# ls -l /etc/nsswitch.conf*
-rw-r--r--. 1 root root 1757 Mar 28  2020 /etc/nsswitch.conf
-rw-r--r--. 1 root root 1725 Mar 28  2020 /etc/nsswitch.conf.bak
-rw-r--r--. 1 root root 1652 Sep 26  2019 /etc/nsswitch.conf.rpmnew


So, 2020-03-28 the file was last modified, which coincides with an update of nss-mdns. And, looking into that package, it still contains:

function mod_nss() {
    if [ -f "$1" ] ; then
        # sed-fu to add mdns4_minimal to the hosts line of /etc/nsswitch.conf
        sed -i.bak '
            /^hosts:/ !b
            /\<mdns\(4\|6\)\?\(_minimal\)\?\>/ b
            s/\<\(files\( myhostname\)\?[[:blank:]]\+\)/\1mdns4_minimal [NOTFOUND=return] /g
            ' "$1"
    fi
}

FILE="$(readlink /etc/nsswitch.conf || echo /etc/nsswitch.conf)"
if [ "$FILE" = "/etc/authselect/nsswitch.conf" ] && authselect check &>/dev/null; then
    mod_nss "/etc/authselect/user-nsswitch.conf"
    authselect apply-changes &> /dev/null || :
else
    mod_nss "$FILE"
    # also apply the same changes to user-nsswitch.conf to affect
    # possible future authselect configuration
    mod_nss "/etc/authselect/user-nsswitch.conf"
fi

So, that probably explains where the nsswitch.conf modifications are coming from.

Comment 30 Michael Catanzaro 2021-04-07 14:13:54 UTC
Of course we see that script knows about authselect and modifies /etc/authselect/nsswitch.conf instead if authselect is in use. That means that for some reason, authselect was not enabled at the time that the nss-mdns scriptlet was run.

I also see nss-mdns didn't know about authselect until F30 though, while authselect was in use since F27. That's probably not good.

So we've established that however this happened, it's Fedora's fault, not user error. That suggests we need to run 'authselect -f' at least once, ideally when upgrading F33 -> F34, to get back to a good state.

Comment 31 Benjamin Berg 2021-04-07 15:08:29 UTC
Yeah, forcing a run of "authselect -f" (and setting the "sssd" profile if no profile is selected), seems like the right thing here in principle.

How could we do this for the F33 -> F34 upgrade? Would this happen in a script inside the authselect package or in some other location?

The only caveat I see is, that theoretically there may be users who modified some of these files intentionally. But, that seems *very* unlikely and those users should be updating their setup to work with authselect anyway.

Comment 32 Michael Catanzaro 2021-04-07 15:37:27 UTC
(In reply to Benjamin Berg from comment #31)
> How could we do this for the F33 -> F34 upgrade? Would this happen in a
> script inside the authselect package or in some other location?

An authselect package scritplet seems like the simplest and least-fragile approach.

> The only caveat I see is, that theoretically there may be users who modified
> some of these files intentionally. But, that seems *very* unlikely and those
> users should be updating their setup to work with authselect anyway.

Or just disable authselect altogether. IMO if authselect is enabled, it should always win over clashing non-authselect configuration, i.e. any Fedora package scriptlets that run authselect should always use 'authselect -f'. But users can always have final say if they want via disabling authselect.

Comment 33 Benjamin Berg 2021-04-09 10:35:51 UTC
OK, my proposal now is the following:

https://src.fedoraproject.org/rpms/authselect/pull-request/11

i.e. the run the code:
FILE="$(readlink -f %{_sysconfdir}/pam.d/fingerprint-auth || echo %{_sysconfdir}/pam.d/fingerprint-auth)"
%__grep -q '^auth[[:space:]]\+sufficient[[:space:]]\+pam_fprintd.so' $FILE && \
  %__sed -i.bak -e 's/^auth[[:space:]]\+sufficient[[:space:]]\+pam_fprintd.so\(.*\)/auth        [success=done default=bad]                   pam_fprintd.so\1/g' $FILE

Which updates the file in-place if it still has "sufficient" in it (after running the usual upgrade routines). As is, it will only run in case authselect is not functional for some reason.

Comment 34 Dmitry Tantsur 2021-04-19 10:01:02 UTC
Interestingly, I've recently had exactly the same problem on F33 (without upgrade). Could the bug get there too?

$ rpm -qa | grep authsel
authselect-libs-1.2.3-1.fc33.x86_64
authselect-1.2.3-1.fc33.x86_64
authselect-compat-1.2.3-1.fc33.x86_64

I also have nsswitch.conf.bak|rpmnew, although I don't think this laptop ever had F27 installed.

I had to un-enroll my keys from fprintd in the text console to be able to log in graphically (it's MATE, so lightdm).

Do you think it's the same issue?

Comment 35 Martin Wolf 2021-04-20 06:57:10 UTC
I updated to F34 just a few days ago and I ran into the bug as well.
I did not find this bug report till now, so I helped myself with : https://help.gnome.org/admin/system-admin-guide/stable/login-fingerprint.html.en

just to be sure, this is the fix?

  authselect select sssd with-fingerprint with-silent-lastlog --force

and I think you are right, my Notebook has Fedora installed since F24 or F25

Comment 36 Fedora Update System 2021-04-20 07:36:23 UTC
FEDORA-2021-48ca6e6b86 has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-48ca6e6b86

Comment 37 Michael Catanzaro 2021-04-20 15:26:39 UTC
(In reply to Geoffrey Marr from comment #22)
> Discussed during the 2021-04-05 blocker review meeting: [0]
> 
> The decision to classify this bug as a "RejectedBlocker (Final)" and an
> "AcceptedFreezeException (Final)" was made as the consensus was that it's
> just too narrow in scope to see as a release blocker (we believe it affects
> systems that started as Fedora 27 or earlier, and systems where nsswitch
> configuration was changed outside of authselect). It is accepted as a freeze
> exception.

I'm going to repropose this issue because we were only partially correct about the scope of the bug at the time it was classified RejectedBlocker. We since discovered that Fedora has automatically edited nsswitch configuration outside authselect at some point in the past, and therefore we expect all or most upgraded systems to be affected.

We already have a proposed solution, and I understand we're going to have another release slip anyway due to an unrelated issue, so it makes sense to not introduce an upgrade cliff when we can reasonably avoid it.

Comment 38 Michael Catanzaro 2021-04-20 15:28:11 UTC
(In reply to Fedora Update System from comment #36)
> FEDORA-2021-48ca6e6b86 has been submitted as an update to Fedora 34.
> https://bodhi.fedoraproject.org/updates/FEDORA-2021-48ca6e6b86

Ah, I guess there's no point if we have an update and a freeze exception already. Benjamin, is that update enough to close this issue?

Comment 39 Benjamin Berg 2021-04-20 15:30:52 UTC
Yes, I believe the submitted update is enough to fix this issue for all users.

Comment 40 Fedora Update System 2021-04-20 22:25:36 UTC
FEDORA-2021-48ca6e6b86 has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-48ca6e6b86`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-48ca6e6b86

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 41 Kamil Páral 2021-04-22 08:48:23 UTC
(In reply to Michael Catanzaro from comment #38)
> Ah, I guess there's no point if we have an update and a freeze exception already.

There is a point. That update is not stable and it's not verified whether it fixes the problem (btw, my home desktop system was also affected by this issue, but I fixed it manually before the update was available). Re-proposing as a blocker per comment 37.

Comment 42 Kamil Páral 2021-04-22 14:15:21 UTC
I wanted to test the fix. I modified /etc/authselect/nsswitch.conf so that `sudo authselect apply-changes` would fail with "[error] [/etc/authselect/nsswitch.conf] has unexpected content!". Then I installed the update from comment 40. However, the file was not regenerated and `sudo authselect apply-changes` still fails with the same error. Is that expected? I thought the update would force-select the right profile and regenerate the config files.

Comment 43 Benjamin Berg 2021-04-22 14:33:11 UTC
Not quite. What it does is update the file if it is not being managed by authselect.

What seems to be your case is that authselect is technically managing /etc/pam.d/fingerprint-auth, but it doesn't do anything because /etc/authselect/nsswitch.conf has been modified by a different package.

So … still stuck in a way.

Pavel, could we maybe get authselect to not refuse updating the rest if only nsswitch.conf has been modified?

Comment 44 Adam Williamson 2021-04-22 14:48:29 UTC
Is that a case that's likely to actually exist?

Comment 45 Fedora Update System 2021-04-22 14:53:47 UTC
FEDORA-2021-48ca6e6b86 has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 46 Kamil Páral 2021-04-22 15:09:39 UTC
(In reply to Adam Williamson from comment #44)
> Is that a case that's likely to actually exist?

In the past I edited (just) nsswitch.conf because nss-mdns didn't update it automatically (I think it even contained instructions that users should do so). Ever since then I carried the config file forward (I didn't even know that it should be automatically managed by authselect, until this bug). I don't know whether it could cause a login failure like demonstrated in this bug.

I'll reopen the bug because the conversation is still ongoing, so that we don't lose it from our blocker tracking systems.

Comment 47 Benjamin Berg 2021-04-22 15:33:35 UTC
Hmm, the file should have a header like:

# Generated by authselect on Fri Apr  9 11:15:22 2021
# Do not modify this file manually.

# If you want to make changes to nsswitch.conf please modify
# /etc/authselect/user-nsswitch.conf and run 'authselect apply-changes'.

Which does give the instructions that users should not manually modify nsswitch.conf, but should rather be editing /etc/authselect/user-nsswitch.conf.

Comment 48 Kamil Páral 2021-04-22 19:32:04 UTC
Yes, the message is there *now* :-) But I was mostly referring to this quote from comment 30:
> I also see nss-mdns didn't know about authselect until F30 though, while authselect was in use since F27. That's probably not good.
where nss-mdns possibly edited the file directly. And in one case it didn't (a bug I suppose), so I had to figure it out and do it manually (very vague memories here). Even if I didn't, I'd still have the config file modified by the package on my other systems.

Btw, I looked at F27 and authselect is not there. It is at F28, but the warning message contains only the first two lines from comment 47 and definitely anything about user-nsswitch.conf. It also doesn't contain mdns4_minimal in hosts:, while nss-mdns is installed by default. But mdns4_minimal is present in F29 and later. So perhaps it was F28 where I needed to manually edit nsswitch.conf? Also interestingly, nsswitch.conf.bak is present on all live images (F28 up to F34).

I don't know what to make out of all this, I hope you do :-) I'm just trying to make sure the login issues (or any other issues, stemming from diverging configs carried from previous releases) won't impact a non-trivial portion of our userbase.

If we assume that a sizeable amount of people have custom nsswitch.conf because of past nss-mdns behavior (it would be nice to have this confirmed, it might have affected just certain installation paths, like netinst, or upgrades), do you believe such situation is OK for the moment, or should we do something about it now, before F34 is out?

Comment 49 Michael Catanzaro 2021-04-22 20:40:39 UTC
(In reply to Kamil Páral from comment #48)
> If we assume that a sizeable amount of people have custom nsswitch.conf
> because of past nss-mdns behavior (it would be nice to have this confirmed,
> it might have affected just certain installation paths, like netinst, or
> upgrades), do you believe such situation is OK for the moment, or should we
> do something about it now, before F34 is out?

That's for blocker bug process to decide. Here we only need to decide how to fix the issue robustly. I stand by my earlier suggestion that RPM scriptlets should always call 'authselect -f' to clobber any user configuration made outside authselect.

Comment 50 Ben Cotton 2021-04-23 03:14:15 UTC
Anecdata: my laptop was originally installed with F28 and I have all five lines of comment 47.

Comment 51 Benjamin Berg 2021-04-23 06:04:31 UTC
(In reply to Michael Catanzaro from comment #49)
> [SNIP]. I stand by my earlier suggestion that RPM scriptlets
> should always call 'authselect -f' to clobber any user configuration made
> outside authselect.

Yes, we really really should just be doing that. Personally, my only concern is that doing so technically requires a fedora change request.

It is just a matter of time until we need to force this on all users. Doing it now rather than waiting for F35 would simplify matters a lot.

Comment 52 Lukas Ruzicka 2021-04-23 08:22:49 UTC
Well, then perhaps this would be a hot candidate for a FeSCo decision.

Comment 53 Michael Catanzaro 2021-04-23 13:27:06 UTC
Benjamin, is the existing fix sufficient to ensure users are always able to log in? Users unable to login is the blocker issue here.

If the current update is sufficient to ensure users can always log in, then 'authselect -f' can be handled later in a non-blocker bug.

(In reply to Kamil Páral from comment #42)
> I wanted to test the fix. I modified /etc/authselect/nsswitch.conf so that
> `sudo authselect apply-changes` would fail with "[error]
> [/etc/authselect/nsswitch.conf] has unexpected content!". Then I installed
> the update from comment 40. However, the file was not regenerated and `sudo
> authselect apply-changes` still fails with the same error. Is that expected?
> I thought the update would force-select the right profile and regenerate the
> config files.

Can you log in?

Comment 54 Benjamin Berg 2021-04-23 14:37:18 UTC
So … there might be some confusion, because I have personally focused on the PAM configuration issue.

We can fix the fact that you cannot login in gnome-shell. See https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/3853

I had been hoping that Marco would get around to look into it properly. Doing the change suggested in the ticket should fix the problem. People will still see a message that fingerprint auth failed, but at least they can continue after that. i.e. it would fix the release critical aspect of this issue.

Comment 55 Michael Catanzaro 2021-04-23 14:47:43 UTC
Can you do a build with a downstream patch for now, please?

Comment 56 Kamil Páral 2021-04-23 16:30:27 UTC
(In reply to Michael Catanzaro from comment #53)
> Can you log in?

In my concrete test scenario, yes. I don't know how to reproduce a broken state (especially now with the fingerprint pam update already stable).

Comment 57 Benjamin Berg 2021-04-23 16:54:56 UTC
Scratch build:
  https://koji.fedoraproject.org/koji/taskinfo?taskID=66552770

Pull request for gnome-shell package:
  https://src.fedoraproject.org/rpms/gnome-shell/pull-request/11

I am not entirely confident with just pushing this. Ideally at least Ray or Marco had a quick look whether it is sane.

Comment 58 Benjamin Berg 2021-04-23 18:02:19 UTC
OK, new scratch build:
  https://koji.fedoraproject.org/koji/taskinfo?taskID=66554820

This now has the rubber stamp from Ray. So probably good to go in. PR for the package is updated.

Upstream MR (maybe Marco will comment there):
  https://gitlab.gnome.org/GNOME/gnome-shell/-/merge_requests/1821

With this, affected users will see a message "Sorry, fingerprint authentication didn't work. Please try again." pop up multiple times. However, they can still use the login prompt without further issues.

Comment 59 Ben Cotton 2021-04-23 18:06:38 UTC
Rejected as a blocker in the (currently running) F34 Go/No-Go meeting:
We can't be sure no one will hit this, but it does not violate the criteria as written. There is a known broad spectrum fix for f34 where we just make login work even if the config is in the bad state

Comment 60 Adam Williamson 2021-04-23 19:23:57 UTC
Note, rejected as a blocker doesn't mean we don't want to fix it. It would still be very great to have a 0-day fix for this. It doesn't need any special status for that; the fix just needs to be submitted for stable by Monday or so. I will look at the PR shortly.

Comment 61 Fedora Update System 2021-04-26 20:05:39 UTC
FEDORA-2021-cd4d2064ac has been submitted as an update to Fedora 34. https://bodhi.fedoraproject.org/updates/FEDORA-2021-cd4d2064ac

Comment 62 Fedora Update System 2021-04-27 01:23:21 UTC
FEDORA-2021-cd4d2064ac has been pushed to the Fedora 34 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2021-cd4d2064ac`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2021-cd4d2064ac

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 63 Luna Jernberg 2021-04-27 06:07:38 UTC
Works great here :)

Comment 64 Fedora Update System 2021-04-27 20:35:49 UTC
FEDORA-2021-cd4d2064ac has been pushed to the Fedora 34 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 65 tiyicot958@testbnk.com 2021-04-28 04:27:31 UTC
Hello, 

I upgraded from Fedora 33 to Fedora 34, have gnome-shell-40.0-4.fc34 installed and still encounter this issue

Comment 66 Jules Colding 2021-04-28 09:33:33 UTC
I have all of the latest f34 updates installed and are still seeing this issue. Sometimes I have to reboot several times before it stops complaining about the fingerprint auth.

Comment 67 Benjamin Berg 2021-04-28 09:38:21 UTC
The fix does *not* address the underlying error of failing fingerprint auth. But it allows you to continue by simply typing in the password and hitting enter.

i.e. it looks the same, but password auth should be working now.

Comment 68 Jules Colding 2021-04-28 10:07:59 UTC
I wish it did :-) 

But it won't let me type much before it shows the fingerprint error. I've tried to out-type the error with no luck. It quickly shows the error and won't let me log in. After a few reboots I can, but it's really annoying.

Comment 69 Benjamin Berg 2021-04-28 10:13:18 UTC
Yikes, something must have messed up during my testing, and I have no idea what exactly.

You are right, the entry becomes insensitive. This should not happen, and I am not entirely sure why it does even.

Comment 70 Jules Colding 2021-04-28 10:24:27 UTC
Let me know if you need any kind of logs (and how to get them) from me, and I'll attach it here.

Comment 71 Michael Catanzaro 2021-04-28 17:09:43 UTC
Benjamin has another gnome-shell update coming, but ultimately it is just a workaround. For a proper fix, we agree that the authselect package must always run 'authselect -f' or authselect should default to -f. Trying to respect user configuration might seem nice, but it's clearly too fragile. authselect needs to win if it is enabled. Reassigning to authselect.

Comment 72 Benjamin Berg 2021-04-28 17:21:37 UTC
The new update is: https://bodhi.fedoraproject.org/updates/FEDORA-2021-734b404996

Comment 73 Kamil Páral 2021-04-29 07:45:08 UTC
(In reply to Michael Catanzaro from comment #71)
> For a proper fix, we agree that the authselect package must
> always run 'authselect -f' or authselect should default to -f. Trying to
> respect user configuration might seem nice, but it's clearly too fragile.
> authselect needs to win if it is enabled. Reassigning to authselect.

Note, it's also good to keep in mind the rpm configuration handling. On my system, I had a lot of .rpmnew files in /etc/authselect. I'm not sure how exactly the whole system works and when they are created. But if the updated config files don't overwrite the old ones, then forcing a profile update might have no effect.

Comment 74 Jules Colding 2021-04-29 17:19:29 UTC
The new update works for me. Still seeing the fingerprint error message, but I can reliably log in. At least the handful of times I tried :-)

Comment 75 Kamil Páral 2021-04-30 07:34:42 UTC
Added a CommonBugs entry:
https://fedoraproject.org/wiki/Common_F34_bugs#login-authselect-fingerprint-error

Devs, please verify that instructions are correct, thanks.

Comment 76 Benjamin Berg 2021-04-30 07:44:57 UTC
Thanks Kamil! The instructions look good to me.

Comment 77 Pavel Březina 2021-04-30 10:32:06 UTC
Benjamin, I got confused from all the comments here. Is this still an issue? Can you summarize what has been done and what is needed from authselect? Thank you.

In longterm, I will try to push and make authselect mandatory so that there are no longer collisions and we are able to deliver PAM changes to all users easily.

Comment 78 Benjamin Berg 2021-04-30 10:47:42 UTC
Kind of understandable.

So, what we have now is:
 1. We update pre-authselect configurations in the PAM package
 2. We have a workaround in gnome-shell that means that affected people only see an error message.

Unfortunately, we do have users with "authselect" configurations, where local modifications were made later on. It is not clear to me why this happens, but I suspect in most cases scripts modified e.g. /etc/nsswitch.conf. One user reported for example that they ran "ipa-client --install", which might be breaking things.

For F34, we do have a working solution with the gnome-shell update (not perfect, but really good enough).

For F35, it would be nice to have the change that makes authselect mandatory. I believe making that change is the only thing left at this point.

Comment 79 Pavel Březina 2021-04-30 11:03:11 UTC
(In reply to Benjamin Berg from comment #78)
> For F35, it would be nice to have the change that makes authselect
> mandatory. I believe making that change is the only thing left at this point.

Making it mandatory in a sense that it cannot be uninstalled/switched off so it can be safely used will require more time to prepare and unfortunately I won't have time for it now, so this is F36.

But perhaps we can make a more simple change for F35 something like if authselect package is upgraded/installed on non-authselect configured systems then enforce the default configuration.

Comment 80 Michael Catanzaro 2021-04-30 12:30:20 UTC
tbh I don't see any reason that we would need mandatory authselect.

What we need is for RPM scriptlets to use 'authselect -f' if authselect is enabled. If authselect is not enabled, that's fine. The problem is having authselect enabled with local customizations made outside authselect to the files it manages, where authselect currently breaks. We should not allow that. When authselect is enabled, user changes to the configuration files that it manages should be wiped away.