The root cause is that user_groups variable is computed only once (user_groupslen == -1):

uidswap.c:

 97         /* set and save the user's groups */
 98         if (user_groupslen == -1) {
 99                 if (initgroups(pw->pw_name, pw->pw_gid) < 0)
100                         fatal("initgroups: %s: %.100s", pw->pw_name,
101                             strerror(errno));
102 
103                 user_groupslen = getgroups(0, NULL);
104                 if (user_groupslen < 0)
105                         fatal("getgroups: %.100s", strerror(errno));
106                 if (user_groupslen > 0) {
107                         user_groups = xreallocarray(user_groups,
108                             user_groupslen, sizeof(gid_t));
109                         if (getgroups(user_groupslen, user_groups) < 0)
110                                 fatal("getgroups: %.100s", strerror(errno));
111                 } else { /* user_groupslen == 0 */
112                         free(user_groups);
113                 }
114         }

In the scenario, the first time temporarily_use_uid() is called is when switching the parent sshd process to "nobody" after forking the AuthorizedKeysCommand program:

pam_ssh_agent_auth-0.10.3/pam_user_key_allowed2.c:

219     debug("Running AuthorizedKeysCommand: \"%s\" as \"%s\" with argument: \"%s\"",
220                           authorized_keys_command, pw->pw_name, username);
...
228     switch ((pid = fork())) {
...
283     temporarily_use_uid(pw);
284 
285     close(p[1]);
...

EDIT: First part is ok, Second isn't.

The first call being made is here, before forking in fact:

auth2-pubkey.c:

 398 static pid_t
 399 subprocess(const char *tag, struct passwd *pw, const char *command,
 400     int ac, char **av, FILE **child)
 401 {
...
 412         debug3("%s: %s command \"%s\" running as %s", __func__,
 413             tag, command, pw->pw_name);
 414 
 415         /* Verify the path exists and is safe-ish to execute */
 416         if (*av[0] != '/') {
 417                 error("%s path is not absolute", tag);
 418                 return 0;
 419         }
 420         temporarily_use_uid(pw);

Issue can be reproduced on fedora27 (openssh-7.6p1-5.fc27.x86_64)

Created attachment 1445835 [details]
proposed patch

Thank you for the good analysis. My initial bet was that some pw structure is being overwritten somewhere.

I reproduced the issue with RHEL7 virtual machine, but from the above steps, I was not able to reproduce the issue in current Fedora 28, even though I do not see any significant difference in the code there. I will investigate this further before reporting upstream.

This is something we should fix. The attached patch is resolving the issue for me. The idea is that we will reuse the stored groups only if we will be swapping for the same uid).

Ok, I reproduced the same issue also in Fedora 28 but for some reason, there is not the permission denied (probably some permissions elsewhere), but the strace provides the proof that the same issue is in upstream:

debug1: temporarily_use_uid: 1005/1009 (e=0/0)
[pid  3869] setgroups(1, [99])          = 0
[pid  3869] setresgid(-1, 1009, -1)     = 0
debug1: trying public key file /home/user1/.ssh/authorized_keys
debug1: fd 10 clearing O_NONBLOCK

The upstream bug is attached.

Workaround is to relax the permissions ("rx" for others on /home"), or relax for group "nobody" only, using the following ACL:

# setfacl -m group:nobody:rx /home

For the record, upstream already committed the fix:

https://github.com/openssh/openssh-portable/commit/26f96ca

> when changing the user.

should be more clear that it is only about the AuthorizedKeysCommand. Something like

> when changing the user for retrieving authorized keys using AuthorizedKeysCommand* configuration options.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:2143