Bug 1496467
| Summary: | Regression: SSH AuthorizedKeysCommand hangs when output is too large | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Konrad Mosoń <mosonkonrad> | ||||
| Component: | openssh | Assignee: | Jakub Jelen <jjelen> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Stefan Dordevic <sdordevi> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 7.4 | CC: | dennis.schridde, nmavrogi, sdordevi | ||||
| Target Milestone: | rc | ||||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | openssh-7.4p1-15.el7 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2018-04-10 18:19:11 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1476743 | ||||||
| Attachments: |
|
||||||
Looking through the upstream bug [1], it looks like it was not completely fixed in 7.4 to which we rebased and there is single change needed to make it working again [2]. Can you verify that this this patch will fix the problem for you? If you wish a testing package or you would like to prioritize this effort, please contact you Red Hat Support. [1] https://bugzilla.mindrot.org/show_bug.cgi?id=2496 [2] https://github.com/openssh/openssh-portable/commit/ddd3d34e Just tested this patch, and OpenSSH with our script works correctly with it. I can login now to Git account via SSH. I'm attaching patch for specfile I used to test this (openssh-7.4p1-fix-authkeys-script-pipe.patch is directly downloaded commit from GitHub [2]). Created attachment 1333620 [details]
Patch for specfile
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2018:0980 |
Description of problem: We use GitLab which has a lot of entries in it's authorized_keys. We also use AuthorizedKeysCommand which parses authorized_keys along with other files. Unfortunately it stopped working after upgrade to RHEL 7.4 cause pipe hungs. I trimmed GitLab's authorized_keys only for testing to my entry and then login worked. Version-Release number of selected component (if applicable): openssh-7.4p1-12.el7_4.x86_64 How reproducible: Always when authorized_keys is big (not sure how big it must be, mine is 119K big) Steps to Reproduce: 1. Install RHEL 7.4 2. Create authorized_keys large enough. 3. Reconfigure sshd_config adding: ------- AuthorizedKeysCommand /usr/libexec/openssh/ssh-pubkey-helper AuthorizedKeysCommandUser root -------- 4. Create helper script: -------- #!/bin/bash USER=$1 HOME=`getent passwd $USER | cut -d: -f6` if [ -f $HOME/.ssh/authorized_keys ]; then cat $HOME/.ssh/authorized_keys* fi -------- 5. Try to login. Actual results: SSH connection hungs, and later fails with "Authentication failed" message. This is strace from broken SSH: -------- [pid 32328] <... read resumed> "", 128) = 0 [pid 32328] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=32329, si_uid=0, si_status=0, si_utime=0, si_stime=0} --- [pid 32328] wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], WNOHANG, NULL) = 32329 [pid 32328] wait4(-1, 0x7ffecd2a9d90, WNOHANG, NULL) = -1 ECHILD (No child processes) [pid 32328] rt_sigreturn({mask=[]}) = 0 [pid 32328] close(3) = 0 [pid 32328] rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 [pid 32328] rt_sigaction(SIGINT, {0x43e780, [], SA_RESTORER, 0x7fe385314270}, {SIG_DFL, [], SA_RESTORER, 0x7fe385314270}, 8) = 0 [pid 32328] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 [pid 32328] rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fe385314270}, {0x43e780, [], SA_RESTORER, 0x7fe385314270}, 8) = 0 [pid 32328] rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 [pid 32328] read(255, "\nif [ -f $HOME/.ssh/authorized_keys ]; then\n\tcat $HOME/.ssh/authorized_keys*\nfi\n", 142) = 80 [pid 32328] rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 [pid 32328] rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 [pid 32328] stat("/var/opt/gitlab/.ssh/authorized_keys", {st_mode=S_IFREG|0644, st_size=121632, ...}) = 0 [pid 32328] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 [pid 32328] openat(AT_FDCWD, "/var/opt/gitlab/.ssh/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3 [pid 32328] getdents(3, /* 4 entries */, 32768) = 128 [pid 32328] getdents(3, /* 0 entries */, 32768) = 0 [pid 32328] close(3) = 0 [pid 32328] stat(".", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 [pid 32328] stat("/usr/local/bin/cat", 0x7ffecd2aa470) = -1 ENOENT (No such file or directory) [pid 32328] stat("/usr/bin/cat", {st_mode=S_IFREG|0755, st_size=54080, ...}) = 0 [pid 32328] stat("/usr/bin/cat", {st_mode=S_IFREG|0755, st_size=54080, ...}) = 0 [pid 32328] geteuid() = 0 [pid 32328] getegid() = 0 [pid 32328] getuid() = 0 [pid 32328] getgid() = 0 [pid 32328] access("/usr/bin/cat", X_OK) = 0 [pid 32328] stat("/usr/bin/cat", {st_mode=S_IFREG|0755, st_size=54080, ...}) = 0 [pid 32328] geteuid() = 0 [pid 32328] getegid() = 0 [pid 32328] getuid() = 0 [pid 32328] getgid() = 0 [pid 32328] access("/usr/bin/cat", R_OK) = 0 [pid 32328] stat("/usr/bin/cat", {st_mode=S_IFREG|0755, st_size=54080, ...}) = 0 [pid 32328] stat("/usr/bin/cat", {st_mode=S_IFREG|0755, st_size=54080, ...}) = 0 [pid 32328] geteuid() = 0 [pid 32328] getegid() = 0 [pid 32328] getuid() = 0 [pid 32328] getgid() = 0 [pid 32328] access("/usr/bin/cat", X_OK) = 0 [pid 32328] stat("/usr/bin/cat", {st_mode=S_IFREG|0755, st_size=54080, ...}) = 0 [pid 32328] geteuid() = 0 [pid 32328] getegid() = 0 [pid 32328] getuid() = 0 [pid 32328] getgid() = 0 [pid 32328] access("/usr/bin/cat", R_OK) = 0 [pid 32328] rt_sigprocmask(SIG_BLOCK, [INT CHLD], [], 8) = 0 [pid 32328] rt_sigprocmask(SIG_BLOCK, [CHLD], [INT CHLD], 8) = 0 [pid 32328] rt_sigprocmask(SIG_SETMASK, [INT CHLD], NULL, 8) = 0 [pid 32328] clone(strace: Process 32332 attached child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fe385ce4a10) = 32332 [pid 32332] close(255 <unfinished ...> [pid 32328] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 [pid 32328] rt_sigprocmask(SIG_BLOCK, [CHLD], <unfinished ...> [pid 32332] <... close resumed> ) = 0 [pid 32328] <... rt_sigprocmask resumed> [], 8) = 0 [pid 32328] rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 [pid 32328] rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0 [pid 32328] rt_sigaction(SIGINT, {0x43e780, [], SA_RESTORER, 0x7fe385314270}, <unfinished ...> [pid 32332] rt_sigprocmask(SIG_SETMASK, [], <unfinished ...> [pid 32328] <... rt_sigaction resumed> {SIG_DFL, [], SA_RESTORER, 0x7fe385314270}, 8) = 0 [pid 32332] <... rt_sigprocmask resumed> NULL, 8) = 0 [pid 32328] wait4(-1, <unfinished ...> [pid 32332] rt_sigaction(SIGTSTP, {SIG_DFL, [], SA_RESTORER, 0x7fe385314270}, {SIG_DFL, [], 0}, 8) = 0 [pid 32332] rt_sigaction(SIGTTIN, {SIG_DFL, [], SA_RESTORER, 0x7fe385314270}, {SIG_DFL, [], 0}, 8) = 0 [pid 32332] rt_sigaction(SIGTTOU, {SIG_DFL, [], SA_RESTORER, 0x7fe385314270}, {SIG_DFL, [], 0}, 8) = 0 [pid 32332] rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fe385314270}, {SIG_DFL, [], SA_RESTORER, 0x7fe385314270}, 8) = 0 [pid 32332] rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7fe385314270}, {SIG_IGN, [], SA_RESTORER, 0x7fe385314270}, 8) = 0 [pid 32332] rt_sigaction(SIGCHLD, {SIG_DFL, [], SA_RESTORER|SA_RESTART, 0x7fe385314270}, {0x441310, [], SA_RESTORER|SA_RESTART, 0x7fe385314270}, 8) = 0 [pid 32332] execve("/usr/bin/cat", ["cat", "/var/opt/gitlab/.ssh/authorized_keys"], [/* 8 vars */]) = 0 [pid 32332] brk(NULL) = 0x2395000 [pid 32332] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e2df83000 [pid 32332] access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) [pid 32332] open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3 [pid 32332] fstat(3, {st_mode=S_IFREG|0644, st_size=34276, ...}) = 0 [pid 32332] mmap(NULL, 34276, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e2df7a000 [pid 32332] close(3) = 0 [pid 32332] open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3 [pid 32332] read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\35\2\0\0\0\0\0@\0\0\0\0\0\0\0(c \0\0\0\0\0\0\0\0\0@\0008\0\n\0@\0K\0J\0\6\0\0\0\5\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0000\2\0\0\0\0\0\0000\2\0\0\0\0\0\0\10\0\0\0\0\0\0\0\3\0\0\0\4\0\0\0\240I\30\0\0\0\0\0\240I\30\0\0\0\0\0\240I\30\0\0\0\0\0\34\0\0\0\0\0\0\0\34\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\224}\33\0\0\0\0\0\224}\33\0\0\0\0\0\0\0 \0\0\0\0\0\1\0\0\0\6\0\0\0000\207\33\0\0\0\0\0000\207;\0\0\0\0\0000\207;\0\0\0\0\0pQ\0\0\0\0\0\0\220\232\0\0\0\0\0\0\0\0 \0\0\0\0\0\2\0\0\0\6\0\0\0\200\273\33\0\0\0\0\0\200\273;\0\0\0\0\0\200\273;\0\0\0\0\0\360\1\0\0\0\0\0\0\360\1\0\0\0\0\0\0\10\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0p\2\0\0\0\0\0\0p\2\0\0\0\0\0\0p\2\0\0\0\0\0\0D\0\0\0\0\0\0\0D\0\0\0\0\0\0\0\4\0\0\0\0\0\0\0\7\0\0\0\4\0\0\0000\207\33\0\0\0\0\0000\207;\0\0\0\0\0000\207;\0\0\0\0\0\20\0\0\0\0\0\0\0\220\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0P\345td\4\0\0\0\274I\30\0\0\0\0\0\274I\30\0\0\0\0\0\274I\30\0\0\0\0\0004i\0\0\0\0\0\0004i\0\0\0\0\0\0\4\0\0\0\0\0\0\0Q\345td\6\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\20\0\0\0\0\0\0\0R\345td\4\0\0\0000\207\33\0\0\0\0\0000\207;\0\0\0\0\0000\207;\0\0\0\0\0\3208\0\0\0\0\0\0\3208\0\0\0\0\0\0\1\0\0\0\0\0\0\0\4\0\0\0\24\0\0\0\3\0\0\0GNU\0\303\362\210\0021J\364\356\206k\370\322\341\265\6\267\273\363L\366\4\0\0\0\20\0"..., 832) = 832 [pid 32332] fstat(3, {st_mode=S_IFREG|0755, st_size=2127336, ...}) = 0 [pid 32332] mmap(NULL, 3940800, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7e2d9a0000 [pid 32332] mprotect(0x7f7e2db58000, 2097152, PROT_NONE) = 0 [pid 32332] mmap(0x7f7e2dd58000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b8000) = 0x7f7e2dd58000 [pid 32332] mmap(0x7f7e2dd5e000, 16832, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7e2dd5e000 [pid 32332] close(3) = 0 [pid 32332] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e2df79000 [pid 32332] mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f7e2df77000 [pid 32332] arch_prctl(ARCH_SET_FS, 0x7f7e2df77740) = 0 [pid 32332] mprotect(0x7f7e2dd58000, 16384, PROT_READ) = 0 [pid 32332] mprotect(0x60b000, 4096, PROT_READ) = 0 [pid 32332] mprotect(0x7f7e2df84000, 4096, PROT_READ) = 0 [pid 32332] munmap(0x7f7e2df7a000, 34276) = 0 [pid 32332] brk(NULL) = 0x2395000 [pid 32332] brk(0x23b6000) = 0x23b6000 [pid 32332] brk(NULL) = 0x23b6000 [pid 32332] open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3 [pid 32332] fstat(3, {st_mode=S_IFREG|0644, st_size=106070960, ...}) = 0 [pid 32332] mmap(NULL, 106070960, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7e27477000 [pid 32332] close(3) = 0 [pid 32332] fstat(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0 [pid 32332] open("/var/opt/gitlab/.ssh/authorized_keys", O_RDONLY) = 3 [pid 32332] fstat(3, {st_mode=S_IFREG|0644, st_size=121632, ...}) = 0 [pid 32332] fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0 [pid 32332] read(3, "# Managed by gitlab-shell\ncommand=\"/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-5\",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa <CENSORED>\ncommand=\"/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-11\",no-port-forwarding,no-X11-forwarding,no-agen"..., 65536) = 65536 [pid 32332] write(1, "# Managed by gitlab-shell\ncommand=\"/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-5\",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa <CENSORED>\ncommand=\"/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-11\",no-port-forwarding,no-X11-forwarding,no-agen"..., 65536) = 65536 [pid 32327] <... read resumed> "# Managed by gitlab-shell\ncommand=\"/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-5\",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa <CENSORED>\ncommand=\"/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-11\",no-port-forwarding,no-X11-forwarding,no-agen"..., 4096) = 4096 [pid 32327] read(4, "itlab-shell key-20\",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa <CENSORED>\ncommand=\"/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-21\",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa <CENSORED>"..., 4096) = 4096 [pid 32327] read(4, "<CENSORED>\ncommand=\"/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-35\",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-dss <CENSORED>"..., 4096) = 4096 [pid 32327] read(4, "<CENSORED>\ncommand=\"/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-55\",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa <CENSORED>\ncommand=\"/opt/gitlab/"..., 4096) = 4096 [pid 32327] wait4(32328, <unfinished ...> [pid 32332] read(3, "<CENSORED>\ncommand=\"/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-213\",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa <CENSORED>\ncommand=\"/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-214\",no-port-forwarding,no-X11-forwarding,no-agent-for"..., 65536) = 56096 [pid 32332] write(1, "<CENSORED>\ncommand=\"/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-213\",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa <CENSORED>\ncommand=\"/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-214\",no-port-forwarding,no-X11-forwarding,no-agent-for"..., 56096 !! >>---- HERE IT HANGS ----<< !! ^Cstrace: Process 32305 detached strace: Process 32327 detached strace: Process 32328 detached strace: Process 32332 detached <detached ...> -------- Expected results: Login should working like it works in openssh-6.6.1p1-35.el7_3.x86_64 Additional info: Downgrading to openssh-6.6.1p1-35.el7_3.x86_64 from RHEL 7.3 workaround this problem at the moment.