Bug 2270686 - Performance Problem: Proftpd Version 1.3.6 on RHEL 8 Reading Password File Byte By Byte
Summary: Performance Problem: Proftpd Version 1.3.6 on RHEL 8 Reading Password File By...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora EPEL
Classification: Fedora
Component: proftpd
Version: epel8
Hardware: x86_64
OS: Linux
unspecified
low
Target Milestone: ---
Assignee: Paul Howarth
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-03-21 12:40 UTC by leighchapman
Modified: 2024-04-04 02:08 UTC (History)
2 users (show)

Fixed In Version: proftpd-1.3.6e-7.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-04-04 02:08:24 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github proftpd proftpd issues 1134 0 None closed AuthUserFile-based logins, directory listings are very slow due to unbuffered reads 2024-03-22 10:11:28 UTC

Description leighchapman 2024-03-21 12:40:35 UTC
Description of problem:

In using Redhat 8 with the latest EPEL Package proftpd-1.3.6e-6.el8.x86_64 we are experiencing performance problems. The CPU Usage is incredibly high when users login. What I have seen via strace is that the password file is being read unbuffered byte for byte. When I test with only 1 user on the system, I have observed that the login takes a few seconds to complete. 


Version-Release number of selected component (if applicable): proftpd-1.3.6e-6.el8.x86_64


How reproducible:

Login on the proftpd server simply with ftp command and enter the username + password. Perform an strace on the PID of the login process.

Steps to Reproduce:
1. Use following in /etc/proftpd.conf

# Use pam to authenticate (default) and be authoritative
#AuthPAMConfig                  proftpd
#AuthOrder                      mod_auth_pam.c* mod_auth_unix.c
#AuthUserFile /etc/proftpd.passwd
AuthUserFile /etc/proftpd.passwd
AuthOrder                       mod_auth_file.c*

2. The /etc/proftpd.passwd contains 1540 lines and the login user is on the last line.
3.Directly on Linux Server login via ftp and note the PID of the login process created. 
4.strace -p PID

5. Login completed after a few seconds.


Actual results:


Strace Output............

stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2335, ...}) = 0

write(6, "2024-03-18 14:39:08,671 [3066] <"..., 61) = 61

fcntl(7, F_SETFD, FD_CLOEXEC)           = 0

lseek(7, 0, SEEK_CUR)                   = 0

lseek(7, 0, SEEK_CUR)                   = 0

read(7, "j", 1)                         = 1

read(7, "a", 1)                         = 1

read(7, "m", 1)                         = 1

read(7, "e", 1)                         = 1

read(7, "s", 1)                         = 1


Expected results:

Login is completed in less than a second. Also that the password file is read using buffering.

Additional info:

On testing with RHEL 9 + proftpd-1.3.8b-2.el9.x86_64 from EPEL, the login of 1 user is far quicker. Here an strace :


openat(AT_FDCWD, "/etc/proftpd.passwd", O_RDONLY) = 9
fcntl(9, F_SETFD, FD_CLOEXEC)           = 0
rt_sigprocmask(SIG_BLOCK, [HUP INT QUIT BUS USR1 ALRM TERM CHLD IO], NULL, 8) = 0
geteuid()                               = 0
setresgid(-1, 65534, -1)                = 0
setresuid(-1, 65534, -1)                = 0
rt_sigprocmask(SIG_UNBLOCK, [HUP INT QUIT BUS USR1 ALRM TERM CHLD IO], NULL, 8) = 0
newfstatat(9, "", {st_mode=S_IFREG|0640, st_size=492642, ...}, AT_EMPTY_PATH) = 0
fcntl(9, F_SETFD, FD_CLOEXEC)           = 0
read(9, "james:$1$7KiQI6HZ$ZUTXt9/qyhNIR0"..., 4096) = 4096
read(9, "HZ$ZUTXt9/qyhNIR0o/M34Ct.:70000:"..., 4096) = 4096


Also note, on a RHEL 7 system with proftpd-1.3.5e-12.el7.x86_64, the login performance is much better than that of RHEL 8. 

It seems that proftpd for RHEL 7 + RHEL 9 read the password file buffered and offer good performance and for some reason proftpd on RHEL 8 behaves differently reading byte for byte causing a high CPU Load.

Comment 1 Paul Howarth 2024-03-22 10:11:28 UTC
Looks like https://github.com/proftpd/proftpd/issues/1134

I can try applying 3 commits from master to mod_auth_file.c that should resolve this; if I did a scratch build would you be able to download it from koji adn give it a try?

Comment 2 Paul Howarth 2024-03-22 19:45:46 UTC
Scratch build for testing:
https://koji.fedoraproject.org/koji/taskinfo?taskID=115307756

Comment 3 leighchapman 2024-03-22 20:03:34 UTC
Thanks for the quick response and yes of course I will give the scratch build a try.

Comment 4 leighchapman 2024-03-26 13:40:42 UTC
Hi,

the build from proftpd-1.3.6e-7.el8.x86_64.rpm has been downloaded and tested and I can confirm that the proftpd now reads the password file with buffering and that the performance is significantly better. The CPU usage is no longer extremely high with this fix and that the login of users is now almost instantaneous. 

Thanks,
Leigh

Comment 5 Paul Howarth 2024-03-26 16:16:56 UTC
Thanks for testing, I'll now do a "real" build and update with the same code.

Comment 6 Fedora Update System 2024-03-26 17:02:36 UTC
FEDORA-EPEL-2024-f100669565 (proftpd-1.3.6e-7.el8) has been submitted as an update to Fedora EPEL 8.
https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2024-f100669565

Comment 7 Fedora Update System 2024-03-27 02:37:02 UTC
FEDORA-EPEL-2024-f100669565 has been pushed to the Fedora EPEL 8 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2024-f100669565

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 8 Fedora Update System 2024-04-04 02:08:24 UTC
FEDORA-EPEL-2024-f100669565 (proftpd-1.3.6e-7.el8) has been pushed to the Fedora EPEL 8 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.