Hide Forgot
Description of problem: cdn-sync (and PostgreSQL?) leaks memory when syncing Version-Release number of selected component (if applicable): Satellite-5.8-RHEL-6-20161114.n.1-Satellite-x86_64 ...with embedded DB How reproducible: always Steps to Reproduce: 1. # cdn-sync -c rhel-x86_64-server-6 2. Notice memory consumption of cdn-sync and postgres processes Actual results: Memory consumption of cdn-sync and postgres processes grows. After about 2 hours of syncing cdn-sync is killed by signal 9 making it to exit with 137 return code (I have seen it after 2 hours when running with "--no-rpms", but raise can be seen without that as well). Expected results: Memory should not raise Additional info: # while true; do date; ps --no-headers -eo rss,comm>a; for comm in $( sed 's/^\s*[0-9]\+\s*\(.*\)$/\1/' a | sort -u ); do size=$( grep "\s$comm" a | sed 's/^\s*\([0-9]\+\)\s*.*$/\1/' | paste -sd+ - | bc ); echo "$size $comm"; done | sort -n | tail -n 5 | grep -e cdn-sync -e postgres; echo; sleep 10; done Tue Nov 22 08:28:23 EST 2016 173460 cdn-sync 477272 postgres Tue Nov 22 08:28:33 EST 2016 174984 cdn-sync 483076 postgres Tue Nov 22 08:28:43 EST 2016 176224 cdn-sync 491544 postgres Tue Nov 22 08:28:54 EST 2016 177852 cdn-sync 494644 postgres Tue Nov 22 08:29:04 EST 2016 180160 cdn-sync 497520 postgres Tue Nov 22 08:29:15 EST 2016 182004 cdn-sync 499248 postgres Tue Nov 22 08:29:25 EST 2016 182344 cdn-sync 499592 postgres Tue Nov 22 08:29:35 EST 2016 182500 cdn-sync 496832 postgres Tue Nov 22 08:29:45 EST 2016 182500 cdn-sync 497560 postgres Tue Nov 22 08:29:56 EST 2016 182624 cdn-sync 498248 postgres Tue Nov 22 08:30:06 EST 2016 182996 cdn-sync 502724 postgres Tue Nov 22 08:30:16 EST 2016 185016 cdn-sync 509652 postgres Tue Nov 22 08:30:27 EST 2016 185036 cdn-sync 516420 postgres Tue Nov 22 08:30:37 EST 2016 193320 cdn-sync 532772 postgres
Patch available in upstream. spacewalk.git: 76552282c5708718d71ddce48031458d78ac7f03
^^^^^ wrong commit hash, correct is b00061fc63c656c666a7894219f5eebeca4ebeaa. Cherry-picked to SATELLITE-5.8: 2278513f873c602261bc5ed254a4a69850f43456
Reproduced on spacewalk-backend-2.5.3-34. When synchronizing a big base channel with multiple child channels (rhel-x86_64-workstation-7 and all its childchannels), cdn-sync memory usage was climbing steadily, reaching gigs of memory at the end. Verified on spacewalk-backend-2.5.3-41. I have used Jan's monitoring snippet from the initial comment. I have attached its results, and created a simple chart to see memory usage for a long running cdn-sync process. VERIFIED
Created attachment 1234391 [details] Memory usage log after fix
Created attachment 1234392 [details] Memory usage chart after fix