Bug 1397417

Summary: cdn-sync leaks memory when syncing
Product: Red Hat Satellite 5 Reporter: Jan Hutař <jhutar>
Component: Satellite SynchronizationAssignee: Gennadii Altukhov <galtukho>
Status: CLOSED CURRENTRELEASE QA Contact: Radovan Drazny <rdrazny>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 580CC: dyordano, galtukho, mkorbel, rdrazny, tlestach
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: spacewalk-backend-2.5.3-37-sat Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-06-21 12:11:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1340444    
Attachments:
Description Flags
Memory usage log after fix
none
Memory usage chart after fix none

Description Jan Hutař 2016-11-22 13:38:43 UTC 
Description of problem:
cdn-sync (and PostgreSQL?) leaks memory when syncing


Version-Release number of selected component (if applicable):
Satellite-5.8-RHEL-6-20161114.n.1-Satellite-x86_64
  ...with embedded DB


How reproducible:
always


Steps to Reproduce:
1. # cdn-sync -c rhel-x86_64-server-6
2. Notice memory consumption of cdn-sync and postgres processes


Actual results:
Memory consumption of cdn-sync and postgres processes grows. After about 2 hours of syncing cdn-sync is killed by signal 9 making it to exit with 137 return code (I have seen it after 2 hours when running with "--no-rpms", but raise can be seen without that as well).


Expected results:
Memory should not raise


Additional info:
# while true; do date; ps --no-headers -eo rss,comm>a; for comm in $( sed 's/^\s*[0-9]\+\s*\(.*\)$/\1/' a | sort -u ); do size=$( grep "\s$comm" a | sed 's/^\s*\([0-9]\+\)\s*.*$/\1/' | paste -sd+ - | bc ); echo "$size $comm"; done | sort -n | tail -n 5 | grep -e cdn-sync -e postgres; echo; sleep 10; done
Tue Nov 22 08:28:23 EST 2016
173460 cdn-sync
477272 postgres

Tue Nov 22 08:28:33 EST 2016
174984 cdn-sync
483076 postgres

Tue Nov 22 08:28:43 EST 2016
176224 cdn-sync
491544 postgres

Tue Nov 22 08:28:54 EST 2016
177852 cdn-sync
494644 postgres

Tue Nov 22 08:29:04 EST 2016
180160 cdn-sync
497520 postgres

Tue Nov 22 08:29:15 EST 2016
182004 cdn-sync
499248 postgres

Tue Nov 22 08:29:25 EST 2016
182344 cdn-sync
499592 postgres

Tue Nov 22 08:29:35 EST 2016
182500 cdn-sync
496832 postgres

Tue Nov 22 08:29:45 EST 2016
182500 cdn-sync
497560 postgres

Tue Nov 22 08:29:56 EST 2016
182624 cdn-sync
498248 postgres

Tue Nov 22 08:30:06 EST 2016
182996 cdn-sync
502724 postgres

Tue Nov 22 08:30:16 EST 2016
185016 cdn-sync
509652 postgres

Tue Nov 22 08:30:27 EST 2016
185036 cdn-sync
516420 postgres

Tue Nov 22 08:30:37 EST 2016
193320 cdn-sync
532772 postgres
Comment 3 Gennadii Altukhov 2016-12-15 09:28:39 UTC 
Patch available in upstream. spacewalk.git:
76552282c5708718d71ddce48031458d78ac7f03
Comment 4 Gennadii Altukhov 2016-12-15 10:40:03 UTC 
^^^^^ wrong commit hash, correct is b00061fc63c656c666a7894219f5eebeca4ebeaa.

Cherry-picked to SATELLITE-5.8:
2278513f873c602261bc5ed254a4a69850f43456
Comment 5 Radovan Drazny 2016-12-21 13:02:44 UTC 
Reproduced on spacewalk-backend-2.5.3-34. When synchronizing a big base channel with multiple child channels (rhel-x86_64-workstation-7 and all its childchannels), cdn-sync memory usage was climbing steadily, reaching gigs of memory at the end.

Verified on spacewalk-backend-2.5.3-41. I have used Jan's monitoring snippet from the initial comment. I have attached its results, and created a simple chart to see memory usage for a long running cdn-sync process.

VERIFIED
Comment 6 Radovan Drazny 2016-12-21 13:04:55 UTC 
Created attachment 1234391 [details]
Memory usage log after fix
Comment 7 Radovan Drazny 2016-12-21 13:05:34 UTC 
Created attachment 1234392 [details]
Memory usage chart after fix