Given that we have 16665 task RPMs in /var/www/beaker/rpms/ building a new yum repo for every recipe is unnecessarily costly. I just noticed that each invocation generates quite a lot of I/O and CPU activity and takes up 560MB resident memory. We should try and find a more efficient way of generating these yum repos.
*** Bug 862014 has been marked as a duplicate of this bug. ***
This is causing gross spikes on our beaker server. LA is jumping to 9 and above fairly regularly. We just don't have the resources to throw any more hardware at the issue, we need to get this fixed. When I've asked previously, I've been told to go speak with the yum devs, but I am not familiar enough with how it is being used to give them any intelligent info on what is going wrong. If we can get a bz opened with them to fix the bloat in genpkgmetadata, or find a way to make repo_update et cetera work better/different, that would be great. It sounds like the load is also negatively impacting testing in our environment, so it is probably worthwhile to get this pushed up to a higher priority.
http://gerrit.beaker-project.org/#/c/1409/
Hi Bill, How does the commit I1c276d9bc33b94eebfbd65e822e167fe1d5ad57f fix the load issue? I was hoping you would read package metadata once an rpm is pushed and store it in SQL then generate the repodata directly using the createrepo python module, not the tool that re-reads it from thousands of rpms.
(In reply to comment #4) > Hi Bill, > > How does the commit I1c276d9bc33b94eebfbd65e822e167fe1d5ad57f fix the load > issue? I was hoping you would read package metadata once an rpm is pushed > and store it in SQL then generate the repodata directly using the createrepo > python module, not the tool that re-reads it from thousands of rpms. Hi Vadim, We would stop running the createrepo program every time we schedule a recipe. we would only run createrepo when a new task has been uploaded. The suggestion you have to record the metadata in the db and generate the repodata ourselves is a much bigger change. I wasn't even aware that that could be done. If it's straightforward then we may want to look into doing that too. Even without the change you recommend this will drastically reduce load since we would only be doing the createrepo when someone uploads a new task rpm. Here are the numbers from last Friday: [bpeck@beaker-02 rpms]$ grep '2012-10-05' /var/log/beaker/beakerd.log*| grep 'Queued to Scheduled' | wc -l 1128 versus the number of tasks that were uploaded to the scheduler on the same day: [bpeck@beaker-02 rpms]$ ls -l | grep 'Oct 5 ..:' | wc -l 43 That is 1085 less times calling createrepo.
Bill, Thanks for clarifying. IIRC it took ~20-30 minutes for createrepo to finish one run, so I doubt we actually ran it 1128 times / day. Are you checking if another copy is already running? If you do - this might explain the difference in numbers. If you aren't - then I might be mistaken and those genpkgmetadata processes I've been seeing were started one after another and thought it's the same one (didn't check the pids). I was not thinking how to optimize the number of times createrepo is ran, but the amount of resources necessary to accomplish the metadata regeneration. I was thinking that it actually is started each time a new task is pushed to the library (which would be logical from my POV).
Hi Vadim, we are running createrepo with the --update option which in theory should make it better, so I'm pretty sure we did run it 1,128 times in one day.
Bulk reassignment of issues as Bill has moved to another team.
*** Bug 658229 has been marked as a duplicate of this bug. ***
*** Bug 626738 has been marked as a duplicate of this bug. ***
This has been working fine in devel environments, but there are some related tests which are failing in dogfood, for example: ====================================================================== FAIL: test_submit_task (bkr.inttest.server.selenium.test_tasks.TestSubmitTask) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/bkr/inttest/server/selenium/test_tasks.py", line 41, in test_submit_task in sel.get_text('css=.flash')) AssertionError: -------------------- >> begin captured stdout << --------------------- 07:19:24.356 INFO - Command request: open[, True] on session cfe51a1f20ce4b70982358c4bd7c5eaa 07:19:25.423 INFO - Got result: OK on session cfe51a1f20ce4b70982358c4bd7c5eaa 07:19:25.453 INFO - Command request: click[link=New Task, ] on session cfe51a1f20ce4b70982358c4bd7c5eaa 07:19:25.491 INFO - Got result: OK on session cfe51a1f20ce4b70982358c4bd7c5eaa 07:19:25.493 INFO - Command request: waitForPageToLoad[30000, ] on session cfe51a1f20ce4b70982358c4bd7c5eaa 07:19:26.561 INFO - Got result: OK on session cfe51a1f20ce4b70982358c4bd7c5eaa 07:19:26.571 INFO - Command request: type[task_task_rpm, /usr/lib/python2.6/site-packages/bkr/inttest/server/selenium/tmp-distribution-beaker-task_test-1.1-0.noarch.rpm] on session cfe51a1f20ce4b70982358c4bd7c5eaa 07:19:26.665 INFO - Got result: OK on session cfe51a1f20ce4b70982358c4bd7c5eaa 07:19:26.667 INFO - Command request: click[//input[@value="Submit Data"], ] on session cfe51a1f20ce4b70982358c4bd7c5eaa 07:19:27.857 INFO - Got result: OK on session cfe51a1f20ce4b70982358c4bd7c5eaa 07:19:27.860 INFO - Command request: waitForPageToLoad[30000, ] on session cfe51a1f20ce4b70982358c4bd7c5eaa 07:19:38.705 INFO - Got result: OK on session cfe51a1f20ce4b70982358c4bd7c5eaa 07:19:38.708 INFO - Command request: getText[css=.flash, ] on session cfe51a1f20ce4b70982358c4bd7c5eaa 07:19:38.739 INFO - Got result: OK,Failed to import task: Command '['createrepo', '-q', '--update', '--checksum', 'sha', '.']' returned non-zero exit status 1 on session cfe51a1f20ce4b70982358c4bd7c5eaa --------------------- >> end captured stdout << ---------------------- It might just be a configuration issue with the dogfood task.
This has now been released