Description of problem: When createrepo is run from cron it ignores some files, but when run manually, it includes them Version-Release number of selected component (if applicable): [root@tangelo epel-x86_64]# rpm -q createrepo createrepo-0.4.11-3.el5 How reproducible: Very Steps to Reproduce: 1. 12 * * * root reposync --arch=x86_64 --repoid=epel-x86_64 --config=/etc/yum.repos.d/epel.reposync -p /var/www/mrepo/ -d -l -g -n -q > /dev/null && createrepo --update -d /var/www/mrepo/epel-x86_64 via /etc/cron.d/reposync_epel for instance 2) run `createrepo --update -d /var/www/mrepo/epel-x86_64` manually Actual results: [root@tangelo mrepo]# head /tmp/createrepo.USG Scanning old repo data Indexed 4389 base nodes Indexed 4389 filelist nodes Indexed 4389 other nodes 1/4389 * gallery2-members-2.3-15.el5.noarch.rpm 2/4389 * gyachi-1.1.71-5.el5.x86_64.rpm 3/4389 * openser-presence_mwi-1.3.4-1.el5.x86_64.rpm 4/4389 * sems-xmlrpc2di-1.1.1-1.el5.1.x86_64.rpm 5/4389 * munge-0.5.8-4.el5.x86_64.rpm 6/4389 * globus-gsi-openssl-error-devel-0.14-4.el5.x86_64.rpm Expected results: [root@tangelo mrepo]# head /tmp/createrepo.USG.manual Scanning old repo data Indexed 4389 base nodes Indexed 4389 filelist nodes Indexed 4389 other nodes 1/4391 * gallery2-members-2.3-15.el5.noarch.rpm 2/4391 * gyachi-1.1.71-5.el5.x86_64.rpm 3/4391 * openser-presence_mwi-1.3.4-1.el5.x86_64.rpm 4/4391 * sems-xmlrpc2di-1.1.1-1.el5.1.x86_64.rpm 5/4391 * munge-0.5.8-4.el5.x86_64.rpm 6/4391 * globus-gsi-openssl-error-devel-0.14-4.el5.x86_64.rpm Note the difference in file count. Additional info: The host is running SELinux. The two ignored files have the same DAC and MAC settings: [root@tangelo mrepo]# ls -Z epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm -rw-r--r-- root root user_u:object_r:httpd_sys_content_t epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm [root@tangelo mrepo]# ls -Z epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm -rw-r--r-- root root user_u:object_r:httpd_sys_content_t epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm ... as another file clearly included (see item 1 above). [root@tangelo mrepo]# ls -Z epel-x86_64/gallery2-members-2.3-15.el5.noarch.rpm -rw-r--r-- root root user_u:object_r:httpd_sys_content_t epel-x86_64/gallery2-members-2.3-15.el5.noarch.rpm file sizes also don't seem to matter: [root@tangelo mrepo]# du -sh !$ du -sh epel-x86_64/gallery2-members-2.3-15.el5.noarch.rpm 88K epel-x86_64/gallery2-members-2.3-15.el5.noarch.rpm ... is fine, but the following are not: [root@tangelo mrepo]# du -sh epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm 16K epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm [root@tangelo mrepo]# du -sh epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm 552K epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm the key checks out:root@tangelo mrepo]# rpm -K !$ rpm -K epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm: (sha1) dsa sha1 md5 gpg OK [root@tangelo mrepo]# rpm -K epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm: (sha1) dsa sha1 md5 gpg OK [root@tangelo mrepo]# rpm -K epel-x86_64/gallery2-members-2.3-15.el5.noarch.rpm epel-x86_64/gallery2-members-2.3-15.el5.noarch.rpm: (sha1) dsa sha1 md5 gpg OK I should note that it is always the same two files that get ignored which makes me think there could be something the matter with the files, but : [root@tangelo mrepo]# file epel-x86_64/gallery2-members-2.3-15.el5.noarch.rpm^[[Aepel-x86_64/gallery2-members-2.3-15.el5.noarch.rpm: RPM v3 bin gallery2-members-2.3-15.el5 [root@tangelo mrepo]# file epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm: RPM v3 bin i386 ruby-shadow-1.4.1-7.el5 [root@tangelo mrepo]# file epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm: RPM v3 bin i386 python-cheetah-2.0.1-1.el5 where i excitedly spotted that the working one has no i386 in its output, but working files do have it at times, even if they are x86_64: [root@tangelo mrepo]# file epel-x86_64/munge-0.5.8-4.el5.x86_64.rpm epel-x86_64/munge-0.5.8-4.el5.x86_64.rpm: RPM v3 bin i386 munge-0.5.8-4.el5
Ok, weird. I have a couple of questions, which might shed some light on the problem: 1. I assume there are no error messages? Can you make sure you do 2> /tmp/blah from cron? 2. does "ls -f" show those packages at the end or beginning? Dito. passing -v and doing a diff. on the output of both runs. 3. Does removing --update solve the problem? 4. Does changing the working directory or LC_ALL from cron help?
(In reply to comment #1) > Ok, weird. I have a couple of questions, which might shed some light on the > problem: > > 1. I assume there are no error messages? Can you make sure you do 2> /tmp/blah > from cron? or is there anything tucked away in /var/spool/mail/root ? just to be sure - none of these are symlinks or hardlinks, right?
1) I've changed it to: 54 17 * * * root createrepo --verbose -d /var/www/mrepo/epel-x86_64 > /tmp/createrepo.USG 2> /tmp/createrepo.error [root@tangelo epel-x86_64]# cat /tmp/createrepo.error [root@tangelo epel-x86_64]# ...or is there anything tucked away in /var/spool/mail/root ? No, we get our mail forwarded though: [root@tangelo ~]# date Wed Sep 30 06:33:46 EST 2009 [root@tangelo ~]# ls -l /var/spool/mail/root -rw------- 1 root root 41434 Dec 10 2008 /var/spool/mail/root so lets see if tangelo mailed me: Sadly only the mail i knew i had: the latest one being a failed build missing the two packages. I don't see any errors in it, but i may be blind. I'll attach it. 2) not 100% sure what the dito bit means (long day); the -f shows the files somewhere in the middle, as does the -v . what am i diff'ing? 3) removing --update doesn't help, it only sees 4389 files I've made the cronjob: 54 17 * * * root createrepo --verbose -d /var/www/mrepo/epel-x86_64 > /tmp/createrepo.USG 2> /tmp/createrepo.error 4) Ok, i've never done anything with LC_ALL (so bear with me) but i gather it overrides LANG, which on my host are set: [root@tangelo ~]$ echo $LC_ALL [root@tangelo ~]$ echo $LANG en_US.UTF-8 I'm in Australia, so the following applies to me [root@tangelo ~]$ locale -a | egrep "en_AU|en_US" en_AU en_AU.iso88591 en_AU.utf8 en_US en_US.iso88591 en_US.iso885915 en_US.utf8 Firstly, what should i change it to? i tried: [root@tangelo ~]# cat /etc/sysconfig/i18n LANG="en_AU.UTF-8" LC_COLLATE="POSIX" SYSFONT="latarcyrheb-sun16" (where LANG had been set to "en_US.UTF-8") which made it not have a value for LANG when i logged in, but when i opened a second shell the values were set right ... (aliens ?) As for working directory from cron or LC_ALL (via cron?) you lost me. 5) just to be sure - none of these are symlinks or hardlinks, right? Heh .. good point, alas, all real directories and files: [root@tangelo ~]# ls -ld /var drwxr-xr-x 32 root root 4096 Sep 23 14:26 /var [root@tangelo ~]# ls -ld /var/www drwxr-xr-x 8 root root 4096 Jul 15 23:04 /var/www [root@tangelo ~]# ls -ld /var/www/mrepo/ drwxr-xr-x 18 root root 4096 Sep 29 12:05 /var/www/mrepo/ [root@tangelo ~]# ls -ld /var/www/mrepo/epel-x86_64/ drwxr-xr-x 4 root root 278528 Sep 30 06:51 /var/www/mrepo/epel-x86_64/ [root@tangelo ~]# ls -ld /var/www/mrepo/epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm -rw-r--r-- 1 root root 9856 Jul 1 18:37 /var/www/mrepo/epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm [root@tangelo ~]# ls -ld /var/www/mrepo/epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm -rw-r--r-- 1 root root 556975 Jul 1 17:45 /var/www/mrepo/epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm [root@tangelo epel-x86_64]# /bin/pwd /var/www/mrepo/epel-x86_64 [root@tangelo epel-x86_64]# pwd /var/www/mrepo/epel-x86_64
Created attachment 363056 [details] last email cron mailed regarding the reposync build, the two files are missing from the list
> 2) not 100% sure what the dito bit means (long day); no problem, I just meant if -v had them show up near the begining or end it might tell us something. But they should be in ls -f order. > 4) Ok, i've never done anything with LC_ALL but i gather it > overrides LANG, which on my host are set: Yeh, I was just wondering if cron was running with LANG=C or something and that was affecting something (it was a pretty wild guess, but the bug you have is impossible :). So try setting it to LC_ALL=en_US.utf8 > last email cron mailed regarding the reposync build We didn't need it in rtf :) ... you aren't doing the reposync again from the shell are you? There isn't anything weird in: /etc/yum.repos.d/epel.reposync
Also given that EPEL is created with this, could you make sure this version of createrepo works from cron: http://infrastructure.fedoraproject.org/releng/i386/createrepo-0.9.7-12.el5.noarch.rpm
No joy under 0.9.7-12 . I set LC_ALL as you said, but now i have nothing for LANG or LC_ALL ... no wait: [root@tangelo ~]# cat /etc/sysconfig/i18n LC_ALL="en_US.utf8" LANG="en_US.utf8" LC_COLLATE="POSIX" SYSFONT="latarcyrheb-sun16" [root@tangelo ~]# echo $LANG en_US.utf8 [root@tangelo ~]# echo $LC_ALL ... But it took ages after the reboot for $LANG to be set (what's up with that ... should be set by the time i log on). How would i override it from cron? I'll bounce the box again when i get a chance with it set to LC_ALL="en_US.UTF-8" and LC_ALL=en_US.utf8 ... though surely the quotes shouldn't have an effect; but as you say, my bug is impossible. Incidentally, this is in a VMware ESX VM, NOT that that should matter, but just in case :) chakkerz
> No joy under 0.9.7-12 . Ok, that heavily suggests it's not createrepo being the problem then. Can you run lsattr on the files (long shot). Can you run getfacl on the files (long shot). Can you run "getfattr -n security.selinux" on the files (really long shot). Have you tried removing the files and running reposync again? If all that fails, I guess run the createrepo under strace and see what that says.
lsattr and getfacl and the getfattr -n ... all don't report anything unusual. Still need to find the time to do the strace ... do you want me to run it and attach it for now?
yeah ... so ... it's just started working ... i did do a `yum update` yesterday morning ... and install some other things: Oct 02 11:02:30 Installed: sharutils-4.6.1-2.x86_64 Oct 02 11:03:04 Updated: USG_AuditFiles-2.2-2.x86_64 Oct 02 11:40:56 Installed: 5:mutt-1.4.2.2-3.0.2.el5.x86_64 Oct 05 07:25:27 Updated: neon-0.25.5-10.el5_4.1.x86_64 Oct 05 07:25:27 Updated: neon-0.25.5-10.el5_4.1.i386 Oct 05 07:25:28 Updated: openssh-4.3p2-36.el5_4.2.x86_64 Oct 05 07:25:46 Updated: 6:kdelibs-3.5.4-22.el5_4.1.x86_64 Oct 05 07:25:47 Updated: augeas-libs-0.5.3-1.el5.x86_64 Oct 05 07:25:47 Updated: newt-0.52.2-12.el5_4.1.x86_64 Oct 05 07:26:08 Installed: kernel-2.6.18-164.2.1.el5.x86_64 Oct 05 07:26:09 Updated: 1:autofs-5.0.1-0.rc2.131.el5_4.1.x86_64 Oct 05 07:26:09 Updated: openssh-server-4.3p2-36.el5_4.2.x86_64 Oct 05 07:26:12 Updated: kernel-headers-2.6.18-164.2.1.el5.x86_64 Oct 05 07:26:17 Updated: tzdata-2009m-1.el5.noarch Oct 05 07:26:21 Updated: elinks-0.11.1-6.el5_4.1.x86_64 Oct 05 07:26:22 Updated: openssh-clients-4.3p2-36.el5_4.2.x86_64 Oct 05 07:26:22 Updated: neon-devel-0.25.5-10.el5_4.1.x86_64 Oct 05 07:26:24 Updated: neon-devel-0.25.5-10.el5_4.1.i386 Oct 05 07:26:24 Updated: 1:dmidecode-2.10-2.el5.x86_64 Oct 05 07:26:24 Updated: USG_AuditFiles-2.4-1.x86_64 Eitherway it works now... I say i just stop touching it. Thanks for your help! chakkerz
Was createrepo running on directories over NFS/autofs? Did you reboot into the new kernel? > Eitherway it works now... I say i just stop touching it. Good plan :)
I did reboot into the new kernel. No NFS or AutoFS on that host. Seems to be holding up though ...