Bug 526143 - Createrepo through cron persistently ignores two files it includes when not run through cron.
Summary: Createrepo through cron persistently ignores two files it includes when not r...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: createrepo
Version: 5.4
Hardware: x86_64
OS: Linux
low
medium
Target Milestone: rc
: ---
Assignee: Packaging Maintenance Team
QA Contact: Red Hat Satellite QA List
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-09-29 03:20 UTC by Christian Unger
Modified: 2014-01-21 06:15 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-03-14 20:07:55 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
last email cron mailed regarding the reposync build, the two files are missing from the list (302.92 KB, text/plain)
2009-09-29 21:09 UTC, Christian Unger
no flags Details

Description Christian Unger 2009-09-29 03:20:41 UTC
Description of problem:
When createrepo is run from cron it ignores some files, but when run manually, it includes them

Version-Release number of selected component (if applicable):
[root@tangelo epel-x86_64]# rpm -q createrepo
createrepo-0.4.11-3.el5


How reproducible:
Very

Steps to Reproduce:
1. 12 * * *      root    reposync --arch=x86_64 --repoid=epel-x86_64 --config=/etc/yum.repos.d/epel.reposync -p /var/www/mrepo/ -d -l -g -n -q > /dev/null && createrepo --update -d /var/www/mrepo/epel-x86_64 

via /etc/cron.d/reposync_epel for instance

2) run `createrepo --update -d /var/www/mrepo/epel-x86_64` manually

Actual results:
[root@tangelo mrepo]# head /tmp/createrepo.USG        
Scanning old repo data
Indexed 4389 base nodes
Indexed 4389 filelist nodes
Indexed 4389 other nodes
1/4389 * gallery2-members-2.3-15.el5.noarch.rpm
2/4389 * gyachi-1.1.71-5.el5.x86_64.rpm
3/4389 * openser-presence_mwi-1.3.4-1.el5.x86_64.rpm
4/4389 * sems-xmlrpc2di-1.1.1-1.el5.1.x86_64.rpm
5/4389 * munge-0.5.8-4.el5.x86_64.rpm
6/4389 * globus-gsi-openssl-error-devel-0.14-4.el5.x86_64.rpm


Expected results:
[root@tangelo mrepo]# head /tmp/createrepo.USG.manual 
Scanning old repo data
Indexed 4389 base nodes
Indexed 4389 filelist nodes
Indexed 4389 other nodes
1/4391 * gallery2-members-2.3-15.el5.noarch.rpm
2/4391 * gyachi-1.1.71-5.el5.x86_64.rpm
3/4391 * openser-presence_mwi-1.3.4-1.el5.x86_64.rpm
4/4391 * sems-xmlrpc2di-1.1.1-1.el5.1.x86_64.rpm
5/4391 * munge-0.5.8-4.el5.x86_64.rpm
6/4391 * globus-gsi-openssl-error-devel-0.14-4.el5.x86_64.rpm


Note the difference in file count.

Additional info:
The host is running SELinux. The two ignored files have the same DAC and MAC settings:

[root@tangelo mrepo]# ls -Z epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm 
-rw-r--r--  root root user_u:object_r:httpd_sys_content_t epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm
[root@tangelo mrepo]# ls -Z epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm    
-rw-r--r--  root root user_u:object_r:httpd_sys_content_t epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm

... as another file clearly included (see item 1 above).

[root@tangelo mrepo]# ls -Z epel-x86_64/gallery2-members-2.3-15.el5.noarch.rpm 
-rw-r--r--  root root user_u:object_r:httpd_sys_content_t epel-x86_64/gallery2-members-2.3-15.el5.noarch.rpm

file sizes also don't seem to matter:
[root@tangelo mrepo]# du -sh !$
du -sh epel-x86_64/gallery2-members-2.3-15.el5.noarch.rpm
88K     epel-x86_64/gallery2-members-2.3-15.el5.noarch.rpm

... is fine, but the following are not:

[root@tangelo mrepo]# du -sh epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm    
16K     epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm
[root@tangelo mrepo]# du -sh epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm 
552K    epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm

the key checks out:root@tangelo mrepo]# rpm -K !$

rpm -K epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm
epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm: (sha1) dsa sha1 md5 gpg OK
[root@tangelo mrepo]# rpm -K epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm   
epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm: (sha1) dsa sha1 md5 gpg OK
[root@tangelo mrepo]# rpm -K epel-x86_64/gallery2-members-2.3-15.el5.noarch.rpm epel-x86_64/gallery2-members-2.3-15.el5.noarch.rpm: (sha1) dsa sha1 md5 gpg OK


I should note that it is always the same two files that get ignored which makes me think there could be something the matter with the files, but :
[root@tangelo mrepo]# file epel-x86_64/gallery2-members-2.3-15.el5.noarch.rpm^[[Aepel-x86_64/gallery2-members-2.3-15.el5.noarch.rpm: RPM v3 bin gallery2-members-2.3-15.el5

[root@tangelo mrepo]# file epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm 
epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm: RPM v3 bin i386 ruby-shadow-1.4.1-7.el5

[root@tangelo mrepo]# file epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm 
epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm: RPM v3 bin i386 python-cheetah-2.0.1-1.el5

where i excitedly spotted that the working one has no i386 in its output, but working files do have it at times, even if they are x86_64:

[root@tangelo mrepo]# file epel-x86_64/munge-0.5.8-4.el5.x86_64.rpm 
epel-x86_64/munge-0.5.8-4.el5.x86_64.rpm: RPM v3 bin i386 munge-0.5.8-4.el5

Comment 1 James Antill 2009-09-29 04:57:38 UTC
Ok, weird. I have a couple of questions, which might shed some light on the problem:

1. I assume there are no error messages? Can you make sure you do 2> /tmp/blah from cron?

2. does "ls -f" show those packages at the end or beginning? Dito. passing -v and doing a diff. on the output of both runs.

3. Does removing --update solve the problem?

4. Does changing the working directory or LC_ALL from cron help?

Comment 2 seth vidal 2009-09-29 14:43:36 UTC
(In reply to comment #1)
> Ok, weird. I have a couple of questions, which might shed some light on the
> problem:
> 
> 1. I assume there are no error messages? Can you make sure you do 2> /tmp/blah
> from cron?

or is there anything tucked away in /var/spool/mail/root ?

 
just to be sure - none of these are symlinks or hardlinks, right?

Comment 3 Christian Unger 2009-09-29 21:07:58 UTC
1) 
I've changed it to:
54 17 * * *	root	createrepo --verbose -d /var/www/mrepo/epel-x86_64 > /tmp/createrepo.USG 2> /tmp/createrepo.error
[root@tangelo epel-x86_64]# cat /tmp/createrepo.error 
[root@tangelo epel-x86_64]# 

...or is there anything tucked away in /var/spool/mail/root ?

No, we get our mail forwarded though:
[root@tangelo ~]# date
Wed Sep 30 06:33:46 EST 2009
[root@tangelo ~]# ls -l /var/spool/mail/root 
-rw------- 1 root root 41434 Dec 10  2008 /var/spool/mail/root

so lets see if tangelo mailed me: Sadly only the mail i knew i had: the latest one being a failed build missing the two packages. I don't see any errors in it, but i may be blind. I'll attach it.


2) not 100% sure what the dito bit means (long day); the -f shows the files somewhere in the middle, as does the -v . what am i diff'ing?

3) removing --update doesn't help, it only sees 4389 files
I've made the cronjob:
54 17 * * *	root	createrepo --verbose -d /var/www/mrepo/epel-x86_64 > /tmp/createrepo.USG 2> /tmp/createrepo.error

4) Ok, i've never done anything with LC_ALL (so bear with me) but i gather it overrides LANG, which on my host are set:
[root@tangelo ~]$ echo $LC_ALL

[root@tangelo ~]$ echo $LANG
en_US.UTF-8

I'm in Australia, so the following applies to me
[root@tangelo ~]$ locale -a | egrep "en_AU|en_US"
en_AU
en_AU.iso88591
en_AU.utf8
en_US
en_US.iso88591
en_US.iso885915
en_US.utf8

Firstly, what should i change it to? i tried:
[root@tangelo ~]# cat /etc/sysconfig/i18n 
LANG="en_AU.UTF-8"
LC_COLLATE="POSIX"
SYSFONT="latarcyrheb-sun16"
(where LANG had been set to "en_US.UTF-8") which made it not have a value for LANG when i logged in, but when i opened a second shell the values were set right ... (aliens ?) 

As for working directory from cron or LC_ALL (via cron?) you lost me.


5) just to be sure - none of these are symlinks or hardlinks, right?

Heh .. good point, alas, all real directories and files:

[root@tangelo ~]# ls -ld /var
drwxr-xr-x 32 root root 4096 Sep 23 14:26 /var
[root@tangelo ~]# ls -ld /var/www
drwxr-xr-x 8 root root 4096 Jul 15 23:04 /var/www
[root@tangelo ~]# ls -ld /var/www/mrepo/
drwxr-xr-x 18 root root 4096 Sep 29 12:05 /var/www/mrepo/
[root@tangelo ~]# ls -ld /var/www/mrepo/epel-x86_64/
drwxr-xr-x 4 root root 278528 Sep 30 06:51 /var/www/mrepo/epel-x86_64/
[root@tangelo ~]# ls -ld /var/www/mrepo/epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm 
-rw-r--r-- 1 root root 9856 Jul  1 18:37 /var/www/mrepo/epel-x86_64/ruby-shadow-1.4.1-7.el5.x86_64.rpm
[root@tangelo ~]# ls -ld /var/www/mrepo/epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm 
-rw-r--r-- 1 root root 556975 Jul  1 17:45 /var/www/mrepo/epel-x86_64/python-cheetah-2.0.1-1.el5.x86_64.rpm
[root@tangelo epel-x86_64]# /bin/pwd
/var/www/mrepo/epel-x86_64
[root@tangelo epel-x86_64]# pwd
/var/www/mrepo/epel-x86_64

Comment 4 Christian Unger 2009-09-29 21:09:49 UTC
Created attachment 363056 [details]
last email cron mailed regarding the reposync build, the two files are missing from the list

Comment 5 James Antill 2009-09-29 21:42:43 UTC
> 2) not 100% sure what the dito bit means (long day);

 no problem, I just meant if -v had them show up near the begining or end it might tell us something. But they should be in ls -f order.

> 4) Ok, i've never done anything with LC_ALL but i gather it
> overrides LANG, which on my host are set:

 Yeh, I was just wondering if cron was running with LANG=C or something and that was affecting something (it was a pretty wild guess, but the bug you have is impossible :).
 So try setting it to LC_ALL=en_US.utf8

> last email cron mailed regarding the reposync build

 We didn't need it in rtf :) ... you aren't doing the reposync again from the shell are you? There isn't anything weird in:

/etc/yum.repos.d/epel.reposync

Comment 6 James Antill 2009-09-29 21:46:09 UTC
 Also given that EPEL is created with this, could you make sure this version of createrepo works from cron:

http://infrastructure.fedoraproject.org/releng/i386/createrepo-0.9.7-12.el5.noarch.rpm

Comment 7 Christian Unger 2009-09-30 02:09:28 UTC
No joy under 0.9.7-12 .

I set LC_ALL as you said, but now i have nothing for LANG or LC_ALL ... no wait:

[root@tangelo ~]# cat /etc/sysconfig/i18n 
LC_ALL="en_US.utf8"
LANG="en_US.utf8"
LC_COLLATE="POSIX"
SYSFONT="latarcyrheb-sun16"
[root@tangelo ~]# echo $LANG
en_US.utf8
[root@tangelo ~]# echo $LC_ALL

...

But it took ages after the reboot for $LANG to be set (what's up with that ... should be set by the time i log on).

How would i override it from cron? I'll bounce the box again when i get a chance with it set to LC_ALL="en_US.UTF-8" and LC_ALL=en_US.utf8 ... though surely the quotes shouldn't have an effect; but as you say, my bug is impossible.

Incidentally, this is in a VMware ESX VM, NOT that that should matter, but just in case :)

chakkerz

Comment 8 James Antill 2009-09-30 03:49:14 UTC
> No joy under 0.9.7-12 .

 Ok, that heavily suggests it's not createrepo being the problem then.

 Can you run lsattr on the files (long shot).

 Can you run getfacl on the files (long shot).

 Can you run "getfattr -n security.selinux" on the files (really long shot).

 Have you tried removing the files and running reposync again?

 If all that fails, I guess run the createrepo under strace and see what that says.

Comment 9 Christian Unger 2009-10-06 04:01:40 UTC
lsattr and getfacl and the getfattr -n ... all don't report anything unusual. 

Still need to find the time to do the strace ... do you want me to run it and attach it for now?

Comment 10 Christian Unger 2009-10-06 04:45:24 UTC
yeah ... so ... it's just started working ... i did do a `yum update` yesterday morning ... and install some other things:

Oct 02 11:02:30 Installed: sharutils-4.6.1-2.x86_64
Oct 02 11:03:04 Updated: USG_AuditFiles-2.2-2.x86_64
Oct 02 11:40:56 Installed: 5:mutt-1.4.2.2-3.0.2.el5.x86_64
Oct 05 07:25:27 Updated: neon-0.25.5-10.el5_4.1.x86_64
Oct 05 07:25:27 Updated: neon-0.25.5-10.el5_4.1.i386
Oct 05 07:25:28 Updated: openssh-4.3p2-36.el5_4.2.x86_64
Oct 05 07:25:46 Updated: 6:kdelibs-3.5.4-22.el5_4.1.x86_64
Oct 05 07:25:47 Updated: augeas-libs-0.5.3-1.el5.x86_64
Oct 05 07:25:47 Updated: newt-0.52.2-12.el5_4.1.x86_64
Oct 05 07:26:08 Installed: kernel-2.6.18-164.2.1.el5.x86_64
Oct 05 07:26:09 Updated: 1:autofs-5.0.1-0.rc2.131.el5_4.1.x86_64
Oct 05 07:26:09 Updated: openssh-server-4.3p2-36.el5_4.2.x86_64
Oct 05 07:26:12 Updated: kernel-headers-2.6.18-164.2.1.el5.x86_64
Oct 05 07:26:17 Updated: tzdata-2009m-1.el5.noarch
Oct 05 07:26:21 Updated: elinks-0.11.1-6.el5_4.1.x86_64
Oct 05 07:26:22 Updated: openssh-clients-4.3p2-36.el5_4.2.x86_64
Oct 05 07:26:22 Updated: neon-devel-0.25.5-10.el5_4.1.x86_64
Oct 05 07:26:24 Updated: neon-devel-0.25.5-10.el5_4.1.i386
Oct 05 07:26:24 Updated: 1:dmidecode-2.10-2.el5.x86_64
Oct 05 07:26:24 Updated: USG_AuditFiles-2.4-1.x86_64

Eitherway it works now... I say i just stop touching it.

Thanks for your help!
chakkerz

Comment 11 James Antill 2009-10-06 14:23:13 UTC
Was createrepo running on directories over NFS/autofs? Did you reboot into the new kernel?

> Eitherway it works now... I say i just stop touching it.

 Good plan :)

Comment 12 Christian Unger 2009-10-09 01:06:59 UTC
I did reboot into the new kernel. No NFS or AutoFS on that host. 

Seems to be holding up though ...


Note You need to log in before you can comment on or make changes to this bug.