Bug 431688

Summary: Backtrace from amcheck when next tape not found
Product: Red Hat Enterprise Linux 5 Reporter: Dave Miller <dave>
Component: amandaAssignee: Radek Brich <rbrich>
Status: CLOSED NOTABUG QA Contact:
Severity: low Docs Contact:
Priority: low    
Version: 5.1   
Target Milestone: rc   
Target Release: ---   
Hardware: i386   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2008-03-19 09:36:25 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dave Miller 2008-02-06 14:54:58 UTC
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.0.12) Gecko/20071130 CentOS/1.5.0.12-7.el5.centos Firefox/1.5.0.12

Description of problem:
Everything appears to be working except amcheck when my tape changer needs to have the tapes changed. That is, amcheck runs just fine if the next tape is found. amdump runs just fine. amtape doesn't seem to have a problem. But when it's time to swap tapes I get:

*** glibc detected *** /usr/sbin/amcheck: double free or corruption (fasttop): 0x0868d998 ***
======= Backtrace: =========
/lib/libc.so.6[0x1edaa6]
/lib/libc.so.6(cfree+0x90)[0x1f0fc0]
/usr/sbin/amcheck[0xd4ce03]
/usr/sbin/amcheck(main+0xc2d)[0xd4e91d]
/lib/libc.so.6(__libc_start_main+0xdc)[0x19cdec]
/usr/sbin/amcheck[0xd49a01]
======= Memory map: ========
0016c000-00185000 r-xp 00000000 09:02 2378079    /lib/ld-2.5.so
00185000-00186000 r-xp 00019000 09:02 2378079    /lib/ld-2.5.so
00186000-00187000 rwxp 0001a000 09:02 2378079    /lib/ld-2.5.so
00187000-002c1000 r-xp 00000000 09:02 2378081    /lib/libc-2.5.so
002c1000-002c3000 r-xp 0013a000 09:02 2378081    /lib/libc-2.5.so
002c3000-002c4000 rwxp 0013c000 09:02 2378081    /lib/libc-2.5.so
002c4000-002c7000 rwxp 002c4000 00:00 0 
00564000-00565000 r-xp 00564000 00:00 0          [vdso]
009e8000-00a09000 r-xp 00000000 09:02 4675888    /usr/lib/libamanda-2.5.0p2.so
00a09000-00a0a000 rwxp 00021000 09:02 4675888    /usr/lib/libamanda-2.5.0p2.so
00a0a000-00a29000 rwxp 00a0a000 00:00 0 
00a63000-00a83000 r-xp 00000000 09:02 4953736    /usr/lib/libamserver-2.5.0p2.so
00a83000-00a85000 rwxp 0001f000 09:02 4953736    /usr/lib/libamserver-2.5.0p2.so
00a85000-00a87000 rwxp 00a85000 00:00 0 
00b2e000-00b3d000 r-xp 00000000 09:02 2378128    /lib/libresolv-2.5.so
00b3d000-00b3e000 r-xp 0000e000 09:02 2378128    /lib/libresolv-2.5.so
00b3e000-00b3f000 rwxp 0000f000 09:02 2378128    /lib/libresolv-2.5.so
00b3f000-00b41000 rwxp 00b3f000 00:00 0 
00bcb000-00bd6000 r-xp 00000000 09:02 2378249    /lib/libgcc_s-4.1.2-20070626.so.1
00bd6000-00bd7000 rwxp 0000a000 09:02 2378249    /lib/libgcc_s-4.1.2-20070626.so.1
00c82000-00c85000 r-xp 00000000 09:02 2378258    /lib/libtermcap.so.2.0.8
00c85000-00c86000 rwxp 00002000 09:02 2378258    /lib/libtermcap.so.2.0.8
00c9f000-00ca8000 r-xp 00000000 09:02 2378088    /lib/libnss_files-2.5.so
00ca8000-00ca9000 r-xp 00008000 09:02 2378088    /lib/libnss_files-2.5.so
00ca9000-00caa000 rwxp 00009000 09:02 2378088    /lib/libnss_files-2.5.so
00d48000-00d51000 r-xp 00000000 09:02 4560646    /usr/sbin/amcheck
00d51000-00d52000 rwxp 00008000 09:02 4560646    /usr/sbin/amcheck
00d60000-00d85000 r-xp 00000000 09:02 2378087    /lib/libm-2.5.so
00d85000-00d86000 r-xp 00024000 09:02 2378087    /lib/libm-2.5.so
00d86000-00d87000 rwxp 00025000 09:02 2378087    /lib/libm-2.5.so
00ece000-00ee1000 r-xp 00000000 09:02 2378125    /lib/libnsl-2.5.so
00ee1000-00ee2000 r-xp 00012000 09:02 2378125    /lib/libnsl-2.5.so
00ee2000-00ee3000 rwxp 00013000 09:02 2378125    /lib/libnsl-2.5.so
00ee3000-00ee5000 rwxp 00ee3000 00:00 0 
00f46000-00f50000 r-xp 00000000 09:02 4674895    /usr/lib/libamtape-2.5.0p2.so
00f50000-00f51000 rwxp 00009000 09:02 4674895    /usr/lib/libamtape-2.5.0p2.so
08687000-086a8000 rw-p 08687000 00:00 0 
b7e00000-b7e21000 rw-p b7e00000 00:00 0 
b7e21000-b7f00000 ---p b7e21000 00:00 0 
b7f19000-b7f1c000 rw-p b7f19000 00:00 0 
b7f25000-b7f27000 rw-p b7f25000 00:00 0 
bf8d2000-bf8e8000 rw-p bf8d2000 00:00 0          [stack]
Amanda Tape Server Host Check
-----------------------------
Holding disk /share2/stage: 90535524 kB disk space available, using 90433124 kB


Amanda Backup Client Hosts Check
--------------------------------
Client check: 4 hosts checked in 0.356 seconds, 0 problems found

(brought to you by Amanda 2.5.0p2)

I guess it works in the sense that, if I get a traceback, I know I need to swap tapes. This seems a little inelegant. This is with CentOS 5.1 (RHEL clone) and all updates applied. The tape changer is a:

# cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 04 Lun: 00
Vendor: HP Model: C1557A Rev: U709
Type: Sequential-Access ANSI SCSI revision: 02
Host: scsi0 Channel: 00 Id: 04 Lun: 01
Vendor: HP Model: C1557A Rev: U709
Type: Medium Changer ANSI SCSI revision: 02

The folks at Zmanda suggested upgrading to the next release.

Version-Release number of selected component (if applicable):
amanda-2.5.0p2-4
amanda-server-2.5.0p2-4 

How reproducible:
Always


Steps to Reproduce:
1. Make sure the "next" tape isn't currently in the tape changer.
2. Run amcheck


Actual Results:
I get the backtrace described above.

Expected Results:
amcheck should have run to completion and then complained that the next tape was not in the tape changer.

Additional info:
Tape changer information is as described above.  Problem started to occur when the server was upgraded from CentOS 4.6 to CentOS 5.1.  That is, amcheck behaved as expected under all 4.X versions.  The upgrade was actually a clean install of 5.0 followed by letting yum upgrade the server to 5.1.

Comment 1 Dave Miller 2008-03-16 22:43:30 UTC
It appears that a recent update to one of the libraries used bt amcheck has
fixed the problem.  My cron job that runs amcheck now completes normally even if
the "next" tape is not in the changer:

Amanda Tape Server Host Check
-----------------------------
Holding disk /share2/stage: 53956964 kB disk space available, using 53854564 kB
slot 5: read label `DailySet-08', date `20080315'
slot 6: not an amanda tape (Input/output error)
slot 1: read label `DailySet-03', date `20080311'
slot 2: read label `DailySet-05', date `20080312'
slot 3: read label `DailySet-06', date `20080313'
slot 4: read label `DailySet-07', date `20080314'

       (expecting tape DailySet-09 or a new tape)
Server check took 355.761 seconds

Amanda Backup Client Hosts Check
--------------------------------
WARNING: spindle: selfcheck request failed: spindle: could not resolve hostname
Client check: 4 hosts checked in 9.670 seconds, 1 problem found

(brought to you by Amanda 2.5.0p2)

This bug can be closed unless someone else is seeing the problem.

Comment 2 Radek Brich 2008-03-19 09:36:25 UTC
Ok. Closing as NOTABUG (not amanda's bug).