Bug 1086532

Summary: xfsdump: inv_core.c:66: get_counters: Assertion `((invt_counter_t *)(*cntpp))->ic_vernum == (inv_version_t) 1' failed
Product: Red Hat Enterprise Linux 7 Reporter: Eryu Guan <eguan>
Component: xfsdumpAssignee: Eric Sandeen <esandeen>
Status: CLOSED ERRATA QA Contact: Zorro Lang <zlang>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.1CC: swhiteho, xzhou, zlang
Target Milestone: rcKeywords: Rebase
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: xfsdump-3.1.7-el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-04-10 18:28:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1113520    

Description Eryu Guan 2014-04-11 05:34:32 UTC
Description of problem:
I saw one xfsdump Assertion failure when running xfstests xfs/302 on 1k block size xfs/s390x host with loop device

meta-data=/dev/loop1             isize=256    agcount=4, agsize=3145984 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0
data     =                       bsize=1024   blocks=12583936, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=1024   blocks=10240, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
/usr/sbin/xfsdump: using file dump (drive_simple) strategy
/usr/sbin/xfsdump: using file dump (drive_simple) strategy
/usr/sbin/xfsdump: version 3.1.3 (dump format 3.0)
/usr/sbin/xfsdump: level 0 dump of ibm-z10-53.rhts.eng.bos.redhat.com:/mnt/testarea/scratch
/usr/sbin/xfsdump: dump date: Thu Apr 10 04:05:56 2014
/usr/sbin/xfsdump: session id: 23e1d825-3322-4564-a37d-8b8acf483438
/usr/sbin/xfsdump: session label: "session"
/usr/sbin/xfsdump: ino map phase 1: constructing initial dump list
/usr/sbin/xfsdump: ino map phase 2: skipping (no pruning necessary)
/usr/sbin/xfsdump: ino map phase 3: identifying stream starting points
/usr/sbin/xfsdump: stream 0: ino 68 offset 0 to ino 70 offset 0
/usr/sbin/xfsdump: stream 1: ino 70 offset 0 to end
/usr/sbin/xfsdump: ino map construction complete
/usr/sbin/xfsdump: estimated dump size: 22400 bytes
/usr/sbin/xfsdump: estimated dump size per stream: 21536 bytes
/usr/sbin/xfsdump: /var/lib/xfsdump/inventory created
/usr/sbin/xfsdump: drive 0: creating dump session media file 0 (media 0, file 0)
/usr/sbin/xfsdump: drive 0: dumping ino map
/usr/sbin/xfsdump: drive 0: dumping directories
/usr/sbin/xfsdump: drive 1: creating dump session media file 0 (media 0, file 0)
/usr/sbin/xfsdump: drive 1: dumping ino map
/usr/sbin/xfsdump: drive 1: dumping non-directory files
/usr/sbin/xfsdump: drive 0: dumping non-directory files
/usr/sbin/xfsdump: drive 0: ending media file
/usr/sbin/xfsdump: drive 1: ending media file
/usr/sbin/xfsdump: drive 0: media file size 22064 bytes
/usr/sbin/xfsdump: drive 1: media file size 21312 bytes
/usr/sbin/xfsdump: drive 1: INV : Unknown version 0 - Expected version 1
xfsdump: inv_core.c:66: get_counters: Assertion `((invt_counter_t *)(*cntpp))->ic_vernum == (inv_version_t) 1' failed.
dump failed

But unfortunately I cannot reproduce it manually, and the original fs image was lost in beaker too..

File one bug for future reference.

Version-Release number of selected component (if applicable):
xfsdump-3.1.3-5.el7

How reproducible:
Unknown

Steps to Reproduce:
1. check xfs/302 on s390x
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Eryu Guan 2014-04-11 05:35:28 UTC
The console log has nothing special either.

[ 7745.104961] XFS (loop0): Mounting Filesystem
[ 7745.108492] XFS (loop0): Ending clean mount
[ 7745.108505] SELinux: initialized (dev loop0, type xfs), uses mountpoint labeling
[ 7745.268851] XFS (loop1): Mounting Filesystem
[ 7745.270631] XFS (loop1): Ending clean mount
[ 7745.270643] SELinux: initialized (dev loop1, type xfs), uses mountpoint labeling
[ 7745.398700] XFS (loop0): Mounting Filesystem
[ 7745.402053] XFS (loop0): Ending clean mount
[ 7745.402061] SELinux: initialized (dev loop0, type xfs), uses xattr
[ 7745.724712] XFS (loop1): Mounting Filesystem
[ 7745.726564] XFS (loop1): Ending clean mount
[ 7745.726579] SELinux: initialized (dev loop1, type xfs), uses mountpoint labeling
[ 7756.211997] XFS (loop1): Mounting Filesystem
[ 7756.214030] XFS (loop1): Ending clean mount
[ 7756.214039] SELinux: initialized (dev loop1, type xfs), uses xattr
[ 7756.327907] XFS (loop0): Mounting Filesystem
[ 7756.330840] XFS (loop0): Ending clean mount
[ 7756.330846] SELinux: initialized (dev loop0, type xfs), uses xattr

Comment 2 Eric Sandeen 2014-07-15 16:52:39 UTC
No reproducer, see comment #0

Comment 3 Eric Sandeen 2014-10-22 16:38:42 UTC
Moving to 7.2, not sure how to reproduce this one.

Comment 4 Eryu Guan 2015-04-13 08:09:21 UTC
See again on upstream 4.0-rc5 kernel with xfsdump-3.1.4-1.el7, run xfstests with MKFS_OPTIONS="-m crc=1 -b size=2048" on x86_64 host.

But still, it's not reproducible at will

xfs/302.full:
meta-data=/dev/sda5              isize=512    agcount=4, agsize=1310720 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0
data     =                       bsize=2048   blocks=5242880, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=2048   blocks=5120, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
/usr/sbin/xfsdump: using file dump (drive_simple) strategy
/usr/sbin/xfsdump: using file dump (drive_simple) strategy
/usr/sbin/xfsdump: version 3.1.4 (dump format 3.0)
/usr/sbin/xfsdump: level 0 dump of hp-sl390s-01.rhts.eng.bos.redhat.com:/mnt/xfstests/mnt2
/usr/sbin/xfsdump: dump date: Wed Mar 25 19:21:21 2015
/usr/sbin/xfsdump: session id: 7a9d6974-7aa7-427f-ad5a-e7ecc3144bd5
/usr/sbin/xfsdump: session label: "session"
/usr/sbin/xfsdump: ino map phase 1: constructing initial dump list
/usr/sbin/xfsdump: ino map phase 2: skipping (no pruning necessary)
/usr/sbin/xfsdump: ino map phase 3: identifying stream starting points
/usr/sbin/xfsdump: stream 0: ino 36 offset 0 to ino 38 offset 0
/usr/sbin/xfsdump: stream 1: ino 38 offset 0 to end
/usr/sbin/xfsdump: ino map construction complete
/usr/sbin/xfsdump: estimated dump size: 22400 bytes
/usr/sbin/xfsdump: estimated dump size per stream: 21536 bytes
/usr/sbin/xfsdump: /var/lib/xfsdump/inventory created
/usr/sbin/xfsdump: drive 0: creating dump session media file 0 (media 0, file 0)
/usr/sbin/xfsdump: drive 0: dumping ino map
/usr/sbin/xfsdump: drive 1: creating dump session media file 0 (media 0, file 0)
/usr/sbin/xfsdump: drive 0: dumping directories
/usr/sbin/xfsdump: drive 1: dumping ino map
/usr/sbin/xfsdump: drive 1: dumping non-directory files
/usr/sbin/xfsdump: drive 0: dumping non-directory files
/usr/sbin/xfsdump: drive 1: ending media file
/usr/sbin/xfsdump: drive 0: ending media file
/usr/sbin/xfsdump: drive 1: media file size 21312 bytes
/usr/sbin/xfsdump: drive 0: media file size 22064 bytes
/usr/sbin/xfsdump: drive 0: INV : Unknown version 0 - Expected version 1
xfsdump: inv_core.c:66: get_counters: Assertion `((invt_counter_t *)(*cntpp))->ic_vernum == (inv_version_t) 1' failed.
dump failed

Comment 6 Eryu Guan 2016-05-03 05:48:28 UTC
This can be reproduced by running multi-stream dump in a tight loop

  mount /dev/≤dev> /mnt/xfs
  mkdir /mnt/xfs/dumpdir
  # populate dumpdir here
  # then run the loop
  while xfsdump -M l1 -M l2 -f d1 -f d2 -L ses /mnt/xfs -s dumpdir; do
  	:
  done

And I sent a patch to upstream, but I'm not sure if that's the correct fix

http://permalink.gmane.org/gmane.comp.file-systems.xfs.general/74734

Comment 7 Steve Whitehouse 2016-12-07 12:24:03 UTC
So from comment #6 we think this is a test issue rather than an xfs_dump issue?

Comment 8 Eryu Guan 2016-12-07 13:33:57 UTC
The fix in comment 6 is for xfs_dump, I think it's an xfs_dump issue, but patch got no comments yet.

Comment 9 Eric Sandeen 2017-07-14 22:33:44 UTC
Patch has been merged upstream,

5903b44 xfsdump: fix race condition between lseek() and read()/write()


Thanks Eryu, and sorry for the 15-month delay :)

Comment 11 Eryu Guan 2017-07-15 04:42:51 UTC
(In reply to Eric Sandeen from comment #9)
> Patch has been merged upstream,
> 
> 5903b44 xfsdump: fix race condition between lseek() and read()/write()
> 
> 
> Thanks Eryu, and sorry for the 15-month delay :)

No problem, thanks for the review! :)

(Also provide qa_ack+ and update QA whiteboard to reflect that multiple tests could hit this failure.)

Comment 12 Eric Sandeen 2017-11-03 14:59:50 UTC
As it turns out, xfsdump in rhel7 no longer even builds due to xfsprogs changes & rebases since RHEL7.0 GA.

I'm going to rebase xfsdump to 3.1.7 to pick up the many, many build fixes introduced since GA - makes more sense than cherry picking a dozen patches, I think. That will also pick up the fix for this bug ...

Comment 17 errata-xmlrpc 2018-04-10 18:28:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0986