The metadata even with latest tools just tell 'bad checksum in metadata index block' - so requires Joe's eye.

However - how the state was reached?

Taking a look at attached logs - it does look like the system started from Mar 6, 17:50 experience many
"Medium errors" and various iscsi failures  and there is tremendous amount of multipath failures.

So far this BZ does not provide any information about layout of device (even lvm2 VG metadata are missing, and those are mandatory for manual try of metadata repair - so please attach).

But if the failing storage was meant to be used for thin-provisioning metadata device - this can cause very unusual and hard to recover state.

Also note the version of lvm2 - 2.02.177(2)-RHEL7 - a bit old?
See https://access.redhat.com/downloads/content/rhel---7/x86_64/2456/lvm2/2.02.180-10.el7_6.3/x86_64/fd431d51/package-changelog -

Trying to distill a couple great questions that I've seen asked but have not yet been answered:

comment 10: Why is multipathd adding the <_tdata> as a map?  We are also getting mpath read errors - are they related?

comment 3: Why do we only see 'vgs' and 'vgdisplay' in the LVM metadata archives?  We seem to be jumping from one state to another (comment 11).

comment 12: It is odd that the affected pools come from the virtio devices - is this exclusively so?

Also, what's with all the multipath strangeness?

Restating interesting, unresolved questions:

comment 10: Why is multipathd adding the <_tdata> as a map?  We are also getting mpath read errors - are they related? [Answered in comment 21 and comment 22]

comment 3: Why do we only see 'vgs' and 'vgdisplay' in the LVM metadata archives?  We seem to be jumping from one state to another (comment 11).

comment 12: It is odd that the affected pools come from the virtio devices - is this exclusively so?

comment 22: Is there a way to distinguish between multipath medium errors and connection errors? (and why are we seeing these errors?)

comment 24: Are there multipath medium/connection errors affecting the other 15 of 16 clusters that didn't have a problem?

(In reply to Jonathan Earl Brassow from comment #25)
> Restating interesting, unresolved questions:
> 
> comment 10: Why is multipathd adding the <_tdata> as a map?  We are also
> getting mpath read errors - are they related? [Answered in comment 21 and
> comment 22]

Having been pointed to comment#10, this observation struck me as interesting:
[gluster] "bricks are not mounted on the node but there are plenty of containers"

The reason for that would seem obvious to me: no bricks were active because gluster's bricks weren't able to be activated due to thinp corruption.

> comment 12: It is odd that the affected pools come from the virtio devices -
> is this exclusively so?

Aren't virtio devices common for the upper layers? (comment#23 seems to say KVM guests are in use, and the arch overview diagram shows virtio devices.. btw that arch diagram is woefully lacking, it doesn't even show what layers are in play, e.g. no mention of dm-multipath.. yet gluster's bricks are apparently thinp layered on dm-multipath.

Think we need Gluster people to give us more context for how they are using thinp (is gluster managing dm-multipath or is it customer that setup multipath for the gluster bricks?)

> comment 22: Is there a way to distinguish between multipath medium errors
> and connection errors? (and why are we seeing these errors?)

As Ben said in comment#22, dm-multipath treats a medium error (BLK_STS_MEDIUM) as not retryable:

static inline bool blk_path_error(blk_status_t error)
{
        switch (error) {
        case BLK_STS_NOTSUPP:
        case BLK_STS_NOSPC:
        case BLK_STS_TARGET:
        case BLK_STS_NEXUS:
        case BLK_STS_MEDIUM:
        case BLK_STS_PROTECTION:
                return false;
        }

        /* Anything else could be a path failure, so should be retried */
        return true;
}

If DM thinp gets an error it will pass that error up the IO stack.  If the dm-thin-pool metadata gets an error it'll switch into fail IO mode.
I've not looked at the logs yet but at the time of the failure do the logs show any dm thin-pool metadata IO mode transitions?  Or messages of the form: "%s: aborting current metadata transaction", dev_name ?

Comment#11 shows that thin-pool data and metadata is ontop of the same virtio device (e.g. /dev/vdd).

Who created these virtio devices?  Which IO mode are they configured to use over the virtio protocol?  What is the KVM host's backing storage for these virtio devices?  Are we certain proper flushing occurs before virtio completes IO (what caching mode is virtio using?  is it passing through the underlying host storage's FLUSH/FUA flags?)

(In reply to Jonathan Earl Brassow from comment #25)
> 
> comment 3: Why do we only see 'vgs' and 'vgdisplay' in the LVM metadata
> archives?  We seem to be jumping from one state to another (comment 11).

Because heketi is controlling the creation of the volumes and it does:
       cmd == "lvcreate -qq --autobackup=<probably='n'>...
From the man page:
       -A|--autobackup y|n
              Specifies if metadata should be backed up automatically after a change.  Enabling this is strongly advised! See vgcfgbackup(8) for more information.

Created attachment 1546711 [details]
infra-1: tmeta mountinfo lvmdump

infra-1-tmeta.tar.bz2 contains a dd of tmeta, output of /proc/mountinfo, and also a lvmdump -am

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days