Bug 1391236 - etcd Backup Fails During Upgrade
Summary: etcd Backup Fails During Upgrade
Keywords:
Status: CLOSED DUPLICATE of bug 1382634
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 3.3.1
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: ---
Assignee: Jason DeTiberus
QA Contact: Anping Li
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-11-02 20:20 UTC by Nick Schuetz
Modified: 2019-12-16 07:17 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-03 14:15:29 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github /coreos etcd issues 6191 0 None None None 2020-08-13 01:36:50 UTC

Description Nick Schuetz 2016-11-02 20:20:51 UTC
Description of problem:


Version-Release number of selected component (if applicable):

3.3.1.3

How reproducible:


Steps to Reproduce:
1. Run the 3.3.1 upgrade playbook against OCP 3.3.0.35 single master installation.
2.
3.

Actual results:

TASK [Generate etcd backup] ****************************************************
fatal: [ocp-master1]: FAILED! => {
    "changed": true, 
    "cmd": [
        "etcdctl", 
        "backup", 
        "--data-dir=/var/lib/origin/openshift.local.etcd", 
        "--backup-dir=/var/lib/origin/etcd-backup-20161102150833"
    ], 
    "delta": "0:00:00.206549", 
    "end": "2016-11-02 15:08:34.163924", 
    "failed": true, 
    "rc": 1, 
    "start": "2016-11-02 15:08:33.957375", 
    "warnings": []
}

STDERR:

2016-11-02 15:08:34.009609 W | snap: skipped unexpected non snapshot file db
2016-11-02 15:08:34.052447 W | wal: ignored file 56.tmp in wal
2016-11-02 15:08:34.052461 W | wal: ignored file 76.tmp in wal
2016-11-02 15:08:34.161967 I | walpb: crc mismatch


Expected results:

Upgrade does not fail and etcd backup succeeds.

Additional info:

Comment 1 Christian Hernandez 2016-11-02 20:33:30 UTC
I ran into the same issue here.

`ansible-playbook  /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_3/upgrade.yml`

```
TASK [Generate etcd backup] ****************************************************
fatal: [ose3-master.example.com]: FAILED! => {
    "changed": true, 
    "cmd": [
        "etcdctl", 
        "backup", 
        "--data-dir=/var/lib/origin/openshift.local.etcd", 
        "--backup-dir=/var/lib/origin/etcd-backup-20161102131456"
    ], 
    "delta": "0:00:00.207127", 
    "end": "2016-11-02 13:14:56.599467", 
    "failed": true, 
    "rc": 1, 
    "start": "2016-11-02 13:14:56.392340", 
    "warnings": []
}

STDERR:

2016-11-02 13:14:56.404638 W | snap: skipped unexpected non snapshot file db
2016-11-02 13:14:56.491595 W | wal: ignored file 0.tmp in wal
2016-11-02 13:14:56.592686 I | walpb: crc mismatch
```

Comment 2 Scott Dodson 2016-11-03 14:15:29 UTC

*** This bug has been marked as a duplicate of bug 1382634 ***

Comment 3 Alfredo Quiroga 2016-11-03 15:25:51 UTC
Workaround for now in my case after it fails since it seems to have created a backup:

1. vim common/openshift-cluster/upgrades/pre.yml

2. Modify the task:

  - name: Generate etcd backup
    command: >
      etcdctl backup --data-dir={{ openshift.etcd.etcd_data_dir }}
      --backup-dir={{ openshift.common.data_dir }}/etcd-backup-{{ timestamp }}
    tags:
      - backup_etcd

Notice that the task now contain a tag.

3. ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_3/upgrade.yml --skip-tags=backup_etcd

Comment 4 Alfredo Quiroga 2016-11-03 15:26:59 UTC
Full path for file:

/usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/pre.yml

Comment 5 Christian Hernandez 2016-11-04 20:14:23 UTC
New filepath

/usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/upgrade_control_plane.yml


Note You need to log in before you can comment on or make changes to this bug.