Bug 1391236
| Summary: | etcd Backup Fails During Upgrade | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Nick Schuetz <nschuetz> |
| Component: | Cluster Version Operator | Assignee: | Jason DeTiberus <jdetiber> |
| Status: | CLOSED DUPLICATE | QA Contact: | Anping Li <anli> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 3.3.1 | CC: | aos-bugs, aquiroga, chernand, jokerman, mmccomas, nschuetz, rajveer.singh, tobias.genannt |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-11-03 14:15:29 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I ran into the same issue here.
`ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_3/upgrade.yml`
```
TASK [Generate etcd backup] ****************************************************
fatal: [ose3-master.example.com]: FAILED! => {
"changed": true,
"cmd": [
"etcdctl",
"backup",
"--data-dir=/var/lib/origin/openshift.local.etcd",
"--backup-dir=/var/lib/origin/etcd-backup-20161102131456"
],
"delta": "0:00:00.207127",
"end": "2016-11-02 13:14:56.599467",
"failed": true,
"rc": 1,
"start": "2016-11-02 13:14:56.392340",
"warnings": []
}
STDERR:
2016-11-02 13:14:56.404638 W | snap: skipped unexpected non snapshot file db
2016-11-02 13:14:56.491595 W | wal: ignored file 0.tmp in wal
2016-11-02 13:14:56.592686 I | walpb: crc mismatch
```
*** This bug has been marked as a duplicate of bug 1382634 *** Workaround for now in my case after it fails since it seems to have created a backup:
1. vim common/openshift-cluster/upgrades/pre.yml
2. Modify the task:
- name: Generate etcd backup
command: >
etcdctl backup --data-dir={{ openshift.etcd.etcd_data_dir }}
--backup-dir={{ openshift.common.data_dir }}/etcd-backup-{{ timestamp }}
tags:
- backup_etcd
Notice that the task now contain a tag.
3. ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_3/upgrade.yml --skip-tags=backup_etcd
Full path for file: /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/pre.yml New filepath /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/upgrade_control_plane.yml |
Description of problem: Version-Release number of selected component (if applicable): 3.3.1.3 How reproducible: Steps to Reproduce: 1. Run the 3.3.1 upgrade playbook against OCP 3.3.0.35 single master installation. 2. 3. Actual results: TASK [Generate etcd backup] **************************************************** fatal: [ocp-master1]: FAILED! => { "changed": true, "cmd": [ "etcdctl", "backup", "--data-dir=/var/lib/origin/openshift.local.etcd", "--backup-dir=/var/lib/origin/etcd-backup-20161102150833" ], "delta": "0:00:00.206549", "end": "2016-11-02 15:08:34.163924", "failed": true, "rc": 1, "start": "2016-11-02 15:08:33.957375", "warnings": [] } STDERR: 2016-11-02 15:08:34.009609 W | snap: skipped unexpected non snapshot file db 2016-11-02 15:08:34.052447 W | wal: ignored file 56.tmp in wal 2016-11-02 15:08:34.052461 W | wal: ignored file 76.tmp in wal 2016-11-02 15:08:34.161967 I | walpb: crc mismatch Expected results: Upgrade does not fail and etcd backup succeeds. Additional info: