Bug 1366668

Summary:	BackupPC 3.3.1-4.el7 and 3.3.1-5.el7 break backups (unable to read 4 bytes)
Product:	[Fedora] Fedora EPEL	Reporter:	Ollie Armstrong <ollie>
Component:	BackupPC	Assignee:	Richard Shaw <hobbes1069>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	epel7	CC:	bjohnson, hobbes1069, kevin, pat
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-04-27 19:06:16 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Ollie Armstrong 2016-08-12 14:47:01 UTC

Description of problem:
Since upgrading to BackupPC version 3.3.1-4.el7 on CentOS 7, backups no longer run with the "Got fatal error during xfer (Unable to read 4 bytes)" error. This is resolved by downgrading to 3.3.1-1.el7.

Version-Release number of selected component (if applicable): 3.3.1-4.el7 and 3.3.1-5.el7

How reproducible: Always

Steps to Reproduce:
1. Add hosts to BackupPC
2. Run backups
3. Upgrade to 3.3.1-4.el7 or 3.3.1-5.el7
4. Run backups

Actual results:
Backups fail with "Got fatal error during xfer (Unable to read 4 bytes)".

Expected results:
Backups run.

Additional info:
Tried against hosts that existed in BackupPC prior to upgrade and adding a new host after upgrade - same issue.

XFer log contains:
full backup started for directory /
Running: /usr/bin/ssh -q -x -l root 1.1.1.1 -- --server --sender --numeric-ids --perms --owner --group -D --links --hard-links --times --block-size=2048 --recursive --ignore-times . /
Xfer PIDs are now 2809
Read EOF: Connection reset by peer
Tried again: got 0 bytes
Done: 0 files, 0 bytes
Got fatal error during xfer (Unable to read 4 bytes)
Backup aborted (Unable to read 4 bytes)
Not saving this as a partial backup since it has fewer files than the prior one (got 0 and 0 files versus 0)

(authorized_keys on host contains `command="/usr/bin/rsync $SSH_ORIGINAL_COMMAND"`)

/var/log/BackupPC/LOG:
2016-08-12 14:36:20 Started full backup on 1.1.1.1 (pid=2806, share=/)
2016-08-12 14:36:26 Backup failed on 1.1.1.1 (Unable to read 4 bytes

No SSH login attempt is recorded in /var/log/secure of the host.

Running the command printed above manually as the BackupPC user runs successfully.

Looks like only systemd stuff changed in the release, potentially something to do with running as the wrong user?

Comment 1 Persona non grata 2016-08-22 09:16:52 UTC Comment hidden (spam)

This comment was flagged as spam, view the edit history to see the original text if required.

Comment 2 Ollie Armstrong 2016-08-23 09:53:32 UTC

I'm not sure, I don't have a solution for you yet. The only difference I can see is the switch from an init.d script to the service file and the tmpfiles.d file. It isn't the tmpfile configuration causing issues as I've had an identical configuration manually there so it leave the systemd unit. It does appear functionally identical to the service script though.

I'll continue to investigate as and when I can, otherwise if I get nowhere we can work together to find the cause when you return.  Cheers, enjoy your vacation.

Comment 3 Patrick Dubois 2016-09-03 01:36:01 UTC

I believe I am also getting this error.  Running the command manually as the backuppc user does *not* complete normally however.  Seems to sit there until timeout or a key is hit producing a rsync protocol version error.

Comment 4 Patrick Dubois 2016-09-08 17:43:14 UTC

Further information to my comment #3.

Reviewing my logs it is clear my backups stopped working immediately after upgrading BackupPC to version '3.3.1-5.el7'.  Glibc, python and some other packages were also upgraded at the same time however Rsync itself did not have an update.

Downgrading the packages does not resolve the issue for me strangely enough.

Running the dump manually as the backuppc user works fine - also strangely enough: 
--
/usr/share/BackupPC/bin/BackupPC_dump  -v -f <machine_name> 
--

Running the Rsync command as reported in the logs times out and after a couple Return_Keys errors out with the rsync protocol version error:
--
/usr/bin/ssh -q -x -l root <machine_name> /usr/bin/rsync --verbose --server --sender --numeric-ids --perms --owner --group -D  --recursive . /etc/
-- 

My backup clients are a combination of Fedora 23/24 as well as Centos 6/7.  The clients were not updated at the same time as the backuppc server.  Updating the clients to the exact same version as the backuppc server does not resolve the issue.

So basically, performing the latest patches on my centos7 backuppc server has killed automatic backups(rsync protocol mismatch error) and yet I can sudo to the backuppc user and manually dump without problem.

I'm wondering if this is a interactive session thingy...

Comment 5 Patrick Dubois 2016-09-08 18:06:36 UTC

While downgrading to 3.1.1-4 did not resolve my problem, downgrading to 3.3.3-1 as suggested above DID resolve it. 

Backups proceeding normally will all patches applied with the exception of BackupPC pinned to 3.1.1-1.

Comment 6 Patrick Dubois 2016-09-08 18:36:48 UTC

Yikes,  apologies for my previous note.  Appears I require coffee.

Should have read : 

While downgrading to 3.3.1-4 did not resolve my problem, downgrading to 3.3.1-1 as suggested above DID resolve it. 

Backups proceeding normally will all patches applied with the exception of BackupPC pinned to 3.3.1-1.

My package updates dates for backuppc were : 
--
Jan 27 03:58:16 Installed: BackupPC-3.3.1-1.el7.x86_64
Aug 01 10:01:22 Updated: BackupPC-3.3.1-4.el7.x86_64
Aug 05 11:26:28 Updated: BackupPC-3.3.1-5.el7.x86_64
Sep 08 13:56:38 Installed: BackupPC-3.3.1-1.el7.x86_64
--
...and backups stopped working the afternoon of Aug 05.

Comment 7 Richard Shaw 2017-04-05 12:42:00 UTC

I have recently taken over maintenance of BackupPC. Is this still a problem? I had 3.3.1 running fine on my CentOS 7 box and have since upgraded to 4.1.1. Because manual intervention is required I am supporting 4.X on EL 6 & 7 on COPR only at this point.

https://copr.fedorainfracloud.org/coprs/hobbes1069/BackupPC/