RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1229353 - Checking status with snapper receiving Failure (org.freedesktop.DBus.Error.NoReply)
Summary: Checking status with snapper receiving Failure (org.freedesktop.DBus.Error.No...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: snapper
Version: 7.1
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Ondrej Kozina
QA Contact: Bruno Goncalves
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-08 14:10 UTC by John Pittman
Modified: 2021-09-03 12:39 UTC (History)
7 users (show)

Fixed In Version: snapper-0.1.7-10.el7
Doc Type: Bug Fix
Doc Text:
Cause: Snapper provides API to deliver file list with modifications made in between two snapshots to its dbus clients. No matter the file list size, the list was transferred via single dbus message over the bus. This was design mistake in snapper since dbus protocol has hard coded limit of message size set to 128 MiB. Consequence: If client asked for file list of modifications that would result in being bigger than 128 MiB after the marshalling took place on server side, the client always received the answer org.freedesktop.DBus.Error.NoReply and snapperd daemon exited with error code = 1 Fix: File list is transferred using pipe IPC between snapper daemon and clients. Result: New DBus method GetFilesByPipe() returns opened pipe file descriptot for client to read file list from. Clients can receive larger file lists and snapper daemon doesn't exit with return code = 1 when the file list is bigger than approximately 128 MiB anymore
Clone Of:
: 1231684 (view as bug list)
Environment:
Last Closed: 2015-11-19 14:39:00 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
File list causing snapperd to exit with error (1.40 MB, application/x-xz)
2015-07-10 08:25 UTC, Ondrej Kozina
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2426 0 normal SHIPPED_LIVE snapper bug fix update 2015-11-19 11:35:00 UTC

Description John Pittman 2015-06-08 14:10:26 UTC
Description of problem:

Checking the status with snapper gives:

# time snapper -c srv status 0..1
Failure (org.freedesktop.DBus.Error.NoReply).
real    2m0.730s
user    0m0.005s
sys     0m0.002s

Version-Release number of selected component (if applicable):

snapper-0.1.7-7.el7.x86_64
snapper-libs-0.1.7-7.el7.x86_64

From customer:

Script started on Fri 22 May 2015 07:56:21 AM EDT

# df /srv
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/data-srv 45065728 16983340 28082388 38% /srv

# cd /srv

# chown -R root: .

# chmod -R 755 .

# ls -al /srv
total 20
drwxr-xr-x. 7 root root 55 May 14 13:51 .
dr-xr-xr-x. 19 root root 4096 May 14 12:57 ..
drwxr-xr-x. 20 root root 4096 May 7 13:35 1
drwxr-xr-x. 20 root root 4096 May 7 13:35 2
drwxr-xr-x. 20 root root 4096 May 7 13:35 3
drwxr-xr-x. 20 root root 4096 May 7 13:35 4
drwxr-xr-x. 2 root root 6 May 22 07:52 .snapshots <<< root:root 755

# snapper -c srv create

# snapper -c srv list
Type | # | Pre # | Date | User | Cleanup | Description | Userdata
-------+---+-------+---------------------------------+------+---------+-------------+---------
single | 0 | | | root | | current |
single | 1 | | Fri 22 May 2015 07:56:43 AM EDT | root | | |

# chown -R test: * <<< change everything except "." files
chmod -R 123 *
ls -al /srv

# chmod -R 123 * <<< change everything except "." files

# ls -al /srv
total 20
drwxr-xr-x. 7 root root 55 May 14 13:51 .
dr-xr-xr-x. 19 root root 4096 May 14 12:57 ..
d--x-w--wx. 20 test test 4096 May 7 13:35 1
d--x-w--wx. 20 test test 4096 May 7 13:35 2
d--x-w--wx. 20 test test 4096 May 7 13:35 3
d--x-w--wx. 20 test test 4096 May 7 13:35 4
drwxr-xr-x. 3 root root 14 May 22 07:56 .snapshots <<<<< still root:root mode 755

# snapper -c srv create

# snapper -c srv list
Type | # | Pre # | Date | User | Cleanup | Description | Userdata
-------+---+-------+---------------------------------+------+---------+-------------+---------
single | 0 | | | root | | current |
single | 1 | | Fri 22 May 2015 07:56:43 AM EDT | root | | |
single | 2 | | Fri 22 May 2015 07:58:25 AM EDT | root | | |
# snapper -c srv status 1..2
Failure (org.freedesktop.DBus.Error.NoReply). <<< same failure from snapper

# exit
Script done on Fri 22 May 2015 08:01:40 AM EDT

The customer re-ran his "status" command and captured the 'snapper -s -d' output and it appears
to be the same as in my testing.  Uploading his capture file snapperd_debug.tgz

Actual results:

Failure (org.freedesktop.DBus.Error.NoReply)

Expected results:

Error free command

**Please let me know if you need anything.  I will be glad to do what I can on my end.

Comment 5 Ondrej Kozina 2015-06-08 17:50:24 UTC
I have a feeling this is already fixed in upstream or at least I recall some bug report for upstream snapper with similar descriptipons. Let me check that...

Comment 8 Ondrej Kozina 2015-06-09 09:51:55 UTC
Anyway I would like to split the bugzilla in two. The bug dealing with segfault and second one dealing with NoReply error.

The thing with timeouts: snapper should _not_ ever timeout as the client calls dbus_connection_send_with_reply_and_block() setting timeout parm to infinite. So it has to be something else. Could you paste here snapper daemon debug log capturing the NoReply error returned by cmd line utility?

Comment 9 Ondrej Kozina 2015-06-12 08:39:04 UTC
Hey John, I'v got the debug data and I think I know what went wrong. The bug exists also in current upstream snapper. Give me some time to investigate how difficult would it be to fix it...

Comment 17 Ondrej Kozina 2015-07-10 08:25:07 UTC
Created attachment 1050607 [details]
File list causing snapperd to exit with error

Comment 18 Ondrej Kozina 2015-07-10 08:25:40 UTC
Hi the reproduce is a little bit longer and time consuming if you decide to take path B. Use path A but for the sake of completeness you can play with path B if you like:

Also, I demonstrate the issue on lvm backend but btrfs is affected as well. So it's not related only to lvm(xfs) or lvm(ext*) backends.

1) create volume group named 'vg' (vgcreate vg /dev/sdx)
2) create thin volume named 'thin_lv' (lvcreate -L5G -T vg/pool -V 30G --name thin_lv) [5GiBs should be enough for pool size or test device size]
3) mkfs.xfs /dev/vg/thin_lv
4) mkdir /mnt/test
5) mount /dev/vg/thin_lv /mnt/test
6) create dummy user named e.g. 'dummy_a' (useradd -m dummy_a)
7) chown dummy_a:dummy_a /mnt/test
8) snapper -c bugtest create-config -f "lvm(xfs)" /mnt/test

Now take either path A, or path B. Both leads to same error but path B will take long time to finish:

Path A:

a9) snapper -c bugtest create -t pre (required to result in snapshot No: 1)
a10) snapper -c bugtest create -t post --pre-num 1
a11) pkill snapperd
a12) copy unpacked attachement in directory /mnt/test/.snapshots/2/ (filelist-1.txt)
a13) snapper -c bugtests status 1..2

Path B (this is correct reproducer from user's perspective but took me long time to finish on my vm):

b9) log in as user 'dummy_a'
b10) cd /mnt/test
b11) run script: "for i in $(seq 1 800000); do mkdir ./dir_$i; touch ./dir_$i/file_$i; done"
b12) as root run: snapper -c bugtest create -t pre (required to result in snapshot No: 1)
b13) as dummy_a run: chmod -R g-w /mnt/test/
b14) as root run: snapper -c bugtest create -t post --pre-num 1
a14) as root run: snapper -c bugtests status 1..2

Both ways should result in response:
Failure (org.freedesktop.DBus.Error.NoReply) after last step

and snapperd daemon will exit with retval == 1 and unfortunately no meaningful log message even in --debug mode.

Comment 19 Ondrej Kozina 2015-07-10 12:13:40 UTC
Thanks to upstream, we have a work around. Unfortunately the workaround is temporary only (note that you need to restart dbus service after increasing the limit):

https://github.com/openSUSE/snapper/issues/176#issuecomment-120321717

If the message transferred over dbus gets larger than 128MB we hit the hard dbus limit for message passed over the bus. IOW the workaround has its own limits as well...

Comment 24 Ondrej Kozina 2015-08-28 08:06:08 UTC
Regarding the step b12) and later:

you can use -p option to print out snapshot number created by the snapper create command:
snapper -c bugtest create -t pre -p

use number resulting in command above in step b14. So If the number was i.e 4 the b14 would look like:

snapper -c bugtest create -t post --pre-num 4 -p

Finally if the number printed out as result of line above was 5 the last command in path be would be:

snapper -c bugtest status 4..5

You can always run "snapper -c bugtest list" to see list of all snapshots

Comment 25 Bruno Goncalves 2015-08-28 14:47:42 UTC
Tested with snapper snapper-0.1.7-10.el7
kernel 3.10.0-306.0.1.el7

# vgs
  VG                  #PV #LV #SN Attr   VSize   VFree 
  rhel_dell-pe2900-02   1   3   0 wz--n- 202.76g 50.06g

# lvcreate -L5G -T rhel_dell-pe2900-02/pool -V 30G --name thin_lv
  WARNING: Sum of all thin volume sizes (30.00 GiB) exceeds the size of thin pool rhel_dell-pe2900-02/pool (5.00 GiB)!
  For thin pool auto extension activation/thin_pool_autoextend_threshold should be below 100.
  Logical volume "thin_lv" created.


# mkfs.xfs /dev/rhel_dell-pe2900-02/thin_lv
meta-data=/dev/rhel_dell-pe2900-02/thin_lv isize=256    agcount=16, agsize=491504 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=7864064, imaxpct=25
         =                       sunit=16     swidth=16 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=4096   blocks=3840, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

# mkdir /mnt/test

# mount /dev/rhel_dell-pe2900-02/thin_lv /mnt/test

# useradd -m dummy_a

# chown dummy_a:dummy_a /mnt/test

# snapper -c bugtest create-config -f "lvm(xfs)" /mnt/test


####  Path A  ####

# snapper -c bugtest create -t pre

# snapper -c bugtest create -t post --pre-num 1

# pkill snapperd

# xz -d filelist-1.txt.xz

# cp filelist-1.txt /mnt/test/.snapshots/2/
cp: overwrite ‘/mnt/test/.snapshots/2/filelist-1.txt’? y

# snapper -c bugtest status 1..2
<snip>
.p.... /mnt/test/dir_99999/file_99999

# echo $?
0


#### Path B  ####
$ cd /mnt/test

$ for i in $(seq 1 800000); do mkdir ./dir_$i; touch ./dir_$i/file_$i; done

$ exit 

# snapper -c bugtest create -t pre -p
5

# su - dummy_a
$ chmod -R g-w /mnt/test
chmod: changing permissions of ‘/mnt/test/.snapshots’: Operation not permitted
chmod: cannot read directory ‘/mnt/test/.snapshots’: Permission denied

$ exit


# snapper -c bugtest create -t post --pre-num 5 -p
6

# snapper -c bugtest status 5..6
<snip>
.p.... /mnt/test/dir_99999/file_99999

# snapper -c bugtest  list
Type   | # | Pre # | Date                         | User | Cleanup  | Description | Userdata
-------+---+-------+------------------------------+------+----------+-------------+---------
single | 0 |       |                              | root |          | current     |         
single | 1 |       | Fri 28 Aug 2015 05:01:01 EDT | root | timeline | timeline    |         
single | 2 |       | Fri 28 Aug 2015 06:01:01 EDT | root | timeline | timeline    |         
single | 3 |       | Fri 28 Aug 2015 07:01:01 EDT | root | timeline | timeline    |         
single | 4 |       | Fri 28 Aug 2015 08:01:02 EDT | root | timeline | timeline    |         
pre    | 5 |       | Fri 28 Aug 2015 08:08:56 EDT | root |          |             |         
post   | 6 | 5     | Fri 28 Aug 2015 08:13:55 EDT | root |          |             |

Comment 29 errata-xmlrpc 2015-11-19 14:39:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2426.html


Note You need to log in before you can comment on or make changes to this bug.