Bug 2276705 - After upgrade to Fedora 40, deja-dup fails with "Could not log into Google servers. <data>:6.3: Parse error: unexpected identifier `ww', expected end of file"
Summary: After upgrade to Fedora 40, deja-dup fails with "Could not log into Google se...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: deja-dup
Version: 40
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Gwyn Ciesla
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-04-23 19:28 UTC by Jonathan Billings
Modified: 2024-05-10 06:11 UTC (History)
10 users (show)

Fixed In Version: deja-dup-45.2-4.fc40 deja-dup-45.2-4.fc39
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-05-01 04:14:33 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)
Error about missing packages (62.33 KB, image/png)
2024-04-23 19:29 UTC, Jonathan Billings
no flags Details
Error from Google Servers (75.70 KB, image/png)
2024-04-23 19:29 UTC, Jonathan Billings
no flags Details

Description Jonathan Billings 2024-04-23 19:28:07 UTC
I had Deja-Dup set up to back up to Google Drive before upgrading to Fedora 40, however, after the update today, many of my colleagues and I are seeing an error when we try to start a backup.





Reproducible: Always

Steps to Reproduce:
1. Have Deja-dup installed and working in Fedora 39
2. Update to Fedora 40
3. Run 'deja-dup --backup'
Actual Results:  
First, there's a briefly appearing window that says, "In order to continue, the following packages need to be installed:" (and no actual package mentioned), and then it shows a window with, "Could not log into Google servers. <data>:6.3: Parse error: unexpected identifier `ww', expected end of file".  I will include screenshots.

Expected Results:  
Normal backups appear.

Running with DEJA_DUP_DEBUG=1 doesn't change any of the output on the command line.  The ~/.cache/deja-dup/duplicity.log doesn't show any recent activity.

Comment 1 Jonathan Billings 2024-04-23 19:29:03 UTC
Created attachment 2028576 [details]
Error about missing packages

Comment 2 Jonathan Billings 2024-04-23 19:29:38 UTC
Created attachment 2028577 [details]
Error from Google Servers

Comment 3 Gwyn Ciesla 2024-04-23 21:35:51 UTC
Odd. What version of python3-PyDrive2 is installed?

Comment 4 Michael Terry 2024-04-24 02:39:36 UTC
Here's some odd things I see here:
- I don't believe Fedora builds with packagekit support enabled (you have to manually turn it on with -Dpackagekit=enabled -- the meson default is "disabled", not "auto")
- So that dialog should not be possible to trigger, since no code should ask to show it.
- There should be names of packages to install in that dialog (if the list is empty, we don't show the dialog -- so this would mean that libpackagekit gave back empty strings for package names... doesn't seem likely)
- Why does that dialog not have Cancel/Install buttons in the header? That would happen for a progress screen, not the install-packages screen.

Altogether, I'm confused. It seems cheap to jump to this theory, but could there be a corrupted memory issue from bad memory management somewhere?

Comment 5 Jonathan Billings 2024-04-24 11:59:57 UTC
(In reply to Gwyn Ciesla from comment #3)
> Odd. What version of python3-PyDrive2 is installed?

python3-PyDrive2-1.19.0-3.fc40.noarch

Comment 6 Jonathan Billings 2024-04-24 12:20:01 UTC
(In reply to Michael Terry from comment #4)
> Here's some odd things I see here:
> - I don't believe Fedora builds with packagekit support enabled (you have to
> manually turn it on with -Dpackagekit=enabled -- the meson default is
> "disabled", not "auto")
> - So that dialog should not be possible to trigger, since no code should ask
> to show it.
> - There should be names of packages to install in that dialog (if the list
> is empty, we don't show the dialog -- so this would mean that libpackagekit
> gave back empty strings for package names... doesn't seem likely)
> - Why does that dialog not have Cancel/Install buttons in the header? That
> would happen for a progress screen, not the install-packages screen.
> 
> Altogether, I'm confused. It seems cheap to jump to this theory, but could
> there be a corrupted memory issue from bad memory management somewhere?

Rebuilding deja-dup's package with -Dpackagekit=enabled eliminates the first dialog about installing packages, but does not fix the issue with Google drive.

One thing I noticed is that the default version of Duplicity in Fedora 39 is v2.1.5, but in Fedora 40 is 2.2.3.

Comment 7 Michael Terry 2024-04-24 12:25:24 UTC
It is delightfully confusing that enabling packagekit prevented that dialog from showing up.

======•

This does not address the immediate issue, but I’ll note that the upcoming 46.0 release switches to rclone for connecting to Google Drive & Microsoft OneDrive.

Duplicity is changing their preferred Drive backend off of pydrive and I figured if I had to switch anyway, might as well unify cloud support onto rclone instead of several small Python libraries.

Comment 8 Jonathan Billings 2024-04-24 12:35:39 UTC
I tried downgrading duplicity to v2.1.5, deja-dup to the version in fedora 39 (v45.2-1) as well as the PyDrive2 package in fedora 39, none seem to have changed the behavior of the backups.

Will changing the backend to rclone allow existing backups to be retrievable?

Comment 9 Michael Terry 2024-04-24 12:42:46 UTC
You can test the new rclone flow (should be invisible to you) by installing 46.beta from flathub’s beta channel. I’m on the go now or I’d link you to some docs for that.

Comment 10 Jonathan Billings 2024-04-24 13:26:44 UTC
Testing the org.gnome.DejaDup 46~beta on flathub-beta remote, and it does successfully connect to Google Drive, although it forces me to re-set up all the includes and excludes, and forces a new backup.  Does this mean that all prior backups are essentially lost?

Comment 11 Michael Terry 2024-04-24 13:34:05 UTC
The flatpak releases use their own gsettings store, so it makes sense that you’d need to res-set-up your includes and excludes etc.

A new backup… could be normal, if it’s been a minute since the last full backup.

You put the same Google Drive folder in as last time? If you go to the Restore tab, do you see dates for the old backups in the bottom right Dates dropdown? If you go to the Drive website, do you see a whole bunch of files still in those folders, with old timestamps in the filenames?

Comment 12 Jonathan Billings 2024-04-24 13:45:03 UTC
It looks like I was simply pointing it at a different directory.  I had set up backups on another laptop (with a different hostname) and just lifted and migrated all the data to the new laptop, so it was backing up to a directory with a different hostname.

Fixing the directory name to be the old hostname in the Preferences let me see old restores from the last backup.

Comment 13 Jonathan Billings 2024-04-24 15:25:43 UTC
I just tried an incremental backup using my existing backups, and got this error from the Flatpak before it gave up and said Backup Failed(redacting client ID):


ERROR 50 put
. Giving up after 5 attempts. BackendException: rclone returned rc = 1: 2024/04/24 10:46:15 DEBUG : Setting --log-level "DEBUG" from environment variable RCLONE_LOG_LEVEL="DEBUG"
. 2024/04/24 10:46:15 DEBUG : rclone: Version "v1.67.0-beta.7794.070cff8a6" starting with parameters ["rclone" "copyto" "/home/username/.var/app/org.gnome.DejaDup/cache/deja-dup/tmp/duplicity-maw4bcno-tempdir/mktemp-a9qip_hq-29" "dejadupdrive:username-thinkpadt14gen4.remote.csb/duplicity-inc.20240418T121450Z.to.20240424T140142Z.vol15.difftar.gpg"]
. 2024/04/24 10:46:15 DEBUG : Creating backend with remote "/home/username/.var/app/org.gnome.DejaDup/cache/deja-dup/tmp/duplicity-maw4bcno-tempdir/mktemp-a9qip_hq-29"
. 2024/04/24 10:46:15 DEBUG : Using config file from "/home/username/.config/rclone/rclone.conf"
. 2024/04/24 10:46:15 DEBUG : fs cache: adding new entry for parent of "/home/username/.var/app/org.gnome.DejaDup/cache/deja-dup/tmp/duplicity-maw4bcno-tempdir/mktemp-a9qip_hq-29", "/home/username/.var/app/org.gnome.DejaDup/cache/deja-dup/tmp/duplicity-maw4bcno-tempdir"
. 2024/04/24 10:46:15 DEBUG : Creating backend with remote "dejadupdrive:username-thinkpadt14gen4.remote.csb/"
. 2024/04/24 10:46:15 DEBUG : Setting type="drive" for "dejadupdrive" from environment variable RCLONE_CONFIG_DEJADUPDRIVE_TYPE
. 2024/04/24 10:46:15 DEBUG : Setting client_id="<redacted>.apps.googleusercontent.com" for "dejadupdrive" from environment variable RCLONE_CONFIG_DEJADUPDRIVE_CLIENT_ID
. 2024/04/24 10:46:15 DEBUG : Setting token="{\"access_token\":\"<redacted>\",\"expires_in\":3599,\"scope\":\"https://www.googleapis.com/auth/drive.file\",\"token_type\":\"Bearer\"}" for "dejadupdrive" from environment variable RCLONE_CONFIG_DEJADUPDRIVE_TOKEN
. 2024/04/24 10:46:15 DEBUG : Setting scope="drive.file" for "dejadupdrive" from environment variable RCLONE_CONFIG_DEJADUPDRIVE_SCOPE
. 2024/04/24 10:46:15 DEBUG : Setting use_trash="false" for "dejadupdrive" from environment variable RCLONE_CONFIG_DEJADUPDRIVE_USE_TRASH
. 2024/04/24 10:46:15 DEBUG : dejadupdrive: detected overridden config - adding "{Wf6RM}" suffix to name
. 2024/04/24 10:46:15 DEBUG : Setting scope="drive.file" for "dejadupdrive" from environment variable RCLONE_CONFIG_DEJADUPDRIVE_SCOPE
. 2024/04/24 10:46:15 DEBUG : Setting use_trash="false" for "dejadupdrive" from environment variable RCLONE_CONFIG_DEJADUPDRIVE_USE_TRASH
. 2024/04/24 10:46:15 DEBUG : Setting client_id="<redacted>.apps.googleusercontent.com" for "dejadupdrive" from environment variable RCLONE_CONFIG_DEJADUPDRIVE_CLIENT_ID
. 2024/04/24 10:46:15 DEBUG : Setting token="{\"access_token\":\"<redacted>\",\"expires_in\":3599,\"scope\":\"https://www.googleapis.com/auth/drive.file\",\"token_type\":\"Bearer\"}" for "dejadupdrive" from environment variable RCLONE_CONFIG_DEJADUPDRIVE_TOKEN
. 2024/04/24 10:46:16 Failed to create file system for destination "dejadupdrive:username-thinkpadt14gen4.remote.csb/": couldn't find root directory ID: googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
. More details:
. Reason: authError, Message: Invalid Credentials

Comment 14 Jonathan Billings 2024-04-25 17:26:21 UTC
I just did a completely fresh install from the latest Fedora Workstation Live image, installed the deja-dup RPM, set it up to back up to Google Drive, and it generates the same error.

Comment 15 Michael Terry 2024-04-26 02:31:42 UTC
OK to summarize, there are three problems mentioned so far in this ticket:

1) The odd packagekit screen.
2) The "ww" error message when trying to authenticate with Google.
3) When using the 46.beta flatpak, an oauth error.

I've done some initial investigation on 1 & 2. #3 is probably out of scope for this ticket, but I'll look at it later too.

For 1, what you are seeing is a not-completely-initialized packagekit install screen, ready to be switched to if needed - it's supposed to be immediately swapped out with the first progress screen. But if things are being slow, it might be visible. I've quick-fixed this in trunk (will be in 46.0): https://gitlab.gnome.org/World/deja-dup/-/commit/045c4078dc4dba26b7476fe70a5066b108553e88

But the tl;dr; is that it's an odd visual blip, but not a problem.

For 2, I've done enough investigation to know that it's an error that pops up when we are parsing the server's response via libsoup3. I don't know if the source of the issue is Google, libsoup3, vala, or Deja Dup. But basically, we see an oauth json response like:

{
  "access_token": "...",
  "expires_in": 3599,
  ...
}ww.googleapis.com

Notice that odd extra string at the end. Looks like an uninitialized array with some garbage in it, being not completely overwritten. As far as I can tell, we are creating a 5k response buffer, initialized to zeros with g_new0. And libsoup is filling the buffer with that bad json. I'll continue investigating this weekend.

This is the line that throws an error: https://gitlab.gnome.org/World/deja-dup/-/blob/main/libdeja/BackendOAuth.vala?ref_type=heads#L106

Comment 16 Michael Lausch 2024-04-26 14:04:30 UTC
The fix for #3:

The bug fix is pretty simple. in BackendOAuth.vala the send_message_raw function calls read_all_async() without passing in the bytes_read paramter, but assumes the dataarray will be properly null terminated, which it isn't. changing the code to:

size_t bytes_read;
yield response.read_all_async(data, GLib.Priority.DEFAULT, null, out bytes_read);
data[bytes_read] = 0;

fixes it. At least it runs on my machine.

Comment 17 Michael Terry 2024-04-26 22:03:30 UTC
For clarity, your bytes_read patch fixes issue #2 - not #3. That still needs investigation.

I've landed your patch (https://gitlab.gnome.org/World/deja-dup/-/merge_requests/94) in main.

I don't think that's the end of the story necessarily. Libsoup is writing content past where it says it did. Which may or may not be a bug. But the patch fixes the immediate issue on Deja Dup's side. Thank you!

For Fedora, I think it's a nice contained hotfix for this issue. I'd recommend applying it to the build.

Comment 18 Michael Lausch 2024-04-28 11:33:24 UTC
Well i think the patch is necessary. 
I debugged a little but more and with a HW watchpoint set just after the returned JSON content, it tripped during zlib deflate in the zlib library's inflate_fast_avx2() functions. The function knows about the size of the output buffer (the one passed down from deja-dup function) and therefore feels free to write more bytes than what the result of the expansion of the input is. This happens, because vector registers (AVX2) are used and it looks like only the first 8 bits are the result from the current usage of the register, the remaining bits of the vector register are leftovers from a previous usage. 
That's not a nice behaviour, but i don't see any violations here. The output buffer is big enough, the result of the deflate is not guaranteed to be a 0 terminated string, the output buffer is big enough and the function returns the number of valid bytes anyway. 

It could be argued that the higher level functions in lobsoup can 0 terminate the message as a C string int he output buffer. But what if the message fits exactly into the buffer? Then the caller must check the received byte count anyway (which the vala function does not do. that still should be fixed) and add a 0 if the buffer is big enough, but don't if it is an exact fit. 


Here's the stacktrace when the watchpoint triggers:
 0 in inflate_fast_avx2 of /usr/src/debug/zlib-ng-2.1.6-2.fc40.x86_64/inffast_tpl.h:166
 1 in inflate of /usr/src/debug/zlib-ng-2.1.6-2.fc40.x86_64/inflate.c:870
 2 in g_zlib_decompressor_convert of ../gio/gzlibdecompressor.c:333
 3 in soup_converter_wrapper_real_convert of ../libsoup/content-decoder/soup-converter-wrapper.c:248
 4 in soup_converter_wrapper_convert of ../libsoup/content-decoder/soup-converter-wrapper.c:331
 5 in read_internal of ../gio/gconverterinputstream.c:437
 6 in soup_filter_input_stream_read_nonblocking of ../libsoup/soup-filter-input-stream.c:167
 7 in soup_client_input_stream_read_nonblocking of ../libsoup/soup-client-input-stream.c:164
 8 in read_async_pollable of ../gio/ginputstream.c:1383
 9 in g_input_stream_real_read_async of ../gio/ginputstream.c:1429
10 in g_input_stream_read_async of ../gio/ginputstream.c:667
11 in read_all_callback of ../gio/ginputstream.c:750
12 in deja_dup_backend_oauth_send_message_raw_co of ../libdeja/BackendOAuth.vala:107
13 in deja_dup_backend_oauth_send_message_raw_ready of ../libdeja/BackendOAuth.vala:102
14 in g_task_return_now of ../gio/gtask.c:1361
15 in g_task_return of ../gio/gtask.c:1430
16 in g_task_return of ../gio/gtask.c:1387
17 in async_send_request_return_result of ../libsoup/soup-session.c:2675
18 in send_async_maybe_complete of ../libsoup/soup-session.c:2789
19 in run_until_read_done of ../libsoup/soup-session.c:2811
20 in g_task_return_now of ../gio/gtask.c:1361
21 in g_task_return of ../gio/gtask.c:1430
22 in g_task_return of ../gio/gtask.c:1387
23 in soup_http2_message_data_check_status of ../libsoup/http2/soup-client-message-io-http2.c:296
24 in g_list_foreach of ../glib/glist.c:1008
25 in io_read_ready of ../libsoup/http2/soup-client-message-io-http2.c:478
26 in g_main_dispatch of ../glib/gmain.c:3344
27 in g_main_context_dispatch_unlocked of ../glib/gmain.c:4152
28 in g_main_context_iterate_unlocked.isra.0 of ../glib/gmain.c:4217
29 in g_main_context_iteration of ../glib/gmain.c:4282
30 in g_application_run of ../gio/gapplication.c:2712
31 in _vala_main of ../app/main.vala:513
32 in main of ../app/main.vala:471

Comment 19 Michael Terry 2024-04-28 15:17:10 UTC
The violation from my perspective is that GInputStream.read_all's API contract is "bytes_read is set to the number of bytes read into buffer" - which feels like a promise not to add extra bytes into the buffer, and allow me to use the trick of providing an all-zeros buffer to a read function in an attempt to defensively avoid bugs like this. But ah well. I thought the zero-bytes buffer thing was common (and thus this new behavior might surprise other apps).

On another note, you're absolutely right about the edge case of an exact fit in the buffer - Deja Dup should be allocating a buffer one byte larger than it tells libsoup is available. That's for sure a bug in the application-side code.

But also, that code was never set up for such large reads. It allocates one large buffer and hopes the server doesn't deliver Hamlet along with the oauth token.

HOWEVER I realized that json-glib can read directly from a GInputStream. So I don't need to do manual buffer nonsense anyway. Don't know why I didn't just do that in the first place. Here's the upstream patch:
https://gitlab.gnome.org/World/deja-dup/-/commit/9de4ff971b3ee82de22fe0662dad899959072286

Fedora folks (Gwyn), here's a patch on top of 45.2 for you to use in the meantime:
```
diff --git a/libdeja/BackendOAuth.vala b/libdeja/BackendOAuth.vala
index 1c122cc2..54eb859d 100644
--- a/libdeja/BackendOAuth.vala
+++ b/libdeja/BackendOAuth.vala
@@ -102,9 +102,10 @@ public abstract class DejaDup.BackendOAuth : Backend
     var response = yield session.send_async(message, Priority.DEFAULT, null);
     if (message.status_code != Soup.Status.OK)
       return null;
-    var data = new uint8[5000]; // assume anything we read will be 5k or smaller
-    yield response.read_all_async(data, GLib.Priority.DEFAULT, null, null);
-    return new Json.Reader(Json.from_string((string)data));
+    var parser = new Json.Parser();
+    if (!yield parser.load_from_stream_async(response, null))
+      return null;
+    return new Json.Reader(parser.get_root());
   }
 
   internal async Json.Reader send_message(Soup.Message message) throws Error
```

Comment 20 Michael Terry 2024-04-28 15:30:30 UTC
To recap:
1) The odd packagekit screen. (FIXED upstream, is just a cosmetic issue)
2) The "ww" error message when trying to authenticate with Google. (FIXED upstream, patch above - I recommend this be applied to Fedora 40)
3) When using the 46.beta flatpak, an oauth error. (HALF-FIXED upstream)

So #1 and #2 are understood and have fixes. #3 is just an issue with the beta release which is not in Fedora anyway. But I'll comment on it briefly.

46.x is using Rclone for its cloud connections, and it has some interactions around refreshing OAuth tokens that needed some workarounds.

What seemed to be happening is that the OAuth token expired behind the scenes (maybe the server gave a short one, maybe the backup job was over an hour). Rclone was not able to refresh it by itself, and you ended up seeing that error.

1) We needed to make sure we fed Rclone a refresh token (duh, otherwise how could it refresh).
https://gitlab.gnome.org/World/deja-dup/-/commit/fc361b0747deaaeb758517743b691db5db3eb00e

2) We needed to make sure we calculated an "expiry" expiration date for Rclone, otherwise it would never attempt to refresh the token (it doesn't attempt to refresh when a token doesn't authenticate - only when it believes the token is already expired).
https://gitlab.gnome.org/World/deja-dup/-/commit/08b7577796ebdb50517ce228eadccba3ffa791fc

3) Rclone needs to fix a bug where it is impossible to not use an OAuth client secret during token refreshes, which Deja Dup doesn't use and causes the server to reject the refresh attempt.
https://github.com/rclone/rclone/pull/7809

#1 and #2 are fixed in Deja Dup trunk, #3 will hopefully be able to slip into the Rclone 1.67 release. We'll see.

But again, that does not affect Fedora yet.

Comment 21 Michael Lausch 2024-04-28 15:48:53 UTC
Sorry for nitpicking, but i think it's important to be very cautious about input data.


(In reply to Michael Terry from comment #19)
> The violation from my perspective is that GInputStream.read_all's API
> contract is "bytes_read is set to the number of bytes read into buffer" -
> which feels like a promise not to add extra bytes into the buffer, and allow

the keyword here is "feel" ;-)

it's not even a problem in GInputStream, but in libz. 
libz's deflate gets some input data, which may be string or some binary data, an output buffer and the lengths of both buffers. deflate() does its thing and uses AVX2 for speed optimization. Because the last bit of the deflated data does not fill a complete AVX2 register, garbage is written back to memory at the end. 
Maybe GInputStream could append a 0 byte at the end of the buffer to terminate a potential C string. But GINputStream does not know if the data is a C string or something else.


> HOWEVER I realized that json-glib can read directly from a GInputStream. So
> I don't need to do manual buffer nonsense anyway. Don't know why I didn't
> just do that in the first place. Here's the upstream patch:
> https://gitlab.gnome.org/World/deja-dup/-/commit/
> 9de4ff971b3ee82de22fe0662dad899959072286


the json-glib read-from-stream functions do exactly this. They terminate the buffer explicitly with a 0 byte at the last position
See https://gitlab.gnome.org/GNOME/json-glib/-/blame/main/json-glib/json-parser.c?ref_type=heads#L1504 for example.

Comment 22 Michael Terry 2024-04-28 16:01:57 UTC
> the keyword here is "feel" ;-)

Well I don’t mean to be so defensive, but I do think a reasonable person could read those words and come to the same conclusion that it is a promise not to write extra garbage bits to the app buffer.

A) GLib should clarify in the API either way, since two devs have read the same docs and come to two different conclusions about what the API guarantees.

B) If my read on the docs is what the GLib devs intended, Libsoup should use its own temporary buffer and then copy into the app buffer only the final bytes.

C) I maintain that this defensive programming practice of using zero buffers is a common tactic, but I don’t have stats for that. I continue to worry that this change in behavior could hit other apps that don’t take the extra step to re-zero out the final byte.

Comment 23 Michael Lausch 2024-04-28 16:15:15 UTC
well, it's hard to decide which component zlib, or GInputStream should do the 0 insertion, and what should it do if the buffer is exactly as long as the returned data, which is no problam at all if the data is for example a jpeg image. I think that would add some complexity.

A) agreed. they could state that it's not guaranteed that the buffer content is unchanged besides the returned data. maybe it's only me, because i did quite some low level hardware and network programming, that i never assume that a buffer is unmodified beyond the returned values. 

B) the code from json-glib I referenced, also does an explicity 0 termination,outside of GInputStream.

Comment 24 Michael Terry 2024-04-28 16:26:00 UTC
This will be my last comment on this line of thought, because this is all off topic at this point since we have a good fix.

But just for clarity:

I’m not proposing anybody adds an extra zero byte except the app. My expected flow is:
1) App create a buffer filled with zeros and with one extra byte .
2) App uses an API that does not add garbage bytes into the app buffer.
3) All done. The string is already zero-terminated.

Json-Glib does zero out the final byte, as it must - since it didn’t use a zero-initialized buffer, probably because it needs to care about performance.

But if an app is not performance sensitive, takes the time to allocate a buffer of all zeros beforehand, and has no reason to believe the read API is being rude, it can safely skip re-zeroing out the final byte.

Comment 25 Jonathan Billings 2024-04-29 14:34:12 UTC
We've tested an updated deja-dup package with the patch in https://gitlab.gnome.org/World/deja-dup/-/merge_requests/94/diffs and it looks like I can now grant access and backups have started and it moves on to the duplicity process.

Comment 26 Gwyn Ciesla 2024-04-29 16:19:00 UTC
(In reply to Michael Terry from comment #19)
> The violation from my perspective is that GInputStream.read_all's API
> contract is "bytes_read is set to the number of bytes read into buffer" -
> which feels like a promise not to add extra bytes into the buffer, and allow
> me to use the trick of providing an all-zeros buffer to a read function in
> an attempt to defensively avoid bugs like this. But ah well. I thought the
> zero-bytes buffer thing was common (and thus this new behavior might
> surprise other apps).
> 
> On another note, you're absolutely right about the edge case of an exact fit
> in the buffer - Deja Dup should be allocating a buffer one byte larger than
> it tells libsoup is available. That's for sure a bug in the application-side
> code.
> 
> But also, that code was never set up for such large reads. It allocates one
> large buffer and hopes the server doesn't deliver Hamlet along with the
> oauth token.
> 
> HOWEVER I realized that json-glib can read directly from a GInputStream. So
> I don't need to do manual buffer nonsense anyway. Don't know why I didn't
> just do that in the first place. Here's the upstream patch:
> https://gitlab.gnome.org/World/deja-dup/-/commit/
> 9de4ff971b3ee82de22fe0662dad899959072286
> 
> Fedora folks (Gwyn), here's a patch on top of 45.2 for you to use in the
> meantime:
> ```
> diff --git a/libdeja/BackendOAuth.vala b/libdeja/BackendOAuth.vala
> index 1c122cc2..54eb859d 100644
> --- a/libdeja/BackendOAuth.vala
> +++ b/libdeja/BackendOAuth.vala
> @@ -102,9 +102,10 @@ public abstract class DejaDup.BackendOAuth : Backend
>      var response = yield session.send_async(message, Priority.DEFAULT,
> null);
>      if (message.status_code != Soup.Status.OK)
>        return null;
> -    var data = new uint8[5000]; // assume anything we read will be 5k or
> smaller
> -    yield response.read_all_async(data, GLib.Priority.DEFAULT, null, null);
> -    return new Json.Reader(Json.from_string((string)data));
> +    var parser = new Json.Parser();
> +    if (!yield parser.load_from_stream_async(response, null))
> +      return null;
> +    return new Json.Reader(parser.get_root());
>    }
>  
>    internal async Json.Reader send_message(Soup.Message message) throws Error
> ```

Thank you, I'll get out an update with this ASAP.

Comment 27 Gwyn Ciesla 2024-04-29 16:36:00 UTC
...except that this seems to introduce some failures:

https://koji.fedoraproject.org/koji/taskinfo?taskID=117038313
https://koji.fedoraproject.org/koji/taskinfo?taskID=117038514

Comment 28 Gwyn Ciesla 2024-04-29 16:36:58 UTC
Ignore that comment, wrong bug.

Comment 29 Fedora Update System 2024-04-29 17:17:26 UTC
FEDORA-2024-2e6b7541ff (deja-dup-45.2-4.fc40) has been submitted as an update to Fedora 40.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-2e6b7541ff

Comment 30 Fedora Update System 2024-04-29 17:17:26 UTC
FEDORA-2024-20a6d0e127 (deja-dup-45.2-4.fc39) has been submitted as an update to Fedora 39.
https://bodhi.fedoraproject.org/updates/FEDORA-2024-20a6d0e127

Comment 31 Justin W. Flory (Red Hat) 2024-04-29 20:52:23 UTC
deja-dup-45.2-4.fc40 works for me.

Comment 32 Fedora Update System 2024-04-30 02:07:02 UTC
FEDORA-2024-2e6b7541ff has been pushed to the Fedora 40 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-2e6b7541ff`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-2e6b7541ff

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 33 Fedora Update System 2024-04-30 02:32:17 UTC
FEDORA-2024-20a6d0e127 has been pushed to the Fedora 39 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --refresh --advisory=FEDORA-2024-20a6d0e127`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2024-20a6d0e127

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 34 Fedora Update System 2024-05-01 04:14:33 UTC
FEDORA-2024-2e6b7541ff (deja-dup-45.2-4.fc40) has been pushed to the Fedora 40 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 35 Fedora Update System 2024-05-01 23:52:07 UTC
FEDORA-FLATPAK-2024-ab0c0c9f6d (deja-dup-flatpak-45.2-6) has been submitted as an update to Fedora 40 Flatpaks.
https://bodhi.fedoraproject.org/updates/FEDORA-FLATPAK-2024-ab0c0c9f6d

Comment 36 Fedora Update System 2024-05-02 02:40:43 UTC
FEDORA-FLATPAK-2024-ab0c0c9f6d has been pushed to the Fedora 40 Flatpaks testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-FLATPAK-2024-ab0c0c9f6d

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 37 Fedora Update System 2024-05-08 01:17:09 UTC
FEDORA-2024-20a6d0e127 (deja-dup-45.2-4.fc39) has been pushed to the Fedora 39 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 38 Fedora Update System 2024-05-10 02:21:02 UTC
FEDORA-FLATPAK-2024-ab0c0c9f6d (deja-dup-flatpak-45.2-6) has been pushed to the Fedora 40 Flatpaks stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.