[FIX] Image autoupload uploads images twice under certain conditions #4571

JuancaG05 · 2025-04-14T07:57:45Z

Related Issues

App: #3983

Add changelog files for the fixed issues in folder changelog/unreleased. More info here
Add feature to Release Notes in ReleaseNotesViewModel.kt creating a new ReleaseNote() with String resources (if required)

QA

(1) Camera folder not selected [FIX] Image autoupload uploads images twice under certain conditions #4571 (comment) [FIXED]
(2) Multiaccount [FIX] Image autoupload uploads images twice under certain conditions #4571 (comment) [FIXED]

joragua

LGTM! Good job @JuancaG05 💯 Let's see if it works.

NOTE: Don't forget to add calens file and release note if this PR is going to be merged into master branch.

jesmrec · 2025-04-15T08:54:57Z

(1) [FIXED]

Install the app from scratch
Add an account
Enable auto uploads for pictures
Choose the Camera folder (required) and submit

Current:

No folder is chosen, therefore no pictures are uploaded

Expected:

Folder chosen is set as subtitle and uploads work

Pixel 7
Xiaomi Redmi 13
Samsung A8
87e6bcec0

jesmrec · 2025-04-15T08:58:04Z

(2) [FIXED]

Add two accounts to the device
Enable Automatic Picture uploads
Select one of them as Account to upload pictures and an upload path
Then, click on Account to upload pictures and select the other account
Click on Picture upload path

Current:

The list of spaces is the one of the first account (the one selected in step 3.)

Expected:

The list of spaces is the one of the second account (the one selected in step 4.)

Pixel 7
Xiaomi Redmi 13
Samsung A8
87e6bcec0

JuancaG05 · 2025-04-15T15:57:48Z

@jesmrec (1) and (2) should be fixed now

jesmrec · 2025-04-16T08:47:45Z

(3)

After some tests, i got some duplications... this is the log if it helps somehow:

owncloud.2025-04-16_08.41.04.log

Samsung A8

JuancaG05 · 2025-04-21T06:41:49Z

@jesmrec regarding (3), those duplicates seen in logs refer to 0B files... let's see at least if worker flow is correct now (timestamp taken is correct, as seen in those logs) and maybe we can address the 0B problem in a separate issue

joragua · 2025-05-07T12:12:46Z

Some conclusions:

We added more verbosity to the AutomaticUploadsWorker to see the point at which it stops. After adding the logs, we noticed that those placed after the filter (where the files to be uploaded are chosen) don’t appear in the logs file. In my opinion, the problem is related to the way Android manages workers (not directly related to the code). After testing on a specific device (Google Pixel 7), we have seen that the system launches a new worker before the previous one has finished and, consequently, the timestamp is updated every time. That is the reason why the files are not uploaded correctly. In any case, if we keep the previous code (update the timestamp after the worker has finished) we will have duplicated files because there are workers running at the same time with the same files, so... 🫠
We changed the existing periodic work policy from KEEP to a new policy (UPDATE) but the behavior is the same.

NOTE: In my device (Samsung Galaxy A51), automatic uploads are working well and I don't have duplicated files.

jesmrec · 2025-05-07T12:21:35Z

Logs over all the tests done are available, just in case

apollo13 · 2025-05-15T19:43:04Z

@joragua Did the logging improvements already make it into a playstore release? I have the problems that nothing get's uploaded (well with a two days delay or so…) and I'd love to look at the logs with more information.

JuancaG05 · 2025-05-16T06:39:00Z

Hi @apollo13! No, unfortunately the verbosity added to logs is not in Play Store, and there are no plans to upload it since it's just some internal testing. But since you are interested on it, I'll provide an APK with the new verbosity.

You can download it here: https://infinite.owncloud.com/s/OeffTWmcFNXtmxi (password: $xP$r58W-,PB)

We would be thankful if you tell us if there is any new finding from your side! 🍻

apollo13 · 2025-05-16T09:58:35Z

@JuancaG05
owncloud_log_sanitized.txt
I have attached a logfile from today where I added comments when I took photos etc… What stands out is that there seems to be many AutomaticUploadsWorker operating somewhat in parallel and that they don't execute continuously given log lines like these:

I: 2025-05-16 10:34:21:079(AutomaticUploadsWorker.kt:136)Timestamp updated correctly. Current timestamp: Fri May 16 10:34:21 GMT+02:00 2025
I: 2025-05-16 10:40:59:975(AutomaticUploadsWorker.kt:269)CurrentTimestamp Fri May 16 10:34:21 GMT+02:00 2025

Looking at the code, it is not clear to me where the worker could sleep between those timestamps. My owncloud is empty since I am currently evaluating it, there is no way it takes 6 minutes to go from updating the timestamp to redisplaying it in getFilesReadyToUpload

I hope this helps, I am running a Pixel 6a with the latest Android. Happy to chat on matrix or whatever if you prefer realtime comms at some point.

apollo13 · 2025-05-16T11:32:00Z

And at 13:30 still no uploads, the last try was:

I: 2025-05-16 12:13:40:060(AutomaticUploadsWorker.kt:78)Starting AutomaticUploadsWorker with UUID c9f87e48-a622-450e-89d0-077cff8721b8
I: 2025-05-16 12:13:40:144(AutomaticUploadsWorker.kt:136)Timestamp updated correctly. Current timestamp: Fri May 16 12:13:40 GMT+02:00 2025
I: 2025-05-16 12:13:40:152(AutomaticUploadsWorker.kt:252)Source uri is: content://com.android.externalstorage.documents/tree/primary%3ADCIM%2FCamera
I: 2025-05-16 12:13:40:159(AutomaticUploadsWorker.kt:256)Document tree is: androidx.documentfile.provider.TreeDocumentFile@237360a
I: 2025-05-16 12:13:40:218(AvailableOfflinePeriodicWorker.kt:48)Available offline files that needs to be synced: 0
I: 2025-05-16 12:13:48:041(AutomaticUploadsWorker.kt:260)The search of local files has finished. Now, all files are going to be filtered

Given that the next logline should be "Last Sync", this code appears to hang in

android/owncloudApp/src/main/java/com/owncloud/android/workers/AutomaticUploadsWorker.kt

Line 262 in b20a9f3

val filteredList: List<DocumentFile> = arrayOfLocalFiles

or it got killed

apollo13 · 2025-05-16T12:10:45Z

@jesmrec As mentioned two comments up, it is a Pixel 6a running latest Android (15). Anything more specific you are looking for?

jesmrec · 2025-05-16T12:11:42Z

@apollo13 i have the same behaviour with Pixel 7, just to know whether the device has something to do. Because with other brands, it's working fine.

jesmrec · 2025-05-16T12:18:41Z

You can reach us in https://talk.owncloud.com/channel/mobile

apollo13 · 2025-05-16T13:39:17Z

Looking at some "debugging" options I think it would be a good idea to implement onStopped in the worker (https://developer.android.com/reference/kotlin/androidx/work/ListenableWorker#onStopped()) and call getStopReason there (https://developer.android.com/reference/kotlin/androidx/work/ListenableWorker#getStopReason()) to see when a worked stops.

I'll try to get some further information via https://developer.android.com/develop/background-work/background-tasks/testing/persistent/debug

JuancaG05 · 2025-05-19T09:08:44Z

Hi @apollo13! Thanks for your ideas!

Seems that since we're using CoroutineWorker, the method onStopped is final there (https://developer.android.com/reference/kotlin/androidx/work/CoroutineWorker#onStopped()), so we cannot override it. In any case, I added in the worker a new try - catch block to catch the CancellationException, which should be triggered in case the worker is cancelled. When this exception is caught, a new log is printed, but to see the real stop reason (method getStopReason), a device with at least API 31 (Android 12) should be used. This stop reason will be printed as a numerical constant, which needs to be checked together with the constants listed in https://developer.android.com/reference/android/app/job/JobParameters.html#constants_1.

Let's try this for the moment, and if we don't get further information, we can try the custom initialization mentioned in https://developer.android.com/develop/background-work/background-tasks/testing/persistent/debug#enable-logging, which will add much more verbosity to the current logs.

The new APK can be downloaded here: https://infinite.owncloud.com/s/cXkqstdShbaMevX (Password: Fja_'O188N,c)

Tell us if you have any new finding, and thanks a lot for your collaboration! 🤜 🤛

… avoid race conditions

…ad are selected

changelog/unreleased/4571

owncloudApp/src/main/res/values/strings.xml

owncloudApp/src/main/java/com/owncloud/android/workers/AutomaticUploadsWorker.kt

apollo13

Overall the changes make sense, left a few comments inline where I think there is room for further improvements.

One thing that still stands out though: We appear to no longer have duplicate runners because it looks as if they finish fast enough now. But it could still happen due to other factors that we cannot control. In that sense I'd prefer some defensive coding by utilizing a lock in doWork (or whatever makes sense for the android platform) and immediately exit the worker if the lock cannot be taken (ie another worker is still running) -- plus some logging on warn or so that one could see that there is a problem early on. Exiting a work run early wouldn't be that bad I guess since the next run would pick up the files anyways (assuming you exit before updating the timestamp).

I also think/feel that my original code using the content providers still performs better. I cannot tell you why and maybe there is some magic going on behind the scenes so it performs better or maybe it was just luck during my tests…

apollo13 · 2025-06-01T08:30:08Z

owncloudApp/src/main/java/com/owncloud/android/workers/AutomaticUploadsWorker.kt

@@ -248,10 +247,13 @@ class AutomaticUploadsWorker(
        val arrayOfLocalFiles = documentTree?.listFiles() ?: arrayOf()

        val filteredList: List<DocumentFile> = arrayOfLocalFiles
+            .asSequence()
+            .filter {
+                it.lastModified() in lastSyncTimestamp..<currentTimestamp &&


I wonder if it would make more sense to move the MimetypeIconUtil check before the lastModified check. The rationale being that the mime type lookup is hopefully just a hash table lookup whereas the modification check has to touch the filesystem. This would save quite a bit of I/O for cases where not many files match (let's say automatic video upload and the folder containing photos & videos consists of only 1% of videos, in such a case you'd have 99% useless I/O calls -- I actually think that such a case is probably not uncommon, even if one has a 50:50 ratio of pictures vs photos you'd check half the file unnecessarily).

That's right, but there are a lot of use cases... 🤔 As you said, you can have a folder in which the number of videos and photos is only a 1% and, with this filter, you will have 99% unnecessary calls. Also you can have a folder with a high number of photos and videos (99%) so, if we change the order there will be a lot of unnecessary calls anyway. In any case, @jesmrec is doing some tests so we will see what happens... But we will take into account your comments if something goes wrong. Thanks a lot for your opinion! 😄

apollo13 · 2025-06-01T08:31:28Z

owncloudApp/src/main/java/com/owncloud/android/workers/AutomaticUploadsWorker.kt

+            .filter {
+                it.lastModified() in lastSyncTimestamp..<currentTimestamp &&
+                        MimetypeIconUtil.getBestMimeTypeByFilename(it.name).startsWith(syncType.prefixForType)
+            }
            .sortedBy { it.lastModified() }


Any reason for sorting? Assuming lastModified is not cached on the file object this would again result in I/O (granted only a small amount since one usually doesn't have many files in 15 minutes). Since the updated timestamp is updated independent of the actual uploads, the order of files does not matter at all.

I'm not sure if it works as you comment here but it could be... 🤔 The reason for sorting is: upload the files in the same order that were taken, so we can have an "history" in the uploads section...

Ah okay, if one sees that as important then ordering makes sense.

apollo13 · 2025-06-01T08:32:21Z

owncloudApp/src/main/java/com/owncloud/android/workers/AutomaticUploadsWorker.kt

-            .filter { it.lastModified() >= lastSyncTimestamp }
-            .filter { it.lastModified() < currentTimestamp }
-            .filter { MimetypeIconUtil.getBestMimeTypeByFilename(it.name).startsWith(syncType.prefixForType) }
+            .toList()


Not knowing kotlin at all I am curious: What is the usecase for asSequence/toList here as opposed to the previous code that does without ?

The sequence is used to load and evaluate the files individually. The list loads all the files together and then the necessary checks are performed. The final .toList() is used to convert the sequence to a list (taking into account that the size of the list which contains the filtered files is smaller than the list of root folder)

jesmrec · 2025-06-03T14:36:48Z

I've done some tests with the current approach. Some conclusions:

Filter goes fine and is more performant than the previous solution
Duplications does not happen. I only reproduced the duplications stuff in one case: pictures taken in high velocity (more than 1 every second), and not in every test.
For a correct behaviour of the feature, mainly in background mode, it's noticeable that every device (with its customization layer) works in a different way. It would need settings adjustments to allow it. We should add this as a warning when the feature is opened.

Results were the same with @apollo13's approach. So, here it's a matter to decide which filtering algorithm fits better to our needs.

jesmrec · 2025-06-04T09:15:46Z

A couple of improvements to add, let me know your ideas:

As stated above:

For a correct behaviour of the feature, mainly in background mode, it's noticeable that every device (with its customization layer) works in a different way. It would need settings adjustments to allow it. We should add this as a warning when the feature is opened.

When the feature is enabled in Settings a warning dialog is displayed to clarify users how the feature works. I'd add an extra statement, like make sure in your device settings that %appname% is allowed to run in background for a better performance. We should check which Android version is the scratch point and maybe showing only for users with newer version than that one.

I realised the following edge scenario:

Enable the feature and set it up
Take 5 pictures -> stored in gallery
Before the worker runs, remove 2 of those recently taken pictures

(the number of pictures is just an example)

When the worker runs, this is the status:

There are 5 uploads, because the 2 removed pictures are moved to kind of trashbin but anyway accesible with a different name, that starts with .trashed (hidden file).

Should we prevent this kind of automatic uploads? i mean, if the file name starts with ., out of the filter.

That effect was reproducible in Pixel and Xiaomi devices, but not in Samsung.

apollo13 · 2025-06-04T09:19:59Z

Should we prevent this kind of automatic uploads? i mean, if the file name starts with ., out of the filter.

Seems sensible, can't decide whether to filter out all hidden files or only the ones starting with .trashed

jesmrec · 2025-06-04T09:23:46Z

Seems sensible, can't decide whether to filter out all hidden files or only the ones starting with .trashed

it'd be only preventing the hidden files coming from automatic uploads, not from other types of uploads... we can maybe assume that the "regular" pictures/videos taken to upload automatically will not be hidden files, but i did not handle every existing version of every customization layer of every vendor 😄

apollo13 · 2025-06-04T09:25:05Z

but i did not handle every existing version of every customization layer of every vendor 😄

yeah, hence my comment :) I would also assume that photos by default are probably not hidden.

jesmrec · 2025-06-04T09:36:41Z

ok, then refusing the ones stating by . , not by .trashed (other variants could rename to .trash or other names).

It's ok to get them out if they are hidden

joragua · 2025-06-04T09:37:46Z

Yeah, we can exclude all hidden files from the filter of the worker in this case 👍🏻

jesmrec · 2025-06-04T10:42:33Z

The fix works 👍

…atic uploads are enabled

jesmrec · 2025-06-05T10:21:16Z

Let's move this approach forward. For sure, if new improvements arise, we are open to include them after validation. Thanks everyone for your engagement, hoping the feature boosts with the news! 💯 🚀

apollo13 · 2025-06-05T10:39:21Z

Lovely thanks, is there any ETA on the 4.6 release? (Guess this has to affect quite a few users)

jesmrec · 2025-06-05T11:03:11Z

not yet, but it should not go beyond the first half of July.

JuancaG05 requested a review from joragua April 14, 2025 07:57

JuancaG05 self-assigned this Apr 14, 2025

JuancaG05 linked an issue Apr 14, 2025 that may be closed by this pull request

[BUG] Image autoupload uploads images twice under certain conditions #3983

Open

joragua approved these changes Apr 14, 2025

View reviewed changes

JuancaG05 force-pushed the fix/duplicated_autouploads branch from 87e6bce to c9139c8 Compare April 15, 2025 15:54

joragua force-pushed the fix/duplicated_autouploads branch 2 times, most recently from d565f45 to b20a9f3 Compare May 6, 2025 09:42

JuancaG05 force-pushed the fix/duplicated_autouploads branch from 9a510d8 to b20a9f3 Compare May 16, 2025 06:33

JuancaG05 added 2 commits May 29, 2025 11:24

fix: save timestamp for automatic uploads process at the beginning to…

a55e45a

… avoid race conditions

refactor: updated 47.json to add missing fields

db2d919

joragua force-pushed the fix/duplicated_autouploads branch 2 times, most recently from 69d4826 to a3a846c Compare May 29, 2025 09:32

refactor: modify the logic from the filter in which the files to uplo…

bb7b0dd

…ad are selected

joragua force-pushed the fix/duplicated_autouploads branch from 302b8fe to 3f82bf9 Compare May 29, 2025 10:14

joragua requested a review from jesmrec May 29, 2025 10:39

jesmrec requested changes May 29, 2025

View reviewed changes

changelog/unreleased/4571 Outdated Show resolved Hide resolved

owncloudApp/src/main/res/values/strings.xml Outdated Show resolved Hide resolved

owncloudApp/src/main/java/com/owncloud/android/workers/AutomaticUploadsWorker.kt Show resolved Hide resolved

joragua added 2 commits May 29, 2025 13:42

feat: add release note

c6b2ea3

chore: add calens file

5383f85

joragua force-pushed the fix/duplicated_autouploads branch from 10da175 to 5383f85 Compare May 29, 2025 11:47

docs: calens changelog updated

428adca

joragua requested a review from jesmrec May 29, 2025 11:49

jesmrec approved these changes May 29, 2025

View reviewed changes

apollo13 reviewed Jun 1, 2025

View reviewed changes

joragua added 2 commits June 4, 2025 14:24

fix: exclude all hidden files from the automatic uploads filter

d4f60b9

feat: add feedback about performance in the warning dialog when autom…

d5cd00d

…atic uploads are enabled

joragua force-pushed the fix/duplicated_autouploads branch from c9113ae to d5cd00d Compare June 4, 2025 12:26

jesmrec added the Approved by qa label Jun 5, 2025

joragua merged commit 4a65bfd into master Jun 5, 2025
8 checks passed

joragua deleted the fix/duplicated_autouploads branch June 5, 2025 10:34

[FIX] Image autoupload uploads images twice under certain conditions #4571

[FIX] Image autoupload uploads images twice under certain conditions #4571

Conversation

JuancaG05 commented Apr 14, 2025 • edited by joragua Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Related Issues

QA

Uh oh!

joragua left a comment

Choose a reason for hiding this comment

Uh oh!

jesmrec commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

(1) [FIXED]

Uh oh!

jesmrec commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

(2) [FIXED]

Uh oh!

JuancaG05 commented Apr 15, 2025

Uh oh!

jesmrec commented Apr 16, 2025

(3)

Uh oh!

JuancaG05 commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joragua commented May 7, 2025

Uh oh!

jesmrec commented May 7, 2025

Uh oh!

apollo13 commented May 15, 2025

Uh oh!

JuancaG05 commented May 16, 2025

Uh oh!

apollo13 commented May 16, 2025

Uh oh!

apollo13 commented May 16, 2025

Uh oh!

apollo13 commented May 16, 2025

Uh oh!

jesmrec commented May 16, 2025

Uh oh!

jesmrec commented May 16, 2025

Uh oh!

apollo13 commented May 16, 2025

Uh oh!

JuancaG05 commented May 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

apollo13 left a comment

Choose a reason for hiding this comment

Uh oh!

apollo13 Jun 1, 2025

Choose a reason for hiding this comment

Uh oh!

joragua Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

apollo13 Jun 1, 2025

Choose a reason for hiding this comment

Uh oh!

joragua Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

apollo13 Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

apollo13 Jun 1, 2025

Choose a reason for hiding this comment

Uh oh!

joragua Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

jesmrec commented Jun 3, 2025

Uh oh!

jesmrec commented Jun 4, 2025

JuancaG05 commented Apr 14, 2025 •

edited by joragua

Loading

jesmrec commented Apr 15, 2025 •

edited

Loading

jesmrec commented Apr 15, 2025 •

edited

Loading

JuancaG05 commented Apr 21, 2025 •

edited

Loading

joragua Jun 2, 2025 •

edited

Loading