Does not obey the Simultaneous Downloads setting

[HeathenSkwerl]

In Settings-> Downloads you can set the "Simultaneous Downloads" settings, but the program will not obey it. When files start to transfer slowly, eventually the program wil spawn an additional download above and beyond what's already downloading. This additional download doesn't seem to stop the previous downloads from downloading (if they were going to finish in the first place, see issue #61 for when that isn't happening), but it seems to permanently increase the number of files downloading at one time. Eventually it seems as though there are too many active files downloading, and none of them will complete, resulting in the program crashing with no error message, or cancelling all remaing downloads without completion.

[jumbled00r]

it takes into account the cdn number on coomer.st (n1, n2, n3, n4, n5). when one locks up and stops responding I think it just moves onto the next video from a corresponding n{n} cdn. I went into downloader.py and changed max_workers to 1 and it basically just does the same thing but only uses one n{n}. so keeping it at default seems like the best idea for this program. it will eventually just come back and retry the partial downloads. its doing its best pulling from all available CDN servers available. coomer is kinda slow tbf

[HeathenSkwerl]

I have not seen it go back and retry the partial downloads ever. I see the part in the code that ios supposed to do that, but it doesn't look like it ever reaches that point for me. Instead one of three things always happens:

1. The download details window fills up completely on my monitor (2560x1600) even when it is expanded veritcally as big as it can go, and none of the visible downloads ever progress. It's possible something might still be happening, but without being able to see the details, it's hard to tell. This is because it never removes any file from the list unless it completes, so stalled files stay in the list indefinitely.
2. Program silently CTDs with nothing in the log or in the window I ran main.py from.
3. I get an "all downloads have been cancelled" message (not sure of the exact test, this has only happened twice).

[jumbled00r]

actually it works a little better imo with max_workers as 1 or 2 instead of default 5. this all comes down to the CDN rate limiting settings of coomer.st. they are very strict any more than 2 simultaneous downloads from the same ip and they basically block your connections. I just restarted the program and it starts continuing my partial downloads from where they left off so no I dont have that problem. CTDs have never happened to me and I launch from cmd (in wine mind you) using python main.py because if I use py main.py the image library doesnt load so I really hate python because of its nature especially on NT kernels.

also you need to restart the entire program after you save your new settings because the program won't automatically load them otherwise lol

[jumbled00r]

this setup seems more efficient bc coomer.st sends the data in a burst at first then leaves you to wait forever.
so this moves onto the next partial and continues the process more efficiently. I think 5 max_workers is optimal because coomer has 5 cdns (I think).

[jumbled00r]

Try my fork of CoomerDL on my profile. I fixed her code now it actually uses the retry_interval. Her code used to use fixed 1 second sleep for a range of errors instead of the retry interval. It would Square itself i *= 2 every retry and make it SOOOO SLOW to download. Its alot faster on my fork. also I fixed max retries so you can set it to 0 and and it won't retry immedietly it will just move on to the next file and return later when the server allows it ( technically already did this when it was 1 but it doesnt look logical so I fixed it )

Now you can easily set max retries to 0 and retry interval to 0 and it will just bang through the downloads asap.

[jumbled00r]

added a fix for INFINITE retries. no there should never be partials given up on. I think this way works best. just never stops retrying lol seriously check out my fork.

jesus fucking christ the threading in this app is broken and its hard to debug. it still leaves partials with max_retry as inf

[jumbled00r]

fixed concurrency in my version. your welcome. now its actually limited by max_workers, no partials if your max_retry is high enough

[HeathenSkwerl]

You aren't kidding that the threading is an issue, I've written multithreaded code in Python before and couldn't quite follow the logic in this one.

I'd be happy to check out your fork as soon as I'm done with the current profile I'm downloading, I have a bunch of partials and I'm using RonStrauss's PR for #97 to handle resuming TMP files. Or I could try your fork right away if you merged his code in with yours.

[HeathenSkwerl]

I also found another weird issue with the Download Progress window, it's set to be modal (using self.progress_window.grab_set() and self.progress_window.grab_release() which means I can't interact with the main window to cancel the downloads, I have to close the progress window to do it... but then even if you restart it never re-opens the progress window ever again. So basically if I need to cancel for any reason I have to shut down and restart the entire thing. Also if you minimize to desktop it seems like there's no way to get it to come back.

I turned off the modal setting for now but I'm sure that will probably break something--it had to be set like that for a reason right?

[jumbled00r]

I cant even get it to respect max_retries. Its still set to 3 no matter what... I give up for now. However, I atleast got multithreading to work in QUEUE not just blantanly starting new concurrent jobs on EVERY retry. I consider that a win. Her code is definetly overwriting max_retries with the default for some reason. Maybe its a clamp somewhere but I can not find it. I am tired I will let my downloads finish now and forget about it for a little bit. after all, I dont have a patreon to support my efforts to fix {Portuguese} python

[jumbled00r]

now mine is hard coded to use 1,000,000 retries no matter what. It turns out you just have to hard code max_retries. I don't know why this occurs but I seriously hate python it always uses the default value even if its set in the function call. If I wrote one of these apps my self it would only use c and a little json library thats it. could easily make this auto resume with wget or something

[HeathenSkwerl]

I did pull a copy of your branch, and started using it. Couple observations:

1. You're right, it is orders of magnitude faster. Files still sometimes stall out and abort, but it's less frequent.
2. Still getting the occasional CTD even with it. No idea what's causing that, no output anywhere to give any clues.
3. Files never resume for me. They either complete, or stall out with Download process failure for <url>: ('Connection broken: IncompleteRead(943718400 bytes read, 1499060181 more expected)', IncompleteRead(943718400 bytes read, 1499060181 more expected)). Retrying entire download in 2.0s. (Attempt 1/4). When this happens, it wipes the temp file and starts the entire download over from scratch. Disheartening when it's downloading a 2GB+ file and is 50MB from completing.
4. If a file hits its max retries, it gets left in the Download Details window at its last progress point and never cleared out.

No idea what the solution is for 2, but I suspect the issue with 3 is that an exception is being thrown in the initial download/write (with open(tmp_path, 'wb') as f:), which causes it to never reach the while loop where the resume code is contained. I think the solution would be to wrap that initial with in a try/except specifically looking for that IncompleteRead exception (since that's the only one I ever see that causes this issue), and just keep going if it hits that. I'll give it a shot and report back.

[jumbled00r]

my newest commit does not delete the tmp file unless it was cancelled by the user. so before her code actually deleted the entire tmp and didn't auto resume properly...

I think this is my last PR for CoomerDL before I maybe make my own simplified verison.

[HeathenSkwerl]

No worries. Maybe I'll fork my own copy, as long as you don't mind me starting with your fork? I fixed the issue with the resuming on the Incomplete Read amd actually simplified the main download loop (there's no need to open 'wb' wait for that to fail, and have a second chunk of code that opens 'ab' -- you can just open 'ab' to start as that should create the file if it doesn't exist).

I have a few other changes I want to make as well.