Version 609

db/help my db is broke.txt

@@ -66,9 +66,16 @@ If you do not have a clean backup, you'll have to try recovering what data you c
 First of all, make a _new_ backup of the corrupted db, just in case something goes wrong with the recovery and we need to try again. You can just copy the client*.db files.
 
 
+*** sqlite3 executable ***
+
+We are going to talk to the database files directly, using SQL. I bundle 'sqlite3.exe' in the normal install, which is the Windows version of the SQLite terminal. If you are on Linux or macOS, you will want to look for your equivalent of the 'command-line tools' here: https://www.sqlite.org/download.html
+
+Or, look for a program like SQLiteStudio, which is a much richer UI that, if you dig around a bit, will let you run direct SQL commands on a particular db file.
+
+
 *** integrity checks ***
 
-To see if a file is not malformed, you need to run an integrity check. To do this, go to the db directory and look for the 'sqlite3' executable. Run it, and it'll bring up a terminal. Type or copy/paste this:
+To see if a file is not malformed, you need to run an integrity check. Go to the db directory and run the sqlite3 executable, and it'll bring up a terminal. Type or copy/paste this:
 
 .open client.db
 PRAGMA integrity_check;
@@ -109,6 +116,7 @@ The four files are separate, so if one is broken but the other three are fine, y
 
 There are three known repair operations. Try one and see if you can pass an integrity_check on the broken file. Clone works well for almost all simple problems, but the others have better success in different situations.
 
+
 ** clone **
 
 This instructs the database to copy itself to a new file. When it comes across damaged data, it will attempt to cleanly correct or nullify the broken links. The new file should be clean, but missing some data. It typically loses anywhere from 15-75%, which you can see by filesize. Losing 1-5% is normal because a clone includes an implicit vacuum.
@@ -143,6 +151,7 @@ Do not delete your original files for now. Just rename them to 'old' and move th
 
 You do not have to do an integrity_check after you clone. If you know your current hard drive is good, you can assume a clone works to create a 'clean' database.
 
+
 ** repair **
 
 This command tries to fix a database file in place. I do not recommend it as it works very very slowly. While it may be able to recover some rows a clones would lose, it cannot fix everything and may leave you with a database that is still malformed, so you'll want to run integrity_check again and do a clone if needed.
@@ -167,6 +176,7 @@ It is very important you have a backup copy of the broken file(s) here, as this
 
 If the recovery works, do the integrity_check. If that is also good, you are good to try to boot.
 
+
 ** manual export and import **
 
 This dumps the db to a (potentially huge) sql file and then reads it back into a new file. This seems to use slightly different access routines to clone and recover, but is extremely slow (think 25-100KB/s), so only try it in desperation. It comes thanks to a user, whose notes I have edited.

docs/changelog.md

@@ -7,6 +7,64 @@ title: Changelog
 !!! note
     This is the new changelog, only the most recent builds. For all versions, see the [old changelog](old_changelog.html).
 
+## [Version 609](https://github.com/hydrusnetwork/hydrus/releases/tag/v609)
+
+### macOS app now comes in a zip
+
+* the macos-13 runner that we recently moved to has been taking way too long to build the App DMG. under ideal conditions it will take ~6 minutes, but sometimes it will be 45+. this is an unfortunate and widespread issue without an excellent fix
+* thus, the macOS App is now distributed in a zip (that takes about 30 seconds to build and is the same size as the DMG lol)
+* unfortunately this means we can't have the Applications shortcut or the ReadMe; you'll be double-clicking the zip to extract the App, and then dragging it to your Applications yourself
+* I have updated the 'getting started - installing' guide with the 'allow unsigned apps' stuff from the ReadMe
+* I am increasingly of the mind that macOS users should consider running from source. the App is built for intel chipsets, so if you are on Silicon, you definitely should: https://hydrusnetwork.github.io/hydrus/running_from_source.html
+
+### misc
+
+* INFO: reducing the 'too many events' mpv error-handling hook worked well. if you saw a bunch of these in the past few weeks, these were almost certainly false-positives. the files were likely not broken, they were just taxing your computer for a bit and the error was firing accidentally. if you set up a special workflow to isolate or deal with these files, please undo it
+* the new 'group by namespace (user)' list widget under `options->sort/collect` now accepts `:` to represent 'all other namespaced tags' if you would like to put them somewhere specific
+* fixed a typo in the new duplicate files 'has duration' comparison strings. it said 'the other has duration, the other does not' when the current file had no duration
+* the score for duration now scales from 1 to 50 depending on how much longer a vid is than the other on millisecond longer = 1, twice as long = 50 (cap). this is all still just hardcoded stuff I'm playing around with, and I expect it to formalise to something completely user-customisable as the tools from duplicates auto-resolution firm up
+* added a little section to 'help my db is broke.txt' talking about the sqlite3 terminal and how to get it on non-Windows
+
+### colour sorting
+
+* after talking with some users, and learning that our blurhash provides the average colour of image thumbnails, I went bananas and added five new 'average colour' file sorts--
+* `lightness` - sorts light-to-dark in the Lab colourspace
+* `hue` - sorts the standard red-orange-yellow-green-cyan-blue-purple colourwheel/rainbow in the HSL colourspace
+* `balance - blue-yellow` - sorts blue-yellow chromaticity in the Lab colourspace
+* `balance - green-red` - sorts green-red chromaticity in the Lab colourspace
+* `chromatic magnitude` - sorts the combined chroma value, which is basically 'is it grey or colourful' in the Lab colourspace
+* this works on any file with a thumbnail!
+* they all work with system:limit, also!
+* and the Client API--new `file_sort_type` definitions are in the Client API reference
+* I am extremely pleased with how this all worked out. there are some weird results because it is average colour, rather, say, than a carefully selected 'primary colour', and I think it could do with some more work, to perhaps, for instance, push remarkably grey images to the bottom of Hue sorts, but this really works and it simply looks cool to do and then scroll through. it is a good base and proof of concept for my original plan to make colour histograms for files and distinguish between images that are muddy all over versus white and red. let me know where this succeeds and fails, and we'll see what improvements we can think of
+
+### janitor stuff
+
+* in the petition processing page, when you process a petition, either the normal way or via the mass approve/deny buttons, the 'currently highlighted petition' moves to _after_ the current one (or the next available one if you process multiple at once). previously this was resetting to the 0 position on every mass approve/deny. it should ripple down a bit more naturally now
+* when you set to apply multiple petitions at once, the pertinent list rows now instantly to show 'uploading...' status
+* I wrote a whack of test code that loads up and processes fake petitions for the petition page, so I can test these workflows much easier now
+* I fixed the 'modify account that uploaded this tag' for tags that include the '+' character. sorry for the long-time trouble--it was a stupid GET args parsing issue I missed in early code
+
+### boring code cleanup
+
+* broke up the 'ManagementPanels' monolith, refactoring it to a bunch of smaller files more under the 'Sidebar' name
+* renamed the ambiguous 'management type' to 'page type', and 'management panel' to 'sidebar', and 'management controller' to 'page manager' across the program
+* cleaned up some old static superclass calls in here to `super()`
+* brushed up the code generally around here, ditching some bad ideas and scope design
+* fixed a small instability trigger in the main file query routine
+* cleaned up some more inartful controller scope code in page and downloader UI
+* cleaned some ugly `QLineEdit` initialisations across the program
+* updated the server requirements.txt, which isn't used for anything but a reference, to use the newer OpenCV; I just forgot to update it a while ago
+
+### pyproject.toml
+
+* a user made a 'pyproject.toml' file. this is basically a cleverer requirements.txt that allows for better dependency options. I learned a bit about these files this week and I quite like them. pip can handle them too, so this would be one way to simplify my setup_venv scripts to a single call and allow users to set up simple custom automated situations
+* so, just for now, there's a pyproject.toml in the base directory. if you use `uv`, feel free to play around with it. I will manually keep it updated and version-synced to our requirements txts, and I wouldn't be surprised if I switch to it as the master soon. I'm going to do some other work and research and testing first though
+
+### duplicates auto-resolution
+
+* I did a little work on the daemon, but I didn't feel great about it tbh. I know roughly the various things it needs to do, but I'm not firm on what should be the correct orientation of everything. I'll keep at it, and I may do more UI so I have better feedback while I am doing final integration here
+
 ## [Version 608](https://github.com/hydrusnetwork/hydrus/releases/tag/v608)
 
 ### mpv
@@ -532,46 +590,3 @@ title: Changelog
 * SQLite is updated from 3.45.3 to 3.47.0
 * mpv is updated from 2023-08-20 to 2024-10-20
 * both dlls are mostly just bugfixes and performance improvements, but the mpv release is slightly special--in the 'future test' we ran a few weeks ago, users with unusual Windows, be that Windows Server, under-updated Windows 10, or Windows 10 on a VM, might see a grid of black bars over some webms. no one on Windows 11 or normal updated Windows 10 reported any problems. the new mpv does perform much better than the older, and I am told it fixes some gif bugs, so I do want to update, but I do so hesitantly. if many users on updated/normal Windows do run into trouble with this release, I expect to roll back again. in the meantime, I have updated the 'running from source' help to talk more about stable versions of mpv on older Windows. users who need to keep their OS under-updated are now recommended to run from source: https://hydrusnetwork.github.io/hydrus/running_from_source.html
-
-## [Version 599](https://github.com/hydrusnetwork/hydrus/releases/tag/v599)
-
-### misc
-
-* some users ended up getting a crazy wide duplicates page after the recent mix-up with the new auto-resolution multi-column list's ID. on update, the list will be reset to default widths, which should fix the rest of the duplicates page. let me know if you have any more trouble! (issue #1625)
-* the e621 downloader is fixed to find search results again (I updated the gallery parser to reflect their recent html changes). I understand they may be updating again soon, so let me know if anything breaks again (issue #1628)
-* I cleaned the media viewer hover window show/hide and size/position code a bit more and think I reduced more layout flicker, particularly the 'do another adjustment layout right after showing' issue the notes hover often does when super tall. I _might_ have also fixed the 'ok I guess I am going to sometimes initialise as a super wide guy in the middle of the viewer and when the user moves the mouse over my ghost I will flicker for one frame before moving where I belong' issue too
-* the timestamps in the media viewer top hover window and all the timestamp lines in the media right-click menu's top-row submenu now have a tooltip that has times with the reverse of your 'always show ISO times' settings. if you see '2022-11-20 14:39:52', it'll say '2 years ago' on the tooltip, and vice versa!
-* when you hit "open files in a new duplicates filter page", the file domain is now set explicitly to "all my files". if you know what you are doing and need to turn this off, there's a new checkbox under `options->duplicates`
-* new duplicates filter pages now also start on "all my files" rather than your default fallback file domain. "all my files" is the ideal default duplicates search context bros
-* if you have an OR predicate under construction in a file search tag autocomplete (use Shift+Enter when you enter stuff), hitting Escape on an empty text input now 'rewinds' it one predicate at a time before cancelling it entirely
-* animations that report they have exactly 100fps are now distrusted and their frames are counted manually. this 'count the frames manually' routine will now trigger generally in more cases and on larger files
-* all files with ~100fps will be metadata-rescanned on update
-
-### new vacuum tech
-
-* the vacuum in the client and server now uses `VACUUM INTO` SQLite tech. rather than writing a copy of the db to your temp dir and then to the WAL and then writing that 'commit' into the original file (which it seems is actually what happens--there's no atomic filename swap at the end), we now vacuum to a fresh file beside the original, no temp dir or WAL gubbins needed, and do a filename swap afterwards. this appears to work significantly faster than the old method, with the only caveat that there is a very brief dangerzone where neither file is named what we need, so if the hard drive disconnects or similar in that 2ms window, hydrus cannot automatically recover itself
-* all operations err on the side of failsafe, and I have added copious error handling code to navigate all the possible problems. if anything does go wrong, the user will be presented with a record of what happened, and in the case where hydrus could not fix itself, how they can fix it
-* the various 'do we have enough space to vacuum?' and 'how long will it take?' tests are adapted to the new rules. it doesn't use the temp dir any more, so if you have been struggling to find system drive space for client.mappings.db, this is no longer a problem
-* when the job is complete, the log message, which previously just said the time taken, now records the size change and the bytes per second, and this message is also now thrown into a popup. on my dev machine SSD, I have seen 170MB/s on database files that are in memory and ~30MB/s on a 4GB file not in memory. I guess we now asymptote to about 10MB/s on a superhuge file. I am interested in what users see in their different situations
-* If you are on an HDD and have a big db, you still have no hope, its over
-* thanks to the user who suggested this option
-
-### duplicates auto-resolution
-
-* _the duplicates auto-resolution system is moving forward. still lots to do, but I'm still feeling good about it all_
-* fleshed out the objects more--most stuff is now serialisable and has typedefs, getter/setters, and summary generation methods
-* connected most of the decision pipeline together, and we are basically ready to process our initial jpeg/png pixel pairs
-* did a hair more UI
-* my taglists now handle setEnabled calls properly, so the stub UI in the duplicates auto-resolution panel is now properly un-editable
-
-### boring cleanup
-
-* overhauled the way medias produce the various nice info strings on the right-click menu top-row submenu (and some other places). rather than a tangle of tuples, there's a couple simple classes being passed around that can do tooltip overrides and stuff. I also cleaned up the code around here generally
-* the main gui window status bar info when you only have one file selected is now much simpler. I previously piped the 'interesting' file info lines to it, but it too often ended up a spammy huge long line--not a good summary!
-* rewrote the 'always show iso time' solution from the old BaseMethod-to-Method replace trick to a simpler and saner global bool, as I recently did with some PIL/ICC settings
-* import folders now 'action' their original files immediately after import is done. previously they would do it in batches of ten, and if the import folder were interrupted by something like program shutdown, they'd have to wait for the next run to be cleared. I'm not totally sure, but I also think import folders set to 'ignore, not try again' on a large number of files may run a bit faster now
-* clarified the additional ways to import downloaders in the Lain import dialog. Lain's paste button now also accepts URI-aware file paths (i.e. if you select some files in your file explorer and tell your OS to 'copy') as the clipboard source
-* converted some duplicates processing code, and, relatedly, some delete-lock reporting stuff, from the being-overhauled `MediaSingleton` to `MediaResult`. this allowed me to clean up some wew code in duplicates auto-resolution, the duplicates filter, test code, and the Client API
-* updated the running from source help to talk about mpv/libmpv on Linux
-* fixed some bad panel/dialog calls into nicer Qt signals
-* added some typedefs to clear out about a hundred more PyUnresolvedReferences that PyCharm found. mostly custom widget calls

docs/developer_api.md

@@ -1802,8 +1802,13 @@ file\_sort\_type is by default _import time_. It is an integer according to the
 * 20 - hash hex (lexicographic/reverse lexicographic)
 * 21 - pixel hash hex (lexicographic/reverse lexicographic)
 * 22 - blurhash (lexicographic/reverse lexicographic)
+* 23 - average colour - lightness (darkest first/lightest first)
+* 24 - average colour - chromatic magnitude (greys first/colours first)
+* 25 - average colour - green/red axis (greens first/reds first)
+* 26 - average colour - blue/yellow axis (blues first/yellows first)
+* 27 - average colour - hue (rainbow - red first/rainbow - purple first)
 
-The pixel and blurhash sorts will put files without one of these (e.g. an mp3) at the end, regardless of asc/desc.
+The pixel, blurhash, and average colour sorts will put files without one of these (e.g. an mp3) at the end, regardless of asc/desc.
 
 Response:
 :   The full list of numerical file ids that match the search.