Sonarr/Radarr/Lidar Code base

[ericflix]

Any chance of moving over the Sonarr/Radarr/Lidar Code base? Pretty much all the *arr apps were forked off Sonarr and modified. There is even a fork in the works for non manga comics - Kapowarr.

This would remove the added work of the UI other than customizations and let you work on the backend.

[Thundernerd]

I would really like to yeah. I initially looked at it and saw the massive code base without any proper guide on how to use/modify it without spending a lot of time myself trying to figure things out. It might be something that is actually worth it even though it'll cost a lot of time, but I'll have to look into that.

[ericflix]

Seeing the Kapowarr is a comic arr app, it might be quicker to fork off them.

I do like your approach on pulling from all the manga/manhwa/manhua sites, makes it much easier.

[Thundernerd]

I'll have a look at that one!

I specifically wanted to support all the websites that serve content out there because a lot of people have their own favorite fan translations that they want to read from and not everything gets uploaded to newsgroups or torrents. It would be nice if it can support both of those down the line though, but it isn't the main focus for now!

ETA:

Kapowarr actually seems like a custom implementation in python, so they aren't forked from Sonarr or any of the others. I might just have to dive into the original one

[Thundernerd]

So I finally had a look at Sonarr and was able to make some modifications to get some things going. It isn't fully working yet but made some promising progress!

Added a bunch of random manga/manhua/manhwa to the overview

And was able to properly check one, though the season information is a bit borked

Figured I'd might as well fork Jackett to see if I can use that as a indexer gateway and was able to make a small bit of progress there as well

[ericflix]

That looks great! How are you handling scraping chapters from different sites? Is that what you are modifying Jackett for? Will it be able to scrape sites and even parse/ingest cbr/cbz files?

Since you are using the Sonarr code base I am hoping that you will support postgres too. I'm not a dev, but if you need testing done on a docker/kubernetes environment let me know.

[Thundernerd]

If Docker supports postgres then this will automatically as well, I'm not really making any breaking changes, or at least not planning to!

The way it currently works is:

1. You search for a manga, it uses mangaupdates.com to map those search results
2. One you've selected/added your manga it'll scan the entry on mangaupdates.com for the releases, these are then converted to "episodes"
3. Whenever Sonarr does its refresh, or whenever you manually look for a specific item it'll contact Jackett
4. Whenever there's a match for an episode/chapter Sonarr will ask Jackett for a torrent file, I am basically abusing torrents as regular data containers
5. These torrents contain links to all images belonging to a chapter, which Sonarr will internally download through regular http requests
6. (it doesn't do this yet) It'll then create a ComicInfo.xml file and together with that bundle it all up into a cbz
7. That cbz file can then be automatically processed and imported by Sonarr putting it in the right place!

[ericflix]

Sounds great. I'm looking forward to trying it out when you get it to the point you want to release it.

Postgres is supported on all the *arr apps now, Directions are on the Servarr site. I'll let you know how it works then there is a container available for it.

[alexandru0-dev]

@Thundernerd That sounds great, do u have a repo/branch where i can find the wip?

[Thundernerd]

These are not yet available, I'm currently looking into making builds for both the Sonarr and Jackett forks and then turn them into docker images! Once I've confirmed that that is working as intended I'll be sure to notify both of you (and anyone else that wants to)

[alexandru0-dev]

I don't really need to be in a functional stage, just curious and if I have time I can look into it

Btw for builds I can help you, I'm also going to create nix builds for myself.

I've quickly seen your jackett fork commits but doesn't your changes completely remove the old jackett behavior?

So I would be a fork for only indexing manga sources no?

[Thundernerd]

These are the repo+branch:
https://github.com/tnrd-org/Sonarr/tree/mangarr
https://github.com/tnrd-org/Jackett/tree/mangarr

I expect them both to be completely separate solutions, as in the functionality won't be PR'd into the original repositories as what I'm doing doesn't match with the intended use of the original software.

In fact, the Jackett one abuses torrents as a whole and simply uses them as data containers that will be read by my Sonarr fork and the "Fake Torrent" download client that I've built, which in turn is quite a bit different from how Sonarr usually works

[alexandru0-dev]

Thx, I was looking at your profile instead of the org

Also can I ask why did u choose jackett over prowlarr?

[Thundernerd]

I am not familiar with Prowlarr, so I’ll have to take a look at it! I used Jackett because I know of it and figured it might be a nice base to start from. I’ll definitely have a look at Prowlarr though

[alexandru0-dev]

Sure, I've looked a bit and integrations of other *rr services should be easier and this also could lead to possibly upstream it.

Tho idk how the fact that mangas are mainly DDL and you should manage different sources could impact upstream.

If I have some answers I will ping you.

Thx for the great work again

[Thundernerd]

@alexandru0-dev I'm planning on switching over to Prowlarr. Most of the leg work for the indexers has already been done in Jackett so I "just" have to port them over!

[alexandru0-dev]

@Thundernerd first of all thx again

Is there a way I can contact you directly
Discord or Matrix for example?

I've asked in the prowlarr's discord server and there are some problems to tackle about multiple sources I think or I didn't understand how your implementation would work

So I wanted to chat properly

[Thundernerd]

@alexandru0-dev You can hit me up on Discord. I'm also in the Servarr discord under the same name as here so feel free to send me a message request there!

[markschrik]

Hi!

Nice work :)
Once this is available with a sonarr base, I am really interested in running it. If you want, please keep me posted too!

[Thundernerd]

I have made quite some progress and have successfully been running it for a couple of days on my own server now!

I can post a docker compose along with some setup steps if y’all are interested?

It is far from finished though, especially one major issue is that there’s no one source of truth when it comes to available chapters. I’m currently writing an indexer that periodically scrapes all sources to and then links them all together so I can use this to figure out what chapters have been released.

Either way, still quite some work to do, but it is possible to run already!

[g-nogueira]

I've just "finished" writing the source for MangaDex, so it would be quite useful to have the "rr" version so that I can see If I can implement the same.

[Thundernerd]

I've just "finished" writing the source for MangaDex, so it would be quite useful to have the "rr" version so that I can see If I can implement the same.

This is the repo for my Prowlarr fork which is where the sources will go:
https://github.com/donderjoekel/Prowlarr

[ericflix]

Have you reached out to the Servarr Wiki? They added the Whisparr project/fork to their list of *arr apps. If you can get Mangarr included to the *arr ecosystem, that would open up a large group in the community of devs that can help get Mangarr to a mature state faster.

It is also a possibility to get your manga scrappers added into the prowlarr project if there is no major changes needed.

[alexandru0-dev]

@ericflix I did, but unfortunately mangarr works only with the jackett/prowlarr forks and are not merge-able with upstream (excluding indexers, the rest is shared)

We can find people that are willing to collaborate on the project there, surely.

The issue about incompatibility is mainly having different sources for DDLs (how different sources need a different way to be downloaded, aggregated etc)
As for now it's not compatible with upstream unfortunately and I hardly think it is ever going to be in the short run especially with the project not having that much of traction rn.
As of now existing indexers are replaced with manga sources.

@Thundernerd correct me if I'm wrong.

[Thundernerd]

Yup that’s correct.

My forks of Sonarr and Prowlarr are not huilt to support torrents and/or newsgroups. Unfortunately from my experience manga isn’t uploaded in agreed-upon standards so it would be very hard to use that.

Instead what my forks do is utilize the fair share of manga hosting websites and turns them into sources that can be used to download the images directly and pack it up into an archive for you to use in your favorite reader.

The core of both my Sonarr and Prowlarr fork have strayed so far from the original intended functionality that these will never be merged upstream.

[jordanfelle]

Will the fork of Sonarr work with stock Prowlarr still for when there are good torrent/newsgroup sources?

[Thundernerd]

Will the fork of Sonarr work with stock Prowlarr still for when there are good torrent/newsgroup sources?

Maybe? I'm honestly not too sue actually! I'm trying to (ab)use as much as I can from stock Sonarr and just build on top of it, so it could very well work. That said, there's a high chance that it will not work, or already isn't working. If the upload community can band together and come up with a workable standard then I'm 100% down to make it work though.

[g-nogueira]

Didn't think you would try making it work with torrent. That's awesome! Nonetheless, it seems you're focusing on the scrapping part, which I do believe is the better way of getting manga/comics. Thank you for the hard work!

I'm really looking forward to give a look at the code base but, first, I'm trying to make my fork of the sonarr_youtubedl work well so that I can try merging it (but I don't have high expectations, as there are already some stale PR's there).

[Thundernerd]

Wanted to drop a message here with a little update that unfortunately I haven't been able to spend proper time on the new codebase.

It is still regularly in the back of my mind but due to sickness, being out of the country for work, and just generally being busy with all kinds of things I haven't been able to dedicate time here.

Fret not though, it is all still very much planned, albeit just a bit slow.