Upload mods

[raymondbh]

A good interface to upload mods and put them the right place from the Solder web-site.

[Arzte]

Wow, I just realized that no one has ever posted that as an issue before, interesting.

Anyhow +1 for this.

One thing to consider is this, #291 (comment) (the second comment when the page loads)

[dstensnes]

+1 from me to

[raymondbh]

Sorry if this is a duplicate, I searched a while before posting...

[Arzte]

It isn't, I checked that

[ghost]

I give this a +1

[spannerman79]

+1
Although I assume people that will use this method would have their solder install in the same location as their repo to make checksuming quick.

Otherwise might have to use some sort of FTP system within solder itself to give a web based interface to FTP - which already exists in some form (net2FTP)

[Arzte]

Well, it can always be a config option that is disabled by default, that way people who have thier repo in a off machine repo can't complain about it not working.

[BentHaase]

Finally, would be awesome if there would be different upload possibilities, one for mods, another for config files (or files in general where you can determine what files should go where relative to the modpacks root). This would be a major usability improvement.

[MaegnusTheBlackMage]

I do hope to see this feature when creating large modpacks it can get very aggravating trying to upload and set them all up

[Indemnity83]

@GenPage you mentioned that "There are a few things that need to be re-coded/designed before we can start [this] feature. It is a massive feature that will be API breaking and essentially be a rewrite of the entire backend."

Can you elaborate, I would hate to put some work in on making this happen and have it turn out I (or anybody else who works on it for a future PR) head off in the wrong direction. Obviously the whole thing isn't crystal clear or it would likely be implemented already, but an outline or summary of how you see this working might be helpful.

[EntranceJew]

Seconding @Indemnity83's request, right now this issue is the only one on the milestone for 0.8 but if there were a handful of others it'd be easy to hop in and bring that around faster.

[GenPage]

@Indemnity83 @EntranceJew The outline or summary relies heavily on the Technic team since it affects their workflow/infrastructure on a deep level. I'm meeting with @sct and @CannibalVox soon to get this fleshed out on how they want this to move forward.

A brief idea is this:

Mod uploads coincide with us restructuring the API to handle "resources" instead of "mods". The resources will have "types" (Jar, config, file, etc) along with something similar to @Indemnity83's PR #438 for version targeting.

Zipping the files will go away completely as the launcher will know how to handle the files instead of just extracting directly into the root of the modpack folder. Handling of the files will be merged internally into Solder as a default, though support for external repositories will most likely stay in some form.

[ghost]

@GenPage I like that idea

[BentHaase]

@GenPage It has been almost a year since this idea / suggestion came up, are there any status updates on this one as this is most likely the one most feature many people are looking for? Please don't get me wrong, I don't want to stress, just a polite question!

[Indemnity83]

This is likely going to have to have a transition mechanism, to ease the transition from the .zip downloads, to resource downloads. As part of the #617 API RFC I'd like to propose a rough mechanism that can be used.

Given that:

• [Modpack] Builds have many Releases (aka Modversions)
• Assets belong to a Release
• Assets have a category (type is a reserved attribute in the JSON API, so best to avoid it)
• [Modpack] Builds have many Assets through Releases

Then the forward looking solution would provide an API endpoint /build/{build_id}?include=assets for the launcher to request a Modpack Build, including all its Assets. The API response would look something like the below:

{
	"data": {
		"type": "build",
		"id": "cac555fe-0325-489c-b3ed-1dc63dcacc2f",
		"attributes": {
			"version": "1.0.0",
			"published": false,
			"tags": {
				"minecraft": "1.7.10",
				"forge": "10.13.4.1614",
				"java": "1.6",
				"memory": 2048
			}
		},
		"relationships": {
			"assets": {
				"data": [{
					"type": "asset",
					"id": "8a4ad291-7bc5-40f9-8e11-6895eda7f7b6"
				}, {
					"type": "asset",
					"id": "344a6211-fafe-4e22-addc-fd2092f251fa"
				}]
			}
		}
	},
	"included": [{
		"type": "asset",
		"id": "8a4ad291-7bc5-40f9-8e11-6895eda7f7b6",
		"attributes": {
			"category": "mod",
			"url": "http://example.com/api/assets/8a4ad291-7bc5-40f9-8e11-6895eda7f7b6/somefile.jar"
		}
	}, {
		"type": "asset",
		"id": "344a6211-fafe-4e22-addc-fd2092f251fa",
		"attributes": {
			"category": "config",
			"url": "http://example.com/api/assets/344a6211-fafe-4e22-addc-fd2092f251fa/somefile.cfg"
		}
	}]
}

This would require that the API have a well defined list of asset categories, and changes to the file structure of Minecraft or Forge would break support until an update to both Solder and Launcher could be released. Worse, there is no way to have a category for all situations, some mods go rogue and keep their configs in non-standard locations.

So, I would propose instead of categorizing assets, we define their location attribute which would tell the launcher, just place the downloaded file in this location. There are security concerns here, but I think the launcher can provide the user protections by prompting for abnormal requests (.jar files not in the mods directory, for example).

The location based API response for an asset would look something like this:

 {
 	"type": "asset",
 	"id": "344a6211-fafe-4e22-addc-fd2092f251fa",
 	"location": {
 		"location": "mod-custom-config-directory",
 		"url": "http://example.com/api/assets/344a6211-fafe-4e22-addc-fd2092f251fa/somefile.cfg"
 	}
 }

Legacy Support

In order to provide legacy support, the application needs to be able to take the individual assets from a release (aka modversion) and zip them together as defined in the v0.7 documentation. This is achievable but the building of these .zip files will have to be done carefully to not impact performance of the application.

Ideally, the zip process would be handled by a worker asynchronously to the uploading of assets, but since thats a somewhat advanced configuration requirement then I suggest that a build of the zip is triggered when a request is made (will slow down the first launch of the modpack, but the .zip should be cached for future download requests).

In addition a manual build button, and CLI command could be added to let the user build the .zip on their own terms.

---

Updated givens to correct relationships; not critical but may have lead to confusion.

[spannerman79]

Ideally, the zip process would be handled by a worker asynchronously to the uploading of assets, but since thats a somewhat advanced configuration requirement then I suggest that a build of the zip is triggered when a request is made (will slow down the first launch of the modpack, but the .zip should be cached for future download requests).
This wouldn't work if the solder application and the actual mod files are hosted on separate systems.

For example I currently use a DreamHost VPS for solder and DreamSpeed CDN to host and deliver modpack files.

I am sure I am not the only one that does use a separate system for their own solder modpacks so I'm not the only one that this will affect. Failing to take that into consideration will make myself and other's move onto a system (like https://github.com/SKCraft/Launcher ) that does take into account different CDN systems.

[Indemnity83]

@spannerman79 that's good feedback; so what does your workflow look like right now to get mods in the system, and would in-app file uploads be a benefit to you at all?

[Indemnity83]

FYI, the Laravel 5.x file system does already support a lot of adapters, including some CDNs. This is the purpose of the 'disk' attribute in my proposal above. However, the DreamSpeed CDN is not on the list:

https://github.com/thephpleague/flysystem#adapters

[spannerman79]

so what does your workflow look like right now

Right now its uploading the changed mods/configs as per usual and adding them in as per usual. I'm seriously looking at moving away from solder and just going ahead with SKCrafts system as it allows me to use any system I want and not be limited.

would in-app file uploads be a benefit to you at all?

To anyone that has a separate system (solder in a VPS, modpack files in a CDN network) in place like I do it won't have any benefit what so ever to either myself or others that do what I do.

---

On a personal level I'm about a hair width away from actually moving my server specific modpacks and using SKCrafts system. The cost alone between the two would be cut by at least 60% perhaps even more including the cost for a code signing certificate. Remember those that have a server modpack don't make any money nor do we ask or want donations/contributions towards hosting it just for the way solder is.

Right now there is a serious dis junction between what solder can do now compared to what the current Technic Launcher & Technic Website displays.

Before thinking about a new API consider that the launcher team actually catch up to what solder can do now

[Indemnity83]

I have a working upload in my local development, including uploads to the DreamHost CDN system. I still need to clean it up before I can commit it but I wanted to share. This works for both in-web uploads, as well as implementing the API described above:

https://youtu.be/fctk0fJ2X8o

[BentHaase]

@Indemnity83 I really appreciate taking up the concept of assets! Looking forward to check it out :)

[ghost]

@Indemnity83 nicee!