Bad SHIFT-JIS to UTF-8 conversions

[norikawa]

Some song names (and possibly other details such as artist names, though I've mostly only seen song names for this) are incorrect when converted to UTF-8 from SHIFT-JIS. For example, "Entropic EnĤαncemEnt" is instead converted to "Entropic En鹹αncemEnt"

For my own WIP server in Crystal, I've gone through and found most of the incorrect characters and made a list of them to automatically replace them after the conversion step:

def fix_bad_utf8(text : String)
		replacements = {
			"齷" => "é",
			"鬮" => "¡",
			"齶" => "♡", 
			"ケロH" => "ケロ⑨",
			"曦" => "à",
			"曩" => "è",
			"龕" => "€",
			"壬" => "ê",
			"驩" => "Ø",
			"=墸Σ≡=｡ﾟ:*.:+｡.☆" => "=͟͟͞ Σ≡=｡ﾟ:*.:+｡.☆",
			"鹹" => "Ĥ",
			"闃" => "Ā",
			"饌" => "²",
			"煢" => "ø",
			"餮" => "Ƶ",
			"蔕" => "ῦ",
			"盥" => "⚙︎",
			"頽" => "ä",
			"隍" => "Ü",
			"雋" => "Ǜ",
			"鬻" => "♃",
			"鬥" => "Ã",
			"鬆" => "Ý",
			"趁" => "Ǣ",
			"驫" => "ā",
			"騫" => "á",
			"齲" => "♥",
			"骭" => "ü"
		}
		replacements.each_key do |key|
			text = text.gsub(key) {replacements[key]}
		end
		return text
end

I'm not familiar enough with TypeScript to know what the equivalent functions would be, but hopefully the list of replacements can be helpful at least!

[norikawa]

I've also gone through and confirmed that the incorrect characters only appear in this particular circumstance, so this method of replacing every instance of the given character/sequence will never replace good Japanese text with bad replacement text.

I haven't confirmed that this list covers all bad conversions though, I doubt I found all of them!

[22vv0]

There is a translation table in some of the js files, so I think we'd just need to add some new characters to the list? Will check.

[lucs100]

I'm working on a tool to import save data into this plugin and ran into the same encoding issue. The translation table in this issue and the plugin JS file mostly worked for me, I've merged them into one and added a couple cases in my code here.

Again there might be a few missing but as far as I can tell this fixes all my issues :)