UnicodeDecodeError

[bulabulabear]

url = 'http://www.gaj.chengdu.gov.cn/inqu_vote.asp?channel=%BE%AF%C3%F1%BB%A5%B6%AF&channelname=jmhd&col=%CD%F8%C9%CF%B5%F7%B2%E9&colname=wsdc&id=89'

when I use urlnorm.norm(url), there are some errors,like these:

File "D:\Python\lib\site-packages\urlnorm.py", line 150, in norm
normalized_tuple = norm_tuple(*url_tuple)
File "D:\Python\lib\site-packages\urlnorm.py", line 166, in norm_tuple
query = unquote_qs(query)
File "D:\Python\lib\site-packages\urlnorm.py", line 119, in unquote_qs
return unquote_safe(s, qs_unsafe_list)
File "D:\Python\lib\site-packages\urlnorm.py", line 144, in unquote_safe
return _unicode(o)
File "D:\Python\lib\site-packages\urlnorm.py", line 245, in _unicode
return value.decode("utf-8")
File "D:\Python\lib\encodings\utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xbe in position 8: invalid start byte

[trifle]

That's because the parameters are not in unicode:

uq = urllib.unquote(url)
chardet.detect(uq)

{'confidence': 0.99, 'encoding': 'GB2312'}
print(uq.decode( 'GB2312'))

http://www.gaj.chengdu.gov.cn/inqu_vote.asp?channel=警民互动&channelname=jmhd&col=网上调查&colname=wsdc&id=89

urlnorm only handles utf-8 urls. There is no foolproof way of detecting the encoding in quoted characters, so it's probably best to leave the guessing part to the user (ping @jehiah)?