Comm channel sending corrupt tensor values to debugger plugin.

[chihuahua]

• TensorBoard and TensorFlow version: 1.7
• OS Platform and version (e.g., Linux Ubuntu 16.04): Mac OS
• Python version (e.g. 2.7, 3.5): 2.7.10

While continuing through a debugger session, the interactive debugger plugin halted and entered an invalid state in which the UI indicates that the session was continuing, but tensor values would not update. This state stemmed from the server issuing a 500 response. Corrupt data from the communications channel caused the handler for the /comm route to raise a UnicodeDecodeError:

E0408 02:11:17.072490 Thread-4 _internal.py:87] Error on request:
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_chizeng/1b1399fef0aaaae96df4708880f141bb/external/org_pocoo_werkzeug/werkzeug/serving.py", line 193, in run_wsgi
    execute(self.server.app)
  File "/private/var/tmp/_bazel_chizeng/1b1399fef0aaaae96df4708880f141bb/external/org_pocoo_werkzeug/werkzeug/serving.py", line 181, in execute
    application_iter = app(environ, start_response)
  File "/Users/chizeng/Desktop/tensorboard/tensorboard/backend/application.py", line 272, in __call__
    return self.data_applications[clean_path](environ, start_response)
  File "/private/var/tmp/_bazel_chizeng/1b1399fef0aaaae96df4708880f141bb/external/org_pocoo_werkzeug/werkzeug/wrappers.py", line 291, in application
    return f(*args[:-2] + (request,))(*args[-2:])
  File "/Users/chizeng/Desktop/tensorboard/tensorboard/plugins/debugger/interactive_debugger_plugin.py", line 171, in _serve_comm
    return http_util.Respond(request, comm_data, 'application/json')
  File "/Users/chizeng/Desktop/tensorboard/tensorboard/backend/http_util.py", line 116, in Respond
    content = json.dumps(json_util.Cleanse(content, encoding),
  File "/Users/chizeng/Desktop/tensorboard/tensorboard/backend/json_util.py", line 72, in Cleanse
    return {Cleanse(k, encoding): Cleanse(v, encoding) for k, v in obj.items()}
  File "/Users/chizeng/Desktop/tensorboard/tensorboard/backend/json_util.py", line 72, in <dictcomp>
    return {Cleanse(k, encoding): Cleanse(v, encoding) for k, v in obj.items()}
  File "/Users/chizeng/Desktop/tensorboard/tensorboard/backend/json_util.py", line 72, in Cleanse
    return {Cleanse(k, encoding): Cleanse(v, encoding) for k, v in obj.items()}
  File "/Users/chizeng/Desktop/tensorboard/tensorboard/backend/json_util.py", line 72, in <dictcomp>
    return {Cleanse(k, encoding): Cleanse(v, encoding) for k, v in obj.items()}
  File "/Users/chizeng/Desktop/tensorboard/tensorboard/backend/json_util.py", line 66, in Cleanse
    return tf.compat.as_text(obj, encoding)
  File "/Library/Python/2.7/site-packages/tensorflow/python/util/compat.py", line 88, in as_text
    return bytes_or_text.decode(encoding)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf5 in position 38: invalid start byte

Indeed, the message from the communication channel seems to have a corrupt tensor value. \n$\n\x1dlayer1/weights/summaries/mean\x15S\xec99 looks nothing like a scalar value.

{
  'timestamp': 1523181167021665.0,
  'type': 'tensor',
  'data': {
    'shape': (),
    'values': '\n$\n\x1dlayer1/weights/summaries/mean\x15S\xec99',
    'dtype': 'object',
    'node_name': u'layer1/weights/summaries/mean',
    'debug_op': u'DebugIdentity',
    'output_slot': 0,
    'maybe_base_expanded_node_name': u'layer1/weights/summaries/mean/(mean)',
    'device_name': u'/job:localhost/replica:0/task:0/device:CPU:0'}
}

To repro, run this training script (that trains a simple MNIST model):
https://gist.github.com/chihuahua/a60ac1dc1e9152d27bce89d9ae092a3a
with tensorboard_debug_address set, ie

python bad_cross_entropy.py --tensorboard_debug_address 6042

and then start TensorBoard

tensorboard \
  --logdir /tmp/tensorflow/mnist/logs/mnist_with_summaries \
  --debugger_port 6042