Merge pull request #106 from MartinHjelmare/0.11

0.11

.coveragerc

@@ -0,0 +1,8 @@
+[report]
+exclude_lines =
+    # need to re-specify the default pragma
+    pragma: no cover
+
+    def __repr__
+    raise AssertionError
+    raise NotImplementedError

CHANGELOG.md

@@ -1,6 +1,19 @@
 # Change Log
 
-## [0.10](https://github.com/theolind/pymysensors/tree/0.10) (2017-05-07)
+## [0.11.0](https://github.com/theolind/pymysensors/tree/0.11.0) (2017-08-21)
+[Full Changelog](https://github.com/theolind/pymysensors/compare/0.10...0.11.0)
+
+**Merged pull requests:**
+
+- Add debug timer logging if handle queue is slow [\#105](https://github.com/theolind/pymysensors/pull/105) ([MartinHjelmare](https://github.com/MartinHjelmare))
+- Update gen\_changelog and release procedure [\#104](https://github.com/theolind/pymysensors/pull/104) ([MartinHjelmare](https://github.com/MartinHjelmare))
+- Update type schema and add message tests [\#103](https://github.com/theolind/pymysensors/pull/103) ([MartinHjelmare](https://github.com/MartinHjelmare))
+- Add validation of message and child values [\#102](https://github.com/theolind/pymysensors/pull/102) [[breaking change](https://github.com/theolind/pymysensors/labels/breaking%20change)] ([MartinHjelmare](https://github.com/MartinHjelmare))
+- Upgrade test requirements [\#101](https://github.com/theolind/pymysensors/pull/101) ([MartinHjelmare](https://github.com/MartinHjelmare))
+- Update const for version 1.5 and 2.0 [\#100](https://github.com/theolind/pymysensors/pull/100) ([MartinHjelmare](https://github.com/MartinHjelmare))
+- Fix subscription to topics with nested prefix [\#99](https://github.com/theolind/pymysensors/pull/99) ([MartinHjelmare](https://github.com/MartinHjelmare))
+
+## [0.10](https://github.com/theolind/pymysensors/tree/0.10) (2017-05-06)
 [Full Changelog](https://github.com/theolind/pymysensors/compare/0.9.1...0.10)
 
 **Closed issues:**
@@ -9,6 +22,7 @@
 
 **Merged pull requests:**
 
+- 0.10 [\#98](https://github.com/theolind/pymysensors/pull/98) ([MartinHjelmare](https://github.com/MartinHjelmare))
 - Add release instructions and update setup [\#97](https://github.com/theolind/pymysensors/pull/97) ([MartinHjelmare](https://github.com/MartinHjelmare))
 - Add manifest and update setup files [\#96](https://github.com/theolind/pymysensors/pull/96) ([MartinHjelmare](https://github.com/MartinHjelmare))
 - Add changelog [\#95](https://github.com/theolind/pymysensors/pull/95) ([MartinHjelmare](https://github.com/MartinHjelmare))
@@ -40,7 +54,7 @@
 - Move gateways into separate modules [\#84](https://github.com/theolind/pymysensors/pull/84) ([MartinHjelmare](https://github.com/MartinHjelmare))
 - Return local time instead of UTC time from Controller [\#81](https://github.com/theolind/pymysensors/pull/81) ([proddy](https://github.com/proddy))
 - Add discover [\#79](https://github.com/theolind/pymysensors/pull/79) ([MartinHjelmare](https://github.com/MartinHjelmare))
-- Event callback extensions [\#78](https://github.com/theolind/pymysensors/pull/78) ([steve-bate](https://github.com/steve-bate))
+- Event callback extensions [\#78](https://github.com/theolind/pymysensors/pull/78) [[breaking change](https://github.com/theolind/pymysensors/labels/breaking%20change)] ([steve-bate](https://github.com/steve-bate))
 - tcp\_check to reconnect in case of connection lost [\#67](https://github.com/theolind/pymysensors/pull/67) ([afeno](https://github.com/afeno))
 
 ## [0.8](https://github.com/theolind/pymysensors/tree/0.8) (2016-10-19)
@@ -96,7 +110,7 @@
 - Fix and add missing set,req and internal api types [\#52](https://github.com/theolind/pymysensors/pull/52) ([MartinHjelmare](https://github.com/MartinHjelmare))
 - Update pyserial to version 3.1.1 [\#50](https://github.com/theolind/pymysensors/pull/50) ([MartinHjelmare](https://github.com/MartinHjelmare))
 - Add MQTT client Gateway layer [\#49](https://github.com/theolind/pymysensors/pull/49) ([MartinHjelmare](https://github.com/MartinHjelmare))
-- Update handle\_req to return zero values [\#48](https://github.com/theolind/pymysensors/pull/48) ([myallh](https://github.com/myallh))
+- Update handle\_req to return zero values [\#48](https://github.com/theolind/pymysensors/pull/48) ([mch3000](https://github.com/mch3000))
 - Handle heartbeat message [\#46](https://github.com/theolind/pymysensors/pull/46) ([MartinHjelmare](https://github.com/MartinHjelmare))
 - Add class for testing protocol\_version 1.5 [\#45](https://github.com/theolind/pymysensors/pull/45) ([MartinHjelmare](https://github.com/MartinHjelmare))
 - Add const for mysensors 2.0 [\#44](https://github.com/theolind/pymysensors/pull/44) ([MartinHjelmare](https://github.com/MartinHjelmare))
@@ -175,4 +189,4 @@
 
 
 
-\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)*
+\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)*

RELEASE.md

@@ -21,9 +21,8 @@
 
 - Create a release branch from dev.
 - Merge master into the release branch to make the PR mergeable.
+- Update version in `mysensors/version.py` to the new version number, eg `'0.2.0'`.
 - Update `CHANGELOG.md` by running `scripts/gen_changelog`.
-- Replace the unreleased header in the changelog with the new release number.
-- Update version in `setup.py`.
 - Commit and push the release branch.
 - Create a pull request from release branch to master with the upcoming release number as the title. Put the changes for the new release from the updated changelog as the PR message.
 - Merge the pull request into master, do not squash.
@@ -38,3 +37,8 @@
   ```
 - Stage release: `twine upload -r test dist/*`
 - Release: `twine upload -r pypi dist/*`
+- Fetch and checkout the master branch.
+- Fetch and checkout the develop branch.
+- Merge master into develop.
+- Update version in `mysensors/version.py` to the new develop version number, eg `'0.3.0.dev0'`
+- Commit the version bump and push to develop branch.

mysensors/__init__.py

@@ -7,13 +7,39 @@
 import threading
 import time
 from collections import deque
+# pylint: disable=import-error, no-name-in-module
+from distutils.version import LooseVersion as parse_ver
 from importlib import import_module
 from queue import Queue
+from timeit import default_timer as timer
+
+import voluptuous as vol
 
 from .ota import OTAFirmware
+from .version import __version__  # noqa: F401
 
 _LOGGER = logging.getLogger(__name__)
 
+BROADCAST_ID = 255
+LOADED_CONST = {}
+SYSTEM_CHILD_ID = 255
+
+
+def get_const(protocol_version):
+    """Return the const module for the protocol_version."""
+    version = str(protocol_version)
+    if parse_ver('1.5') <= parse_ver(version) < parse_ver('2.0'):
+        path = 'mysensors.const_15'
+    elif parse_ver(version) >= parse_ver('2.0'):
+        path = 'mysensors.const_20'
+    else:
+        path = 'mysensors.const_14'
+    if path in LOADED_CONST:
+        return LOADED_CONST[path]
+    const = import_module(path)
+    LOADED_CONST[path] = const  # Cache the module
+    return const
+
 
 class Gateway(object):
     """Base implementation for a MySensors Gateway."""
@@ -31,21 +57,19 @@ def __init__(self, event_callback=None, persistence=False,
         self.persistence = persistence  # if true - save sensors to disk
         self.persistence_file = persistence_file  # path to persistence file
         self.persistence_bak = '{}.bak'.format(self.persistence_file)
-        self.protocol_version = float(protocol_version)
-        if 1.5 <= self.protocol_version < 2.0:
-            _const = import_module('mysensors.const_15')
-        elif self.protocol_version >= 2.0:
-            _const = import_module('mysensors.const_20')
-        else:
-            _const = import_module('mysensors.const_14')
-        self.const = _const
+        self.protocol_version = protocol_version
+        self.const = get_const(self.protocol_version)
         self.ota = OTAFirmware(self.sensors, self.const)
         if persistence:
             self._safe_load_sensors()
 
+    def __repr__(self):
+        """Return the representation."""
+        return self.__class__.__name__
+
     def _handle_presentation(self, msg):
         """Process a presentation message."""
-        if msg.child_id == 255:
+        if msg.child_id == SYSTEM_CHILD_ID:
             # this is a presentation of the sensor platform
             sensorid = self.add_sensor(msg.node_id)
             if sensorid is None:
@@ -82,8 +106,8 @@ def _handle_set(self, msg):
         # Check if reboot is true
         if self.sensors[msg.node_id].reboot:
             return msg.modify(
-                child_id=255, type=self.const.MessageType.internal, ack=0,
-                sub_type=self.const.Internal.I_REBOOT, payload='')
+                child_id=SYSTEM_CHILD_ID, type=self.const.MessageType.internal,
+                ack=0, sub_type=self.const.Internal.I_REBOOT, payload='')
 
     def _handle_req(self, msg):
         """Process a req message.
@@ -166,7 +190,9 @@ def logic(self, data):
         ret = None
         try:
             msg = Message(data, self)
-        except ValueError:
+            msg.validate(self.protocol_version)
+        except (ValueError, vol.Invalid) as exc:
+            _LOGGER.warning('Not a valid message: %s', exc)
             return
 
         if msg.type == self.const.MessageType.presentation:
@@ -205,8 +231,8 @@ def _save_json(self, filename):
     def _load_json(self, filename):
         """Load sensors from json file."""
         with open(filename, 'r') as file_handle:
-            self.sensors.update(
-                json.load(file_handle, cls=MySensorsJSONDecoder))
+            self.sensors.update(json.load(
+                file_handle, cls=MySensorsJSONDecoder))
 
     def _save_sensors(self):
         """Save sensors to file."""
@@ -292,7 +318,7 @@ def _get_next_id(self):
             next_id = max(self.sensors.keys()) + 1
         else:
             next_id = 1
-        if next_id <= 254:
+        if next_id <= self.const.MAX_NODE_ID:
             return next_id
 
     def add_sensor(self, sensorid=None):
@@ -312,11 +338,11 @@ def is_sensor(self, sensorid, child_id=None):
             ret = child_id in self.sensors[sensorid].children
             if not ret:
                 _LOGGER.warning('Child %s is unknown', child_id)
-        if not ret and self.protocol_version >= 2.0:
+        if not ret and parse_ver(self.protocol_version) >= parse_ver('2.0'):
             _LOGGER.info('Requesting new presentation for node %s',
                          sensorid)
             msg = Message(gateway=self).modify(
-                node_id=sensorid, child_id=255,
+                node_id=sensorid, child_id=SYSTEM_CHILD_ID,
                 type=self.const.MessageType.internal,
                 sub_type=self.const.Internal.I_PRESENTATION)
             if self._route_message(msg):
@@ -342,9 +368,15 @@ def handle_queue(self, queue=None):
         if queue is None:
             queue = self.queue
         if not queue.empty():
+            start = timer()
             func, args, kwargs = queue.get()
             reply = func(*args, **kwargs)
             queue.task_done()
+            end = timer()
+            if end - start > 0.1:
+                _LOGGER.debug(
+                    'Handle queue with call %s(%s, %s) took %.3f seconds',
+                    func, args, kwargs, end - start)
             return reply
 
     def fill_queue(self, func, args=None, kwargs=None, queue=None):
@@ -400,7 +432,7 @@ def __init__(self, sensor_id):
         self.sketch_name = None
         self.sketch_version = None
         self._battery_level = 0
-        self.protocol_version = None
+        self.protocol_version = '1.4'
         self.new_state = {}
         self.queue = deque()
         self.reboot = False
@@ -415,6 +447,11 @@ def __setstate__(self, state):
         self.queue = deque()
         self.reboot = False
 
+    def __repr__(self):
+        """Return the representation."""
+        return '<Sensor sensor_id={}, children: {}>'.format(
+            self.sensor_id, self.children)
+
     @property
     def battery_level(self):
         """Return battery level."""
@@ -443,7 +480,7 @@ def set_child_value(self, child_id, value_type, value, **kwargs):
             return
         msg_type = kwargs.get('msg_type', 1)
         ack = kwargs.get('ack', 0)
-        msg = Message(gateway=self).modify(
+        msg = Message().modify(
             node_id=self.sensor_id, child_id=child_id, type=msg_type, ack=ack,
             sub_type=value_type, payload=value)
         msg_string = msg.encode()
@@ -454,11 +491,13 @@ def set_child_value(self, child_id, value_type, value, **kwargs):
                 self.sensor_id, child_id, msg_type, ack, value_type, value)
             return
         try:
-            msg = Message(msg_string)  # Validate child values
-        except (ValueError, AttributeError) as exception:
-            _LOGGER.error('Error validating child values: %s', exception)
+            msg = Message(msg_string)
+            msg.validate(self.protocol_version)
+        except (ValueError, AttributeError, vol.Invalid) as exc:
+            _LOGGER.error('Not a valid message: %s', exc)
             return
-        children[msg.child_id].values[msg.sub_type] = msg.payload
+        child = children[msg.child_id]
+        child.values[msg.sub_type] = msg.payload
         return msg_string
 
 
@@ -485,14 +524,23 @@ def __setstate__(self, state):
 
     def __repr__(self):
         """Return the representation."""
-        return self.__str__()
-
-    def __str__(self):
-        """Return the string representation."""
-        ret = ('child_id={0!s}, child_type={1!s}, description={2!s}, '
-               'values = {3!s}')
+        ret = ('<ChildSensor child_id={0!s}, child_type={1!s}, '
+               'description={2!s}, values: {3!s}>')
         return ret.format(self.id, self.type, self.description, self.values)
 
+    def get_schema(self, protocol_version):
+        """Return the child schema for the correct const version."""
+        const = get_const(protocol_version)
+        return vol.Schema({
+            typ.value: const.VALID_SETREQ[typ]
+            for typ in const.VALID_TYPES[self.type]})
+
+    def validate(self, protocol_version, values=None):
+        """Validate child value types and values against protocol_version."""
+        if values is None:
+            values = self.values
+        return self.get_schema(protocol_version)(values)
+
 
 class Message(object):
     """Represent a message from the gateway."""
@@ -509,6 +557,12 @@ def __init__(self, data=None, gateway=None):
         if data is not None:
             self.decode(data)
 
+    def __repr__(self):
+        """Return the representation."""
+        return '<Message data="{};{};{};{};{};{}">'.format(
+            self.node_id, self.child_id, self.type, self.ack, self.sub_type,
+            self.payload)
+
     def copy(self, **kwargs):
         """Copy a message, optionally replace attributes with kwargs."""
         msg = Message(self.encode(), self.gateway)
@@ -551,6 +605,51 @@ def encode(self, delimiter=';'):
         except ValueError:
             _LOGGER.error('Error encoding message to gateway')
 
+    def validate(self, protocol_version):
+        """Validate message."""
+        const = get_const(protocol_version)
+        valid_node_ids = vol.All(vol.Coerce(int), vol.Range(
+            min=0, max=BROADCAST_ID, msg='Not valid node_id: {}'.format(
+                self.node_id)))
+        valid_child_ids = vol.All(vol.Coerce(int), vol.Range(
+            min=0, max=SYSTEM_CHILD_ID, msg='Not valid child_id: {}'.format(
+                self.child_id)))
+        if self.type in (const.MessageType.internal, const.MessageType.stream):
+            valid_child_ids = vol.All(vol.Coerce(int), vol.In(
+                [SYSTEM_CHILD_ID],
+                msg='When message type is {}, child_id must be {}'.format(
+                    self.type, SYSTEM_CHILD_ID)))
+        if (self.type == const.MessageType.internal and
+                self.sub_type in [
+                    const.Internal.I_ID_REQUEST,
+                    const.Internal.I_ID_RESPONSE]):
+            valid_child_ids = vol.Coerce(int)
+        valid_types = vol.All(vol.Coerce(int), vol.In(
+            [member.value for member in const.VALID_MESSAGE_TYPES],
+            msg='Not valid message type: {}'.format(self.type)))
+        if self.child_id == SYSTEM_CHILD_ID:
+            valid_types = vol.All(vol.Coerce(int), vol.In(
+                [const.MessageType.presentation.value,
+                 const.MessageType.internal.value,
+                 const.MessageType.stream.value],
+                msg=(
+                    'When child_id is {}, {} is not a valid '
+                    'message type'.format(SYSTEM_CHILD_ID, self.type))))
+        valid_ack = vol.In([0, 1], msg='Not valid ack flag: {}'.format(
+            self.ack))
+        valid_sub_types = vol.In(
+            [member.value for member
+             in const.VALID_MESSAGE_TYPES.get(self.type, [])],
+            msg='Not valid message sub-type: {}'.format(self.sub_type))
+        valid_payload = const.VALID_PAYLOADS.get(
+            self.type, {}).get(self.sub_type, '')
+        schema = vol.Schema({
+            'node_id': valid_node_ids, 'child_id': valid_child_ids,
+            'type': valid_types, 'ack': valid_ack, 'sub_type': valid_sub_types,
+            'payload': valid_payload})
+        to_validate = {attr: getattr(self, attr) for attr in schema.schema}
+        return schema(to_validate)
+
 
 class MySensorsJSONEncoder(json.JSONEncoder):
     """JSON encoder."""
@@ -595,11 +694,8 @@ def dict_to_object(self, obj):  # pylint: disable=no-self-use
                 setattr(sensor, key, val)
             return sensor
         elif all(k in obj for k in ['id', 'type', 'values']):
-            # Handle new optional description attribute
-            if 'description' in obj:
-                child = ChildSensor(obj['id'], obj['type'], obj['description'])
-            else:
-                child = ChildSensor(obj['id'], obj['type'])
+            child = ChildSensor(
+                obj['id'], obj['type'], obj.get('description', ''))
             child.values = obj['values']
             return child
         elif all(k.isdigit() for k in obj.keys()):