Added support for auto-updating of zhmcclient resource objects

Details:

* Infrastructure: Added object_topic and job_topic properties to the Session
  class. They are returned from the HMC upon login.

* In FakedSession, set the new object_topic and job_topic properties to faked
  values for now (FakedSession does not model logon, so this is done in init).

* Infrastructure: Added a ResourceUpdater class that maintains a registry of
  resources that are to be auto-updated, and that listens for the object
  notifications and updates the properties of the registered resources.

* Added methods for subscribing and unsubscribing for auto-update, and for
  returning the current subscription status to the Session class.
  Subscribing the session for auto-update causes HMC object notifications
  to be sent as resources change on the HMC.

* Added methods for enabling and disabling auto-update, and for testing the
  current enablement status to the BaseResource class.
  Enabling a resource object for auto-update causes that resource object to be
  auto-updated as changes happen on the HMC.

* Added a new test module test_resource_updater.py that tests the
  ResourceUpdater class by using a FakedSession mock setup. This also tests
  the auto-update methods on BaseResource and Session (FakedSession inherits
  inherits them from Session).

* Enhanced the MockedStompConnection.mock_add_message() method to also accept
  the message as a string, so that invalid JSON strings can be defined in test
  cases.

* Added an example script show_object_notifications.py that uses the new topic
  for object notifications and displays received object notifications in a
  loop, for demonstration purposes.

* Added an example script show_auto_update.py that enables auto-update for a
  partition and that displays a specific partition property in a loop, so that
  any changes to the property are shown.

Signed-off-by: Andreas Maier <[email protected]>

docs/changes.rst

@@ -52,6 +52,9 @@ Released: not yet
 * Made read and write access to the properties dictionary of zhmcclient resource
   objects thread-safe by adding a Python threading.RLock on each resource object.
 
+* Added support for auto-updating of resources. For details, see the new
+  section 'Concepts -> Auto-updating of resources'. (issue #762)
+
 **Cleanup:**
 
 * Removed old build tools that were needed on Travis and Appveyor

docs/concepts.rst

@@ -335,3 +335,100 @@ Examples:
       adapter1 = cpc.adapters.find(**{'object-id':oid})
 
   The returned resource object will have only a minimal set of properties.
+
+
+.. _`Auto-updating of resources`:
+
+Auto-updating of resources
+--------------------------
+
+The resource objects returned by the zhmcclient library support auto-updating
+of resource properties.
+
+By default, auto-updating is disabled for any resource objects, and the user can
+use the :meth:`~zhmcclient.BaseResource.pull_full_properties` method to have the
+properties of the resource object updated explicitly.
+
+If auto-updating is enabled for a resource object (by means of
+:meth:`~zhmcclient.BaseResource.enable_auto_update`), the zhmcclient library
+subscribes on the HMC for object notifications that inform the client about
+changes to resource properties. When receiving such notifications, the client
+updates the properties on the local resource objects that are enabled for
+auto-updating, to the new values.
+
+There is only one subscription at the HMC for each zhmcclient session that has
+auto-updating enabled, so if auto-updating is enabled for a second and further
+resource objects, the already existing subscription is used. When disabling
+auto-updating, the last resource that is disabled will unsubscribe at the HMC.
+
+The subscription for object notifications will cause the following notifications
+to be sent from the HMC to the client:
+
+*  property change notifications for any properties that have the
+   property-change (pc) qualifier set, for all resources,
+*  status change notifications for any properties that have the
+   status-change (sc) qualifier set, for all resources,
+*  inventory change notifications for any resources that come into existence or
+   go out of existence.
+
+The auto-update support in zhmcclient processes the property and status change
+notifications by updating the correponding properties in those resource objects
+that have been enabled for auto-updating. As a result, these properties will
+always have the value the resource object has on the HMC.
+
+The auto-update support in zhmcclient ignores property and status change
+notifications for resource objects that have not been enabled for auto-updating
+and it also ignores inventory change notifications.
+
+The delay for the new property value to become visible in the zhmcclient
+resource object after it has been changed on the HMC, is in the order of 1
+second.
+
+Note that accessing the properties of a zhmcclient resource object is not any
+slower when auto-update is enabled - the auto-update happens asynchronously
+to the access, and depending on whether the access happens before or after an
+auto-update, you get the old or new value.
+
+Example:
+
+.. code-block:: python
+
+    cpc = ...  # A zhmcclient.Cpc object
+    partition_name = 'PART1'
+
+    # Two different zhmcclient.Partition objects representing the same partition on the HMC
+    partition1 = cpc.partitions.find(name=partition_name)
+    partition2 = cpc.partitions.find(name=partition_name)
+    assert id(partition1) != id(partition2)
+
+    partition1.enable_auto_update()  # Enable auto-update for this partition object
+    prop_name = 'description'
+
+    while True:
+        value1 = partition1.prop(prop_name)
+        value2 = partition2.prop(prop_name)
+        print("Property '{}' of objects 1: {!r}, 2: {!r}".
+              format(prop_name, value1, value2))
+        sleep(1)
+
+This example creates two different partition objects representing the same
+partition on the HMC. It enables auto-update for one of the partition objects
+but not for the other, in order to show the different behavior.
+
+The example then prints the value of the 'description' property of both
+partition objects in a loop, so that in parallel, a change of the description
+of the partition can be performed on the HMC (not shown in the example).
+
+Once the description of the partition on the HMC is changed, the partition
+object that has auto-update enabled will show the new value, while the other
+one will show the same value unchanged:
+
+.. code-block:: text
+
+    Property 'description' of objects 1: 'foo', 2: 'foo'
+    Property 'description' of objects 1: 'foo', 2: 'foo'
+    Property 'description' of objects 1: 'foo', 2: 'foo'
+    # description property is changed to 'bar' on the HMC
+    Property 'description' of objects 1: 'bar', 2: 'foo'
+    Property 'description' of objects 1: 'bar', 2: 'foo'
+    Property 'description' of objects 1: 'bar', 2: 'foo'

docs/general.rst

@@ -53,6 +53,20 @@ Retry / timeout configuration
    :special-members: __str__
 
 
+.. _`ResourceUpdater`:
+
+ResourceUpdater
+---------------
+
+.. automodule:: zhmcclient._resource_updater
+
+.. autoclass:: zhmcclient.ResourceUpdater
+  :members:
+  :autosummary:
+  :autosummary-inherited-members:
+  :special-members: __str__
+
+
 .. _`Client`:
 
 Client

examples/show_auto_update.py

@@ -0,0 +1,144 @@
+#!/usr/bin/env python
+# Copyright 2021 IBM Corp. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Example that shows the auto-update of resource properties.
+"""
+
+import sys
+import logging
+import yaml
+import requests.packages.urllib3
+from time import sleep
+
+import zhmcclient
+
+# Print metadata for each OS message, before each message
+PRINT_METADATA = False
+
+requests.packages.urllib3.disable_warnings()
+
+if len(sys.argv) != 2:
+    print("Usage: %s hmccreds.yaml" % sys.argv[0])
+    sys.exit(2)
+hmccreds_file = sys.argv[1]
+
+with open(hmccreds_file, 'r') as fp:
+    hmccreds = yaml.safe_load(fp)
+
+examples = hmccreds.get("examples", None)
+if examples is None:
+    print("examples not found in credentials file %s" % \
+          (hmccreds_file))
+    sys.exit(1)
+
+show_auto_update = examples.get("show_auto_update", None)
+if show_auto_update is None:
+    print("show_auto_update not found in credentials file %s" % \
+          (hmccreds_file))
+    sys.exit(1)
+
+loglevel = show_auto_update.get("loglevel", None)
+if loglevel is not None:
+    level = getattr(logging, loglevel.upper(), None)
+    if level is None:
+        print("Invalid value for loglevel in credentials file %s: %s" % \
+              (hmccreds_file, loglevel))
+        sys.exit(1)
+    logmodule = show_auto_update.get("logmodule", None)
+    if logmodule is None:
+        logmodule = ''  # root logger
+    print("Logging for module %s with level %s" % (logmodule, loglevel))
+    handler = logging.StreamHandler()
+    format_string = '%(asctime)s - %(threadName)s - %(name)s - %(levelname)s - %(message)s'
+    handler.setFormatter(logging.Formatter(format_string))
+    logger = logging.getLogger(logmodule)
+    logger.addHandler(handler)
+    logger.setLevel(level)
+
+hmc = show_auto_update["hmc"]
+cpcname = show_auto_update["cpcname"]
+partname = show_auto_update["partname"]
+partprop = show_auto_update["partprop"]
+
+cred = hmccreds.get(hmc, None)
+if cred is None:
+    print("Credentials for HMC %s not found in credentials file %s" % \
+          (hmc, hmccreds_file))
+    sys.exit(1)
+
+userid = cred['userid']
+password = cred['password']
+
+print(__doc__)
+
+print("Using HMC %s with userid %s ..." % (hmc, userid))
+session = zhmcclient.Session(hmc, userid, password, verify_cert=False)
+cl = zhmcclient.Client(session)
+
+timestats = show_auto_update.get("timestats", False)
+if timestats:
+    session.time_stats_keeper.enable()
+
+session.logon()
+topic = session.object_topic
+if topic is None:
+    print("Error: Object notification topic is not set")
+    sys.exit(1)
+
+print("Object notification topic: %s" % topic)
+
+cpc = cl.cpcs.find(name=cpcname)
+
+# Two different zhmcclient.Partition objects representing the same partition
+# on the HMC
+partition1 = cpc.partitions.find(name=partname)
+partition2 = cpc.partitions.find(name=partname)
+
+print("Enabling auto-update for partition object 1 for partition {} (id={})".
+      format(partname, id(partition1)))
+sys.stdout.flush()
+partition1.enable_auto_update()
+
+print("Not enabling auto-update for partition object 2 for partition {} "
+      "(id={})".format(partname, id(partition2)))
+
+print("Entering loop that displays property '{}' of partition objects 1 and 2".
+      format(partprop))
+sys.stdout.flush()
+try:
+    while True:
+        value1 = partition1.prop(partprop)
+        value2 = partition2.prop(partprop)
+        print("Property '{}' on partition objects 1: {!r}, 2: {!r}".
+              format(partprop, value1, value2))
+        sys.stdout.flush()
+        sleep(1)
+except KeyboardInterrupt:
+    print("Keyboard interrupt - leaving loop")
+    sys.stdout.flush()
+finally:
+    print("Disabling auto-update for partition object 1")
+    sys.stdout.flush()
+    partition1.disable_auto_update()
+
+print("Logging off...")
+sys.stdout.flush()
+session.logoff()
+
+if timestats:
+    print(session.time_stats_keeper)
+
+print("Done.")