FOGL-1499 Shutdown & restart using signals from systemctl script to avoid authentication issues

Makefile

@@ -256,6 +256,7 @@ generate_selfcertificate:
 	scripts/auth_certificates ca $(AUTH_NAME) $(SSL_DAYS)
 	scripts/auth_certificates user user $(SSL_DAYS)
 	scripts/auth_certificates user admin $(SSL_DAYS)
+	scripts/auth_certificates user systemctl $(SSL_DAYS)
 
 ###############################################################################
 ############################ C BUILD/INSTALL TARGETS ##########################

VERSION

@@ -1,2 +1,2 @@
 fledge_version=3.1.0
-fledge_schema=76
+fledge_schema=77

docs/quick_start/starting.rst

@@ -18,13 +18,19 @@ For example, to start the Fledge system, open a session to the Fledge device and
 
 If authentication is enabled, which is the default mode for Fledge version 3.0 onward, then a number of the  commands require authentication. Authentication can be accomplished by several means;
 
-  - Set the environment variable *USERNAME* to be the user name.
+  - Set the environment variable *FLEDGE_USER* to be the user name.
 
   - Pass the *-u* flag flag to the command to specify a user name.
 
+  - Create an authentication file
+
   - If neither of the above are done the user will be prompted to enter a user name.
 
-In both cases the user will be prompted to enter a password. It is possible, but not recommended, to set an environment variable *PASSWORD* or pass the *-p* flag on the command line, with the plain text version of the password.
+.. note::
+
+   It is recommended to create an authentication file rather than pass parameters to the fledge command or set environment varaiables as both these methods can expose plain text user names, or passwords, to other users of the system.
+
+In both cases the user will be prompted to enter a password. It is possible, but not recommended, to set an environment variable *FLEDGE_PASSWORD* or pass the *-p* flag on the command line, with the plain text version of the password.
 
 .. code-block:: console
 
@@ -47,6 +53,44 @@ It is also possible to use certificate based authentication to login to the syst
 
 .. note::
 
-   Extreme caution should be taken when storing certificate files that they not be readable by any other user within the system.
+   Extreme caution should be taken when storing certificate files. They must not be readable by any other users within the system.
 
 Following a successful authentication attempt a time based token is issued that allows the user to run further commands, for a limited time, without the need to authenticate again.
+
+Authentication File
+-------------------
+
+The prompting for username and password when using the *fledge* script can be bypassed if an authentication file is created. This is a file that should be created in a directory called *.fledge* in the user's home directory.
+
+The file created should be called *auth* and contains the credentials required to login. This may either be a username and password or the filename of a certficate to use to authenticate.
+
+.. note::
+
+   The *auth* file will only be read if the permissions on that file are set such that only the owner can read the file.
+
+   .. code-block:: console
+
+       $ chmod 600 ~/.fledge/auth
+
+   In older versions of Fledge the *auth* file was simply called *~/.fledge*. If the older *.fledge* file exists it will still be used.
+
+An example *auth* file, using the default username and password would be as follows
+
+.. code-block:: console
+
+   FLEDGE_USER=admin
+   FLEDGE_PASSWORD=fledge
+
+If using a certificate to authenticate the file would look as follow
+
+.. code-block:: console
+
+   FLEDGE_CERT=~/.auth/user.cert
+
+The file name, minus the extension, should match the user name of the user.
+
+.. note::
+
+   In the above example the certificate has been placed in the .auth directory, this is not a requirement and the user name choose to place the certificate in any location that is convienent for them. However the certificate file should be protected sich that it can not be red or copied by other users.
+
+

docs/securing_fledge.rst

@@ -227,6 +227,27 @@ Fledge provides a mechanism to limit the age of passwords in use within the syst
 
 Whenever a user logs into Fledge the age of their password is checked against the maximum allowed password age. If their password has reached that age then the user is not logged in, but is instead forced to enter a new password. They must then login with that new password. In addition the system maintains a history of the last three passwords the user has used and prevents them being reused.
 
+Management Script Login
+-----------------------
+
+Fledge supports a number of mechanisms for user authentication when using the *fledge* script to manage startup, shutdown and other operations.
+
+  - The default authentication mechanism will prompt the entry of user name and password interactively.
+
+  - The user may set one or both of the FLEDGE_USER and FLEDGE_PASSWORD environment variables to automate the entry of the username or password.
+
+  - The user may pass either or both of the *-u* and *-p* flags to set the username and password on the command line.
+
+  - The user may pass the *-c* flag and a path to a certificate file that is used to authenticate the user.
+
+  - The user creates an authentication file that contains the username and password
+
+.. note::
+
+   In all cases where a password is stored or passed to the script this is done in plain text. Care should be taken to protect these plain text passwords. In the case of the authentication file, the file permissions must be such that only the Linux user is able to read the file.
+
+   Using a certificate to authenticate rather than username and password, alleviates the need to have plain text passwords stored on the system. However access to the certificate must be protected. The ability to access the certificate provides the ability to authenticate wit the Fledge instance without need for any further information.
+
 
 User Management
 ===============
@@ -346,7 +367,6 @@ To add a new certificate select the *Import* icon in the top right of the certif
 
 A dialog will appear that allows a key file and/or a certificate file to be selected and uploaded to the *Certificate Store*. An option allows to allow overwrite of an existing certificate. By default certificates may not be overwritten.
 
-
 Custom Certificate Authority (CA)
 ---------------------------------
 
@@ -370,12 +390,16 @@ After obtaining the certificates, modify the **Root CA Certificate** setting (cu
 
     If there are any intermediate certificates forming a chain, consolidate them into a single file named **intermediate.cert or intermediate.pem** and store it in the same directory specified earlier.
 
-Generate a new auth certificates for user login
------------------------------------------------
+Generate a new certificates for user login
+------------------------------------------
+
+A default certificate authority (CA) certificate is available inside $FLEDGE_DATA/etc/certs and named as ca.cert. Also default admin, systemctl and non-admin certificates are available in the same location which will be used for Login with Certificate in Fledge i.e admin.cert, systemctl.cert and user.cert. See |Require User Login|
+
+.. note::
 
-Default ca certificate is available inside $FLEDGE_DATA/etc/certs and named as ca.cert. Also default admin and non-admin certs are available in the same location which will be used for Login with Certificate in Fledge i.e admin.cert, user.cert. See |Require User Login|
+   The systemctl.cert certificate is used by the Linux systemctl scripts to start, stop and monitor the state of the Fledge instance, using the status command, and should not be removed.
 
-Below are the steps to create custom certificate along with existing fledge based ca signed for auth certificates.
+Below are the steps to create custom certificates along with existing Fledge based CA signed for authentication certificates.
 
 **Using cURL**
 
@@ -408,7 +432,7 @@ You may also refer the documentation of |REST API| cURL commands. If you are not
 
 .. note::
 
-   Steps a (cert creation) and b (create user) can be executed in any order.
+   Steps a (cert creation) and b (create user) can be executed in either order.
 
 c) Now you can login with the newly created user **test**, with the following cURL
 

extras/scripts/fledge.service

@@ -55,6 +55,7 @@ fi
 FLEDGE_ROOT="/usr/local/fledge"
 FLEDGE_DATA="${FLEDGE_ROOT}/data"
 FLEDGE_USER=`ls -ld "${FLEDGE_DATA}" | awk '{print $3}'`
+FLEDGE_CERT="${FLEDGE_ROOT}/data/etc/certs/systemctl.cert"
 PID_FILE="${FLEDGE_DATA}/var/run/fledge.core.pid"
 PID=0
 
@@ -68,15 +69,27 @@ get_pid() {
 
 fledge_start() {
     if [ "$IS_RHEL" = "" ]; then
-        sudo -u ${FLEDGE_USER} "${FLEDGE_ROOT}/bin/fledge" start > /dev/null
+        if [ -f "${FLEDGE_CERT}" ]; then
+            sudo -u ${FLEDGE_USER} "${FLEDGE_ROOT}/bin/fledge" -c "$FLEDGE_CERT}" start > /dev/null
+        else
+            sudo -u ${FLEDGE_USER} "${FLEDGE_ROOT}/bin/fledge" start > /dev/null
+	fi
+    elif [ -f "${FLEDGE_CERT}" ]; then
+        "${FLEDGE_ROOT}/bin/fledge" -c "${FLEDGE_CERT}" start > /dev/null
     else
         "${FLEDGE_ROOT}/bin/fledge" start > /dev/null
     fi
 }
 
 fledge_stop() {
     if [ "$IS_RHEL" = "" ]; then
-        sudo -u ${FLEDGE_USER} "${FLEDGE_ROOT}/bin/fledge" stop > /dev/null
+	if [ -f "${FLEDGE_CERT}" ]; then
+            sudo -u ${FLEDGE_USER} "${FLEDGE_ROOT}/bin/fledge" -c "${FLEDGE_CERT}" stop > /dev/null
+	else
+            sudo -u ${FLEDGE_USER} "${FLEDGE_ROOT}/bin/fledge" stop > /dev/null
+	fi
+    elif [ -f "${FLEDGE_CERT}" ]; then
+        "${FLEDGE_ROOT}/bin/fledge" -c "${FLEDGE_CERT}" stop > /dev/null
     else
         "${FLEDGE_ROOT}/bin/fledge" stop > /dev/null
     fi
@@ -139,7 +152,7 @@ case "$1" in
         rm -f $PID_FILE
         exit 1
       else
-        fledge_stop
+        kill -TERM $PID
         echo "Fledge stopped [$PID]"
       fi
     fi
@@ -148,7 +161,19 @@ case "$1" in
 
   restart)
 
-    $0 fledge_stop
+    get_pid
+    if [ $PID -eq 0 ]; then
+      echo "Fledge not running"
+    else
+      ps -p $PID
+      if [ $? -eq 1 ]; then
+        echo "Fledge not running (process dead but PID file exists)"
+        rm -f $PID_FILE
+      else
+        kill -TERM $PID
+        echo "Fledge stopped [$PID]"
+      fi
+    fi
     $0 fledge_start
     ;;
 

python/fledge/common/web/middleware.py

@@ -200,13 +200,26 @@ async def validate_requests(request):
            - same as normal user can do
            - All CRUD's privileges for control scripts
            - All CRUD's privileges for control pipelines
+        e) With "systemctl" based user role id = 6 only
+           - login and log out
+           - ping, health, audit, service, profile (GET call)
+           - shutdown (PUT call)
     """
     user_id = request.user['id']
     # Only URL's which are specific meant for Admin user
     if not request.user_is_admin and request.method == 'GET':
         # Special case: Allowed GET user for Control user
         if int(request.user["role_id"]) != 5 and str(request.rel_url) == '/fledge/user':
             raise web.HTTPForbidden
+    # Admin restrictions
+    if int(request.user["role_id"]) == 1:
+        # Admin cannot delete and update the systemctl user (userid = 3)
+        if request.method == 'DELETE':
+            if str(request.rel_url).startswith('/fledge/admin/3/delete'):
+                raise web.HTTPForbidden
+        elif request.method == 'PUT':
+            if str(request.rel_url).startswith('/fledge/admin/3'):
+                raise web.HTTPForbidden
     # Normal/Editor user
     if int(request.user["role_id"]) == 2 and request.method != 'GET':
         # Special case: Allowed control entrypoint update request and handling of rejection in its handler
@@ -237,7 +250,23 @@ async def validate_requests(request):
                 raise web.HTTPForbidden
         else:
             raise web.HTTPForbidden
-
+    # Systemctl user
+    elif int(request.user["role_id"]) == 6:
+        if request.method == 'GET':
+            supported_endpoints = ['/fledge/ping', '/fledge/health/logging', '/fledge/health/storage',
+                                   '/fledge/user?id=3', '/fledge/audit?source=SRVFL']
+            if not (str(request.rel_url).startswith(tuple(supported_endpoints)) or
+                    str(request.rel_url).endswith('/fledge/service')):
+                raise web.HTTPForbidden
+        elif request.method == 'PUT':
+            supported_endpoints = ['/fledge/shutdown', '/logout']
+            if not str(request.rel_url).endswith(tuple(supported_endpoints)):
+                raise web.HTTPForbidden
+        elif request.method == 'post':
+            if not str(request.rel_url).startswith('fledge/login'):
+                raise web.HTTPForbidden
+        else:
+            raise web.HTTPForbidden
 
 def check_firewall(req: web.Request) -> None:
     # FIXME: Need to check on other environments like AWS, docker; May be ideal with socket

python/fledge/services/core/api/auth.py

@@ -436,6 +436,9 @@ async def create_user(request):
     if not (await is_valid_role(role_id)):
         msg = "Invalid role ID."
         raise web.HTTPBadRequest(reason=msg, body=json.dumps({"message": msg}))
+    if role_id == 6:
+        msg = "We cannot create a user with systemctl role."
+        raise web.HTTPBadRequest(reason=msg, body=json.dumps({"message": msg}))
     users = await User.Objects.all()
     unames = [u['uname'] for u in users]
     if username in unames:

python/fledge/services/core/api/certificate_store.py

@@ -101,8 +101,8 @@ async def upload(request):
     cert_filename = cert_file.filename
 
     # default installed auth cert keys can be deleted, for matching/debugging disallow overwrite
-    if cert_filename in ['admin.cert', 'admin.key', 'user.cert', 'user.key', 'fledge.key', 'fledge.cert', 'ca.key',
-                         'ca.cert']:
+    if cert_filename in ['admin.cert', 'admin.key', 'user.cert', 'user.key', 'systemctl.cert', 'systemctl.key',
+                         'fledge.key', 'fledge.cert', 'ca.key', 'ca.cert']:
         if request.is_auth_optional:
             _logger.warning(FORBIDDEN_MSG)
             raise web.HTTPForbidden(reason=FORBIDDEN_MSG, body=json.dumps({"message": FORBIDDEN_MSG}))
@@ -197,7 +197,7 @@ async def delete_certificate(request):
     if not file_name.endswith(valid_extensions):
         msg = "Accepted file extensions are {}".format(valid_extensions)
         raise web.HTTPBadRequest(reason=msg, body=json.dumps({"message": msg}))
-    if file_name in ['admin.cert', 'user.cert', 'fledge.key', 'fledge.cert', 'ca.key', 'ca.cert']:
+    if file_name in ['admin.cert', 'user.cert', 'systemctl.cert', 'fledge.key', 'fledge.cert', 'ca.key', 'ca.cert']:
         if request.is_auth_optional:
             _logger.warning(FORBIDDEN_MSG)
             raise web.HTTPForbidden(reason=FORBIDDEN_MSG, body=json.dumps({"message": FORBIDDEN_MSG}))

python/fledge/services/core/server.py

@@ -405,6 +405,9 @@ class Server:
     _alert_manager = None
     """ Alert Manager """
 
+    _loop = None
+    """ Event loop reference for signal handlers """
+
     running_in_safe_mode = False
     """ Fledge running in Safe mode """
 
@@ -1048,12 +1051,89 @@ async def _get_alerts(cls):
         cls._alert_manager = AlertManager(cls._storage_client_async)
         await cls._alert_manager.get_all()
 
+    @classmethod
+    def _signal_handler(cls, signum, frame):
+        """Signal handler for graceful shutdown and restart
+
+        Args:
+            signum: Signal number
+            frame: Current stack frame
+        """
+        signal_name = signal.Signals(signum).name
+
+        # Handle SIGHUP as restart, others as shutdown
+        if signum == signal.SIGHUP:
+            _logger.info("Received signal %s (%d), initiating graceful restart...", signal_name, signum)
+            if cls._loop and not cls._loop.is_closed():
+                asyncio.run_coroutine_threadsafe(cls._signal_restart(), cls._loop)
+            else:
+                _logger.error("Event loop not available for graceful restart")
+                sys.exit(1)
+        else:
+            _logger.info("Received signal %s (%d), initiating graceful shutdown...", signal_name, signum)
+            if cls._loop and not cls._loop.is_closed():
+                asyncio.run_coroutine_threadsafe(cls._signal_shutdown(), cls._loop)
+            else:
+                _logger.error("Event loop not available for graceful shutdown")
+                sys.exit(1)
+
+    @classmethod
+    async def _signal_shutdown(cls):
+        """Immediate shutdown method for signal handlers (no delay)
+
+        This method provides immediate shutdown without the 2-second delay
+        that's appropriate for HTTP API shutdowns but not for signal handling.
+        """
+        try:
+            await cls._stop()
+            _logger.info("Stopping the Fledge Core event loop. Good Bye!")
+            cls._loop.stop()
+        except Exception as ex:
+            _logger.error("Error during signal shutdown: %s", str(ex))
+            cls._loop.stop()
+
+    @classmethod
+    async def _signal_restart(cls):
+        """Immediate restart method for signal handlers (no delay)
+
+        This method provides immediate restart without the 2-second delay
+        that's appropriate for HTTP API restarts but not for signal handling.
+        """
+        try:
+            await cls._stop()
+            _logger.info("Restarting Fledge Core...")
+
+            # Allow some time for cleanup
+            await asyncio.sleep(1.0)
+
+            # Remove safe-mode from sys.argv if present
+            if 'safe-mode' in sys.argv:
+                sys.argv.remove('safe-mode')
+                sys.argv.append('')
+
+            # Restart the process
+            python3 = sys.executable
+            os.execl(python3, python3, *sys.argv)  # Replaces current process, no return
+        except Exception as ex:
+            _logger.error("Error during signal restart: %s", str(ex))
+            cls._loop.stop()
+
     @classmethod
     def _start_core(cls, loop=None):
         if cls.running_in_safe_mode:
             _logger.info("Starting in SAFE MODE ...")
         else:
             _logger.info("Starting ...")
+
+        # Store the loop reference for signal handlers
+        cls._loop = loop
+
+        # Register signal handlers for graceful shutdown
+        signal.signal(signal.SIGHUP, cls._signal_handler)
+        signal.signal(signal.SIGTERM, cls._signal_handler)
+        signal.signal(signal.SIGINT, cls._signal_handler)
+        _logger.info("Signal handlers registered for SIGHUP, SIGTERM, and SIGINT")
+
         try:
             host = cls._host