Settings placeholder substitution (#164)

* Use text formatting for default factory values to produce dynamic names.
Header files contains duplicates of factory values defined in factory_settings.ini Removed them to simplify the code.

* Use text formatting for default factory values to produce dynamic names.
Header files contains duplicates of factory values defined in factory_settings.ini Removed them to simplify the code.

* Configured the WiFi host name to contain the device id by default

* Removed possibility to use placeholders for FACTORY_WIFI_SSID factory setting.

* Update README.md

Updated documentation

* Use text formatting for default factory values to produce dynamic names.
Header files contains duplicates of factory values defined in factory_settings.ini Removed them to simplify the code.

* Configured the WiFi host name to contain the device id by default

* Removed possibility to use placeholders for FACTORY_WIFI_SSID factory setting.

* Added a space to the end of the file to comply project code style

* fix typos
clang formatting
use 2 spaces in ini files
use ${platform}-${chip_id} for hostname
put chip id in brackets in AP SSID

* restore (and update) factory setting ifndefs

- this is so src can be built without an exaustive set build-time defines
- standardize ordering of defines: factory settings, paths, config

* format and modify comment

* escape spaces in pio defines
experiment with removing $'s from our format strings (they are being substituted with empty values by pio)

* fix formatting in readme
rename FactoryValue to SettingValue, put in own header
give example of direct usage of FactorySetting::format in README.md

* auto format

* use hash to delimit placeholders

* fix factory_settings.ini

* remove flash string helpers

* format ini file

* use MAC address instead of chip id for properly unique identifier

* use lower case hex encoding for unique id
use chip id and unique id for more secure secret

* fix comment

* Use random values for JWT secret
Arduino uses the ESP random number generator for "true random" numbers on both esp32 and esp8266
This makes a better JWT secret and may be useful for other factory defaults too
In addition a modification has been made to force the FSPersistance to save the file if applying defaults

* Don't use spaces in default AP SSID

* restore helpful comment in factory_settings.ini
fix default defines

Co-authored-by: kasedy <[email protected]>

Author: rjwats

README.md

@@ -169,12 +169,12 @@ Many of the framework's built in features may be enabled or disabled as required
 Customize the settings as you see fit. A value of 0 will disable the specified feature:
 
 ```ini
-    -D FT_PROJECT=1
-    -D FT_SECURITY=1
-    -D FT_MQTT=1
-    -D FT_NTP=1
-    -D FT_OTA=1
-    -D FT_UPLOAD_FIRMWARE=1
+  -D FT_PROJECT=1
+  -D FT_SECURITY=1
+  -D FT_MQTT=1
+  -D FT_NTP=1
+  -D FT_OTA=1
+  -D FT_UPLOAD_FIRMWARE=1
 ```
 
 Flag               | Description
@@ -193,9 +193,9 @@ The framework has built-in factory settings which act as default values for the
 Customize the settings as you see fit, for example you might configure your home WiFi network as the factory default:
 
 ```ini
-    -D FACTORY_WIFI_SSID=\"My Awesome WiFi Network\"
-    -D FACTORY_WIFI_PASSWORD=\"secret\"
-    -D FACTORY_WIFI_HOSTNAME=\"awesome_light_controller\"
+  -D FACTORY_WIFI_SSID=\"My Awesome WiFi Network\"
+  -D FACTORY_WIFI_PASSWORD=\"secret\"
+  -D FACTORY_WIFI_HOSTNAME=\"awesome_light_controller\"
 ```
 
 ### Default access point settings
@@ -221,15 +221,31 @@ It is recommended that you change the user credentials from their defaults bette
 Changing factory time zone setting is a common requirement. This requires a little effort because the time zone name and POSIX format are stored as separate values for the moment. The time zone names and POSIX formats are contained in the UI code in [TZ.tsx](interface/src/ntp/TZ.tsx). Take the appropriate pair of values from there, for example, for Los Angeles you would use:
 
 ```ini
-    -D FACTORY_NTP_TIME_ZONE_LABEL=\"America/Los_Angeles\"
-    -D FACTORY_NTP_TIME_ZONE_FORMAT=\"PST8PDT,M3.2.0,M11.1.0\"
+  -D FACTORY_NTP_TIME_ZONE_LABEL=\"America/Los_Angeles\"
+  -D FACTORY_NTP_TIME_ZONE_FORMAT=\"PST8PDT,M3.2.0,M11.1.0\"
 ```
 
-### Device ID factory defaults
+### Placeholder substitution
 
-If not overridden with a build flag, the framework will use the device ID to generate factory defaults for settings such as the JWT secret and MQTT client ID. 
+Various settings support placeholder substitution, indicated by comments in [factory_settings.ini](factory_settings.ini). This can be particularly useful where settings need to be unique, such as the Access Point SSID or MQTT client id. The following placeholders are supported:
 
-> **Tip**: Random values are generally better defaults for these settings, so it is recommended you leave these flags undefined.
+Placeholder  | Substituted value 
+-----------  | -----------------
+#{platform}  | The microcontroller platform, e.g. "esp32" or "esp8266"
+#{unique_id} | A unique identifier derived from the MAC address, e.g. "0b0a859d6816"
+#{random}    | A random number encoded as a hex string, e.g. "55722f94"
+
+You may use SettingValue::format in your own code if you require the use of these placeholders. This is demonstrated in the demo project:
+
+```cpp
+  static StateUpdateResult update(JsonObject& root, LightMqttSettings& settings) {
+    settings.mqttPath = root["mqtt_path"] | SettingValue::format("homeassistant/light/#{unique_id}");
+    settings.name = root["name"] | SettingValue::format("light-#{unique_id}");
+    settings.uniqueId = root["unique_id"] | SettingValue::format("light-#{unique_id}");
+    return StateUpdateResult::CHANGED;
+  }
+};
+```
 
 ## Building for different devices
 

factory_settings.ini

@@ -1,48 +1,51 @@
+; The indicated settings support placeholder substitution as follows:
+;
+;  #{platform} - The microcontroller platform, e.g. "esp32" or "esp8266"
+;  #{unique_id} - A unique identifier derived from the MAC address, e.g. "0b0a859d6816"
+;  #{random} - A random number encoded as a hex string, e.g. "55722f94"
+
 [factory_settings]
-build_flags = 
-    ; WiFi settings
-    -D FACTORY_WIFI_SSID=\"\"
-    -D FACTORY_WIFI_PASSWORD=\"\"
-    ; if unspecified the devices hardware ID will be used
-    ; -D FACTORY_WIFI_HOSTNAME=\"esp-react\"  
+build_flags =
+  ; WiFi settings
+  -D FACTORY_WIFI_SSID=\"\"
+  -D FACTORY_WIFI_PASSWORD=\"\"
+  -D FACTORY_WIFI_HOSTNAME=\"#{platform}-#{unique_id}\" ; supports placeholders
 
-    ; Access point settings
-    -D FACTORY_AP_PROVISION_MODE=AP_MODE_DISCONNECTED
-    -D FACTORY_AP_SSID=\"ESP8266-React\" ; 1-64 characters
-    -D FACTORY_AP_PASSWORD=\"esp-react\" ; 8-64 characters
-    -D FACTORY_AP_LOCAL_IP=\"192.168.4.1\"
-    -D FACTORY_AP_GATEWAY_IP=\"192.168.4.1\"
-    -D FACTORY_AP_SUBNET_MASK=\"255.255.255.0\"
+  ; Access point settings
+  -D FACTORY_AP_PROVISION_MODE=AP_MODE_DISCONNECTED
+  -D FACTORY_AP_SSID=\"ESP8266-React-#{unique_id}\" ; 1-64 characters, supports placeholders
+  -D FACTORY_AP_PASSWORD=\"esp-react\" ; 8-64 characters
+  -D FACTORY_AP_LOCAL_IP=\"192.168.4.1\"
+  -D FACTORY_AP_GATEWAY_IP=\"192.168.4.1\"
+  -D FACTORY_AP_SUBNET_MASK=\"255.255.255.0\"
 
-    ; User credentials for admin and guest user
-    -D FACTORY_ADMIN_USERNAME=\"admin\"
-    -D FACTORY_ADMIN_PASSWORD=\"admin\"
-    -D FACTORY_GUEST_USERNAME=\"guest\"
-    -D FACTORY_GUEST_PASSWORD=\"guest\"
+  ; User credentials for admin and guest user
+  -D FACTORY_ADMIN_USERNAME=\"admin\"
+  -D FACTORY_ADMIN_PASSWORD=\"admin\"
+  -D FACTORY_GUEST_USERNAME=\"guest\"
+  -D FACTORY_GUEST_PASSWORD=\"guest\"
 
-    ; NTP settings
-    -D FACTORY_NTP_ENABLED=true
-    -D FACTORY_NTP_TIME_ZONE_LABEL=\"Europe/London\"
-    -D FACTORY_NTP_TIME_ZONE_FORMAT=\"GMT0BST,M3.5.0/1,M10.5.0\"
-    -D FACTORY_NTP_SERVER=\"time.google.com\"
+  ; NTP settings
+  -D FACTORY_NTP_ENABLED=true
+  -D FACTORY_NTP_TIME_ZONE_LABEL=\"Europe/London\"
+  -D FACTORY_NTP_TIME_ZONE_FORMAT=\"GMT0BST,M3.5.0/1,M10.5.0\"
+  -D FACTORY_NTP_SERVER=\"time.google.com\"
 
-    ; OTA settings
-    -D FACTORY_OTA_PORT=8266
-    -D FACTORY_OTA_PASSWORD=\"esp-react\"
-    -D FACTORY_OTA_ENABLED=true
+  ; OTA settings
+  -D FACTORY_OTA_PORT=8266
+  -D FACTORY_OTA_PASSWORD=\"esp-react\"
+  -D FACTORY_OTA_ENABLED=true
 
-    ; MQTT settings
-    -D FACTORY_MQTT_ENABLED=false
-    -D FACTORY_MQTT_HOST=\"test.mosquitto.org\"
-    -D FACTORY_MQTT_PORT=1883
-    -D FACTORY_MQTT_USERNAME=\"\"
-    -D FACTORY_MQTT_PASSWORD=\"\"
-    ; if unspecified the devices hardware ID will be used
-    ;-D FACTORY_MQTT_CLIENT_ID=\"esp-react\"
-    -D FACTORY_MQTT_KEEP_ALIVE=60
-    -D FACTORY_MQTT_CLEAN_SESSION=true
-    -D FACTORY_MQTT_MAX_TOPIC_LENGTH=128
+  ; MQTT settings
+  -D FACTORY_MQTT_ENABLED=false
+  -D FACTORY_MQTT_HOST=\"test.mosquitto.org\"
+  -D FACTORY_MQTT_PORT=1883
+  -D FACTORY_MQTT_USERNAME=\"\" ; supports placeholders
+  -D FACTORY_MQTT_PASSWORD=\"\"
+  -D FACTORY_MQTT_CLIENT_ID=\"#{platform}-#{unique_id}\" ; supports placeholders
+  -D FACTORY_MQTT_KEEP_ALIVE=60
+  -D FACTORY_MQTT_CLEAN_SESSION=true
+  -D FACTORY_MQTT_MAX_TOPIC_LENGTH=128
 
-    ; JWT Secret
-    ; if unspecified the devices hardware ID will be used
-    ; -D FACTORY_JWT_SECRET=\"esp8266-react\"
+  ; JWT Secret
+  -D FACTORY_JWT_SECRET=\"#{random}-#{random}\" ; supports placeholders

features.ini

@@ -1,8 +1,8 @@
 [features]
 build_flags = 
-    -D FT_PROJECT=1
-    -D FT_SECURITY=1
-    -D FT_MQTT=1
-    -D FT_NTP=1
-    -D FT_OTA=1
-    -D FT_UPLOAD_FIRMWARE=1
+  -D FT_PROJECT=1
+  -D FT_SECURITY=1
+  -D FT_MQTT=1
+  -D FT_NTP=1
+  -D FT_OTA=1
+  -D FT_UPLOAD_FIRMWARE=1

lib/framework/APSettingsService.h

@@ -1,27 +1,20 @@
 #ifndef APSettingsConfig_h
 #define APSettingsConfig_h
 
+#include <SettingValue.h>
 #include <HttpEndpoint.h>
 #include <FSPersistence.h>
 #include <JsonUtils.h>
 
 #include <DNSServer.h>
 #include <IPAddress.h>
 
-#define MANAGE_NETWORK_DELAY 10000
-
-#define AP_MODE_ALWAYS 0
-#define AP_MODE_DISCONNECTED 1
-#define AP_MODE_NEVER 2
-
-#define DNS_PORT 53
-
 #ifndef FACTORY_AP_PROVISION_MODE
 #define FACTORY_AP_PROVISION_MODE AP_MODE_DISCONNECTED
 #endif
 
 #ifndef FACTORY_AP_SSID
-#define FACTORY_AP_SSID "ESP8266-React"
+#define FACTORY_AP_SSID "ESP8266-React-#{unique_id}"
 #endif
 
 #ifndef FACTORY_AP_PASSWORD
@@ -43,6 +36,14 @@
 #define AP_SETTINGS_FILE "/config/apSettings.json"
 #define AP_SETTINGS_SERVICE_PATH "/rest/apSettings"
 
+#define MANAGE_NETWORK_DELAY 10000
+
+#define AP_MODE_ALWAYS 0
+#define AP_MODE_DISCONNECTED 1
+#define AP_MODE_NEVER 2
+
+#define DNS_PORT 53
+
 enum APNetworkStatus { ACTIVE = 0, INACTIVE, LINGERING };
 
 class APSettings {
@@ -79,7 +80,7 @@ class APSettings {
       default:
         newSettings.provisionMode = AP_MODE_ALWAYS;
     }
-    newSettings.ssid = root["ssid"] | FACTORY_AP_SSID;
+    newSettings.ssid = root["ssid"] | SettingValue::format(FACTORY_AP_SSID);
     newSettings.password = root["password"] | FACTORY_AP_PASSWORD;
 
     JsonUtils::readIP(root, "local_ip", newSettings.localIP, FACTORY_AP_LOCAL_IP);

lib/framework/ESPUtils.h

@@ -38,9 +38,11 @@ class FSPersistence {
       settingsFile.close();
     }
 
-    // If we reach here we have not been successful in loading the config,
-    // hard-coded emergency defaults are now applied.
+    // If we reach here we have not been successful in loading the config and hard-coded defaults are now applied.
+    // The settings are then written back to the file system so the defaults persist between resets. This last step is
+    // required as in some cases defaults contain randomly generated values which would otherwise be modified on reset.
     applyDefaults();
+    writeToFS();
   }
 
   bool writeToFS() {

lib/framework/FSPersistence.h

@@ -5,12 +5,7 @@
 #include <HttpEndpoint.h>
 #include <FSPersistence.h>
 #include <AsyncMqttClient.h>
-#include <ESPUtils.h>
-
-#define MQTT_RECONNECTION_DELAY 5000
-
-#define MQTT_SETTINGS_FILE "/config/mqttSettings.json"
-#define MQTT_SETTINGS_SERVICE_PATH "/rest/mqttSettings"
+#include <SettingValue.h>
 
 #ifndef FACTORY_MQTT_ENABLED
 #define FACTORY_MQTT_ENABLED false
@@ -33,7 +28,7 @@
 #endif
 
 #ifndef FACTORY_MQTT_CLIENT_ID
-#define FACTORY_MQTT_CLIENT_ID generateClientId()
+#define FACTORY_MQTT_CLIENT_ID "#{platform}-#{unique_id}"
 #endif
 
 #ifndef FACTORY_MQTT_KEEP_ALIVE
@@ -48,13 +43,10 @@
 #define FACTORY_MQTT_MAX_TOPIC_LENGTH 128
 #endif
 
-static String generateClientId() {
-#ifdef ESP32
-  return ESPUtils::defaultDeviceValue("esp32-");
-#elif defined(ESP8266)
-  return ESPUtils::defaultDeviceValue("esp8266-");
-#endif
-}
+#define MQTT_SETTINGS_FILE "/config/mqttSettings.json"
+#define MQTT_SETTINGS_SERVICE_PATH "/rest/mqttSettings"
+
+#define MQTT_RECONNECTION_DELAY 5000
 
 class MqttSettings {
  public:
@@ -91,9 +83,9 @@ class MqttSettings {
     settings.enabled = root["enabled"] | FACTORY_MQTT_ENABLED;
     settings.host = root["host"] | FACTORY_MQTT_HOST;
     settings.port = root["port"] | FACTORY_MQTT_PORT;
-    settings.username = root["username"] | FACTORY_MQTT_USERNAME;
+    settings.username = root["username"] | SettingValue::format(FACTORY_MQTT_USERNAME);
     settings.password = root["password"] | FACTORY_MQTT_PASSWORD;
-    settings.clientId = root["client_id"] | FACTORY_MQTT_CLIENT_ID;
+    settings.clientId = root["client_id"] | SettingValue::format(FACTORY_MQTT_CLIENT_ID);
     settings.keepAlive = root["keep_alive"] | FACTORY_MQTT_KEEP_ALIVE;
     settings.cleanSession = root["clean_session"] | FACTORY_MQTT_CLEAN_SESSION;
     settings.maxTopicLength = root["max_topic_length"] | FACTORY_MQTT_MAX_TOPIC_LENGTH;

lib/framework/MqttSettingsService.h

@@ -4,14 +4,9 @@
 #include <Features.h>
 #include <ArduinoJsonJWT.h>
 #include <ESPAsyncWebServer.h>
-#include <ESPUtils.h>
 #include <AsyncJson.h>
 #include <list>
 
-#ifndef FACTORY_JWT_SECRET
-#define FACTORY_JWT_SECRET ESPUtils::defaultDeviceValue()
-#endif
-
 #define ACCESS_TOKEN_PARAMATER "access_token"
 
 #define AUTHORIZATION_HEADER "Authorization"

lib/framework/SecurityManager.h

@@ -1,11 +1,16 @@
 #ifndef SecuritySettingsService_h
 #define SecuritySettingsService_h
 
+#include <SettingValue.h>
 #include <Features.h>
 #include <SecurityManager.h>
 #include <HttpEndpoint.h>
 #include <FSPersistence.h>
 
+#ifndef FACTORY_JWT_SECRET
+#define FACTORY_JWT_SECRET "#{random}-#{random}"
+#endif
+
 #ifndef FACTORY_ADMIN_USERNAME
 #define FACTORY_ADMIN_USERNAME "admin"
 #endif
@@ -48,7 +53,7 @@ class SecuritySettings {
 
   static StateUpdateResult update(JsonObject& root, SecuritySettings& settings) {
     // secret
-    settings.jwtSecret = root["jwt_secret"] | FACTORY_JWT_SECRET;
+    settings.jwtSecret = root["jwt_secret"] | SettingValue::format(FACTORY_JWT_SECRET);
 
     // users
     settings.users.clear();
@@ -103,7 +108,7 @@ class SecuritySettingsService : public SecurityManager {
   SecuritySettingsService(AsyncWebServer* server, FS* fs);
   ~SecuritySettingsService();
 
-  // minimal set of functions to support framework with security settings disabled  
+  // minimal set of functions to support framework with security settings disabled
   Authentication authenticateRequest(AsyncWebServerRequest* request);
   ArRequestFilterFunction filterRequest(AuthenticationPredicate predicate);
   ArRequestHandlerFunction wrapRequest(ArRequestHandlerFunction onRequest, AuthenticationPredicate predicate);