Sync update 2017-09-10

Author: nkolban

cpp_utils/.gitignore

@@ -1,2 +1 @@
 /docs/
-/Arduino/

cpp_utils/Arduino/ESP32_BLE.zip

@@ -1,15 +1,13 @@
 # Arduino BLE Support
-As part of this Github project, we provide libraries for Bluetooth Low Energy (BLE) for the ESP32 Arduino environment.  Support for this capability is still in the process of being cooked (as of August 2017).  As such you should not build product based on these functions as changes to the API and implementation are extremely likely over the next couple of months.
+As part of this Github project, we provide libraries for Bluetooth Low Energy (BLE) for the ESP32 Arduino environment.  Support for this capability is still in the process of being cooked (as of September 2017).  As such you should not build product based on these functions as changes to the API and implementation are extremely likely over the next couple of months.
 
 That said, we now have the ability to produce a driver you can use for testing.  This will give you early access to the function while give those who choose to build and maintain the code early feedback on what works, what doesn't and what can be improved from a usage perspective.
 
 When complete, the BLE support for ESP32 Arduino will be available as a single ZIP file.  The file will be called **ESP32_BLE.zip**.  It is this file that will be able to be imported into the Arduino IDE from the `Sketch -> Include Library -> Add .ZIP library`.  When initial development of the library has been completed, this ZIP will be placed under some form of release control so that an ordinary Arduino IDE user can simply download this as a unit and install.
 
-However, today, we are choosing **not** to distribute the BLE support as a downloadable library.   The reason for this is that we believe that the code base for the BLE support is too fluid and it is being actively worked upon.  Each day there will be changes and that means that for any given instance of the ZIP you get, if it is more than a day old, it is likely out of date.
+We provide sample builds of the `ESP32_BLE.zip` file in the `Arduino` folder found relative to this directory.
 
-Instead of giving you a fish (a new ESP32_BLE.zip) we are going to teach you to fish so that you yourself can build the very latest ESP32_BLE.zip yourself.  Hopefully you will find this quite easy to accomplish.
-
-Here is the recipe.
+The build of the Arduino support will be current as of the date of the ZIP file however should you wish to build your own instance of the ZIP from the source, here is the recipe.
 
 1. Create a new directory called `build`
 2. Enter that directory and run `git clone https://github.com/nkolban/esp32-snippets.git`
@@ -24,41 +22,22 @@ And here you will find the `ESP32_BLE.zip` that is build from the latest source.
 If you have previously installed a version of the Arduino BLE Support and need to install a new one, follow the steps above to build yourself a new instance of the `ESP32_BLE.zip` that is ready for installation.  I recommend removing the old one before installing the new one.  To remove the old one, find the directory where the Arduino IDE stores your libraries (on Linux this is commonly `$HOME/Arduino`).  In there you will find a directory called `libraries` and, if you have previously installed the BLE support, you will find a directory called `ESP32_BLE`.  Remove that directory.
 
 ## Switching on debugging
-The BLE support contains extensive internal diagnostics which can be switched on by editing the file called `sdkconfig` and finding the lines which read:
+The BLE support contains extensive internal diagnostics which can be switched on by editing the file called `sdkconfig.h` and finding the lines which read:
 
 ```
-#
-# Log output
-#
-# CONFIG_LOG_DEFAULT_LEVEL_NONE is not set
-CONFIG_LOG_DEFAULT_LEVEL_ERROR=y
-# CONFIG_LOG_DEFAULT_LEVEL_WARN is not set
-# CONFIG_LOG_DEFAULT_LEVEL_INFO is not set
-# CONFIG_LOG_DEFAULT_LEVEL_DEBUG is not set
-# CONFIG_LOG_DEFAULT_LEVEL_VERBOSE is not set
-CONFIG_LOG_DEFAULT_LEVEL=1
+#define CONFIG_LOG_DEFAULT 1
 ```
 
 Change this to:
 
 ```
-#
-# Log output
-#
-# CONFIG_LOG_DEFAULT_LEVEL_NONE is not set
-# CONFIG_LOG_DEFAULT_LEVEL_ERROR=y
-# CONFIG_LOG_DEFAULT_LEVEL_WARN is not set
-# CONFIG_LOG_DEFAULT_LEVEL_INFO is not set
-# CONFIG_LOG_DEFAULT_LEVEL_DEBUG is not set
-CONFIG_LOG_DEFAULT_LEVEL_VERBOSE=y
-CONFIG_LOG_DEFAULT_LEVEL=5
-# CONFIG_LOG_COLORS is not set
+#define CONFIG_LOG_DEFAULT 5
 ```
 
 and rebuild/deploy your project.
 
 This file can be found in your Arduino IDE installation directories at:
 
 ```
-<ArduinoIDE>/hardware/espressif/esp32/tools/sdk
+<ArduinoIDE>/hardware/espressif/esp32/tools/sdk/include/config
 ```

cpp_utils/ArduinoBLE.md

@@ -13,11 +13,11 @@ For example, imagine a WebSocket client wants to send a file of 1MByte to the ES
 
 This sounds workable ... so let us now think about how we might go about creating an input stream for a WebSocket message.  Each WebSocket message starts with a WebSocket frame which contains, amongst other things, the length of the payload data.  This means that we know up front how much of the remaining data is payload.  This becomes essential as we can't rely on an "end of file" marker in the input stream to indicate the end of the WebSocket payload.  The reason for this is that the WebSocket is a TCP stream that will be used to carry multiple sequential messages.
 
-Let us now invent a new class.  Let us call it a SocketInputRecordStream.
+Let us now invent a new class.  Let us call it a SocketInputRecordStreambuf.
 It will have a constructor of the form:
 
 ```
-SocketInputRecordStream(Socket &socket, size_t dataLength, size_t bufferSize=512)
+SocketInputRecordStreambuf(Socket &socket, size_t dataLength, size_t bufferSize=512)
 ```
 
 The `socket` is the TCP/IP socket that we are going to read data from.  The `dataLength` is the size of the data we wish to read.  The class will extend `std::streambuf`.  It will internally maintain a data buffer of size `bufferSize`.  Initially, the buffer will be empty.  When a read is performed on the stream, a call to `underflow()` will be made (this is a `std::streambuf` virtual function).
@@ -27,4 +27,38 @@ Our rules for this class include:
 * We must **not** read more the `dataLength` bytes from the socket.
 * We must **indicate** and `EOF` once we have had `dataLength` bytes consumed by a stream reader.
 * The class must implement a `discard()` method that will discard further bytes from the socket such that the total number of bytes read from the socket will equal `dataLength`.
-* Deleting an instance of the class must invoke `discard()`.
+* Deleting an instance of the class must invoke `discard()`.
+
+## File transfer
+WebSockets make a great file transfer technology.  While this is more an application utilization of the technology than the design of the framework, we'll capture it here.  Let us first set the scene.  We have a client application that wishes to transmit a file.  We will assume that the file is composed of three logical components:
+
+* The file name
+* The file length
+* The content of the file
+
+It would be wrong to expect the client to send the file as one continuous unit in one WebSocket message.  The reason for this is that the client would have to have loaded the complete file into its memory buffers in order to send it.  As such, we should assume that the client will send the files as one or more "parts" where each part represents a piece of the file.
+
+We thus invent the following protocol:
+
+For the first message we have:
+
+```
++-----------------------+-----------+----+--------------------+-----------+
+| transferId (32bit/LE) | file name | \0 | length (32bits/LE) | Data .... |
++-----------------------+-----------+----+--------------------+-----------+
+```
+
+For subsequent messages we have:
+
+```
++-----------------------+-----------+
+| transferId (32bit/LE) | Data .... |
++-----------------------+-----------+
+```
+Let us look at these.
+
+* `transferId` - An Id that is randomly generated by the client.  This is used to associate multiple messages for the same file together.
+* `file name` - The name of the file that the client wishes to send.  Can include paths.  This will be used to determine where on the file system the file will be written.
+* `length` - The size of the file in bytes.  Knowing the size will allow us to know when the whole file has been received.
+
+We will create an encapsulation class called `WebSocketFileTransfer`.

cpp_utils/DesignNotes/WebSockets.md

@@ -26,10 +26,10 @@
  * @param [in] partitionName The name of the partition used to store the FAT file system.
  */
 FATFS_VFS::FATFS_VFS(std::string mountPath, std::string partitionName) {
-	m_mountPath = mountPath;
+	m_mountPath     = mountPath;
 	m_partitionName = partitionName;
-	m_maxFiles = 4;
-	m_wl_handle = WL_INVALID_HANDLE;
+	m_maxFiles      = 4;
+	m_wl_handle     = WL_INVALID_HANDLE;
 } // FATFS_VFS
 
 

cpp_utils/FATFS_VFS.cpp

@@ -10,6 +10,7 @@
 #include <string>
 extern "C" {
 #include <esp_vfs_fat.h>
+#include <wear_levelling.h>
 }
 /**
  * @brief Provide access to the FAT file system on %SPI flash.
@@ -42,14 +43,15 @@ class FATFS_VFS {
 public:
 	FATFS_VFS(std::string mountPath, std::string partitionName);
 	virtual ~FATFS_VFS();
+
 	void mount();
 	void setMaxFiles(int maxFiles);
 	void unmount();
 private:
 	wl_handle_t m_wl_handle;
 	std::string m_mountPath;
 	std::string m_partitionName;
-	int m_maxFiles;
+	int         m_maxFiles;
 };
 
 #endif /* COMPONENTS_CPP_UTILS_FATFS_VFS_H_ */

cpp_utils/FATFS_VFS.h

@@ -122,7 +122,7 @@ HttpRequest::~HttpRequest() {
 } // ~HttpRequest
 
 
-void HttpRequest::close_cpp() {
+void HttpRequest::close() {
 	m_clientSocket.close_cpp();
 } // close_cpp
 

cpp_utils/HttpRequest.cpp

@@ -14,7 +14,7 @@
 #include "WebSocket.h"
 #include "HttpParser.h"
 
-
+#undef close
 
 class HttpRequest {
 private:
@@ -51,10 +51,8 @@ class HttpRequest {
 	static const std::string HTTP_METHOD_POST;
 	static const std::string HTTP_METHOD_PUT;
 
-
-
-	void close_cpp();
-	void dump();
+	void        close();
+	void        dump();
 	std::string getBody();
 	std::string getHeader(std::string name);
 	std::map<std::string, std::string> getHeaders();

cpp_utils/HttpRequest.h

@@ -50,9 +50,9 @@ void HttpResponse::addHeader(const std::string name, const std::string value) {
 } // addHeader
 
 
-void HttpResponse::close_cpp() {
-	m_request->close_cpp();
-} // close_cpp
+void HttpResponse::close() {
+	m_request->close();
+} // close
 
 
 std::string HttpResponse::getHeader(std::string name) {
@@ -74,6 +74,7 @@ std::map<std::string, std::string> HttpResponse::getHeaders() {
  * @param [in] data The data to send to the partner.
  */
 void HttpResponse::sendData(std::string data) {
+	// If we haven't yet sent the header of the data, send that now.
 	if (m_headerCommitted == false) {
 		std::ostringstream oss;
 		oss << m_request->getVersion() << " " << m_status << " " << m_statusMessage << lineTerminator;
@@ -84,6 +85,8 @@ void HttpResponse::sendData(std::string data) {
 		m_headerCommitted = true;
 		m_request->getSocket().send_cpp(oss.str());
 	}
+
+	// Send the payload data.
 	m_request->getSocket().send_cpp(data);
 } // sendData
 

cpp_utils/HttpResponse.cpp

@@ -36,7 +36,7 @@ class HttpResponse {
 	virtual ~HttpResponse();
 
 	void addHeader(std::string name, std::string value);
-	void close_cpp();
+	void close();
 	//std::string getRootPath();
 	std::string getHeader(std::string name);
 	std::map<std::string, std::string> getHeaders();

cpp_utils/HttpResponse.h

@@ -5,6 +5,7 @@
  *      Author: kolban
  */
 
+#include <fstream>
 #include "HttpServer.h"
 #include "SockServ.h"
 #include "Task.h"
@@ -14,6 +15,7 @@
 #include "WebSocket.h"
 static const char* LOG_TAG = "HttpServer";
 
+#undef close
 /**
  * Constructor for HTTP Server
  */
@@ -30,13 +32,16 @@ HttpServer::~HttpServer() {
  */
 class HttpServerTask: public Task {
 public:
-	HttpServerTask(std::string name): Task(name) {};
+	HttpServerTask(std::string name): Task(name) {
+		m_pHttpServer = nullptr;
+	};
 
 private:
 	HttpServer* m_pHttpServer; // Reference to the HTTP Server
 
 	/**
 	 * @brief Process an incoming HTTP Request
+	 *
 	 * We examine each of the path handlers to see if we have a match for the method/path pair.  If we do,
 	 * we invoke the handler callback passing in both the request and response.
 	 *
@@ -45,52 +50,81 @@ class HttpServerTask: public Task {
 	 * @param [in] request The HTTP request to process.
 	 */
 	void processRequest(HttpRequest &request) {
-		ESP_LOGD(LOG_TAG, ">> processRequest: Method: %s, Path: %s",
-			request.getMethod().c_str(), request.getPath().c_str())
+		ESP_LOGD("HttpServerTask", ">> processRequest: Method: %s, Path: %s",
+			request.getMethod().c_str(), request.getPath().c_str());
+
 		for (auto it = m_pHttpServer->m_pathHandlers.begin(); it != m_pHttpServer->m_pathHandlers.end(); ++it) {
 			if (it->match(request.getMethod(), request.getPath())) {
-				ESP_LOGD(LOG_TAG, "Found a path handler match!!");
+				ESP_LOGD("HttpServerTask", "Found a path handler match!!");
 				if (request.isWebsocket()) {
-					it->invoke(&request, nullptr);
+					it->invokePathHandler(&request, nullptr);
 					request.getWebSocket()->startReader();
 				} else {
 					HttpResponse response(&request);
-
-					it->invoke(&request, &response);
+					it->invokePathHandler(&request, &response);
 				}
 				return;
 			} // Path handler match
 		} // For each path handler
 
+		ESP_LOGD("HttpServerTask", "No Path handler found");
+		// If we reach here, then we did not find a handler for the request.
+
+		// Check to see if we have an un-handled WebSocket
+		if (request.isWebsocket()) {
+			request.getWebSocket()->close_cpp();
+			return;
+		}
+
 		// Serve up the content from the file on the file system ... if found ...
 
-		ESP_LOGD(LOG_TAG, "No Path handler found");
+		std::ifstream ifStream;
+		std::string fileName = m_pHttpServer->getRootPath() + request.getPath();
+		ESP_LOGD("HttpServerTask", "Opening file: %s", fileName.c_str());
+		ifStream.open(fileName, std::ifstream::in | std::ifstream::binary);
+
+		// If we failed to open the requested file, then it probably didn't exist so return a not found.
+		if (!ifStream.is_open()) {
+			HttpResponse response(&request);
+			response.setStatus(HttpResponse::HTTP_STATUS_NOT_FOUND, "Not Found");
+			response.sendData("");
+			return;
+		}
+
+		// We now have an open file and want to push the content of that file through to the browser.
 		HttpResponse response(&request);
-		response.setStatus(HttpResponse::HTTP_STATUS_NOT_FOUND, "Not Found");
-		response.sendData("");
+		response.setStatus(HttpResponse::HTTP_STATUS_OK, "OK");
+		std::stringstream ss;
+		ss << ifStream.rdbuf();
+		response.sendData(ss.str());
+		ifStream.close();
 
 	} // processRequest
 
 
 	/**
-	 * Perform the task handling for server.
+	 * @brief Perform the task handling for server.
+	 * @param [in] data A reference to the HttpServer.
 	 */
 	void run(void* data) {
 		m_pHttpServer = (HttpServer*)data;
+
+		// Create a socket server and start it running.
 		SockServ sockServ(m_pHttpServer->getPort());
 		sockServ.start();
-		ESP_LOGD(LOG_TAG, "Listening on port %d", m_pHttpServer->getPort());
+		ESP_LOGD("HttpServerTask", "Listening on port %d", m_pHttpServer->getPort());
+
 		while(1) {
-			ESP_LOGD(LOG_TAG, "Waiting for new client");
+			ESP_LOGD("HttpServerTask", "Waiting for new peer client");
 			Socket clientSocket = sockServ.waitForNewClient();
-			ESP_LOGD(LOG_TAG, "Got a new client");
+			ESP_LOGD("HttpServerTask", "HttpServer listening on port %d received a new client connection", m_pHttpServer->getPort());
+
 			HttpRequest request(clientSocket);
 			request.dump();
 			processRequest(request);
 			if (!request.isWebsocket()) {
-				request.close_cpp();
+				request.close();
 			}
-
 		} // while
 	} // run
 }; // HttpServerTask
@@ -132,6 +166,7 @@ uint16_t HttpServer::getPort() {
 	return m_portNumber;
 } // getPort
 
+
 /**
  * @brief Get the current root path.
  * @return The current root path.
@@ -140,6 +175,7 @@ std::string HttpServer::getRootPath() {
 	return m_rootPath;
 } // getRootPath
 
+
 /**
  * @brief Set the root path for URL file mapping.
  *
@@ -200,7 +236,7 @@ PathHandler::PathHandler(std::string method, std::string pathPattern,
  * @return True if the path matches.
  */
 bool PathHandler::match(std::string method, std::string path) {
-	ESP_LOGD(LOG_TAG, "match: %s with %s", m_textPattern.c_str(), path.c_str());
+	ESP_LOGD(LOG_TAG, "matching: %s with %s", m_textPattern.c_str(), path.c_str());
 	if (method != m_method) {
 		return false;
 	}
@@ -214,6 +250,6 @@ bool PathHandler::match(std::string method, std::string path) {
  * @param [in] response An object representing the response.
  * @return N/A.
  */
-void PathHandler::invoke(HttpRequest* request, HttpResponse *response) {
+void PathHandler::invokePathHandler(HttpRequest* request, HttpResponse *response) {
 	m_requestHandler(request, response);
-} // invoke
+} // invokePathHandler