Better support for Windows Subsystem for Linux (WSL2) (#2194)

Building Sming within a Linux virtual machine is a good way to improve performance of Windows builds.

I've typically done this using Virtualbox since it works the best out of all the others I've tried.
Performance aside, the stumbling block is usually gaining access to serial ports for flashing devices, etc.

I discovered that whilst Windows WSL2 doesn't support USB serial devices natively, there is an easy way
to work around the limitation by using Windows COM ports directly via powershell. See https://github.com/abobija/idfx.

This PR integrates support for that into Sming, plus some documentation additions.

Sming/Components/esptool/component.mk

@@ -47,7 +47,7 @@ COM_SPEED_ESPTOOL		?= $(COM_SPEED)
 
 COMPONENT_SUBMODULES	+= esptool
 DEBUG_VARS				+= ESPTOOL
-ESPTOOL					:= $(PYTHON) $(COMPONENT_PATH)/esptool/esptool.py
+ESPTOOL					:= $(COMPONENT_PATH)/esptool/esptool.py
 ESPTOOL_SUBMODULE		:= $(COMPONENT_PATH)/esptool
 
 $(ESPTOOL): $(ESPTOOL_SUBMODULE)/.submodule
@@ -60,20 +60,30 @@ else ifeq ($(MAKE_DOCS),)
 $(error esptool unsupported arch: $(SMING_ARCH))
 endif
 
-ESPTOOL_CMDLINE := $(ESPTOOL) \
+ESPTOOL_CMDLINE := $(PYTHON) $(ESPTOOL) \
 	-p $(COM_PORT_ESPTOOL) -b $(COM_SPEED_ESPTOOL) \
 	--chip $(ESP_CHIP) --before default_reset --after hard_reset
 
+#
+# USB serial ports are not available under WSL2,
+# but we can use powershell with the regular Windows COM port
+# $1 -> Arguments
+ifdef WSL_ROOT
+ESPTOOL_EXECUTE = powershell.exe -Command "$(ESPTOOL_CMDLINE) $1"
+else
+ESPTOOL_EXECUTE = $(ESPTOOL_CMDLINE) $1
+endif
+
 # Write file contents to Flash
 # $1 -> List of `Offset=File` chunks
 define WriteFlash
 	$(if $1,\
 		$(info WriteFlash $1) \
-		$(ESPTOOL_CMDLINE) write_flash -z $(flashimageoptions) $(subst =, ,$1) \
+		$(call ESPTOOL_EXECUTE,write_flash -z $(flashimageoptions) $(subst =, ,$1)) \
 	)
 endef
 
 # Erase flash memory contents
 define EraseFlash
-	$(ESPTOOL_CMDLINE) erase_flash
+	$(call ESPTOOL_EXECUTE,erase_flash)
 endef

Sming/Components/terminal/component.mk

@@ -17,7 +17,11 @@ endif
 CACHE_VARS		+= COM_OPTS KILL_TERM TERMINAL
 COM_OPTS		?= --raw --encoding ascii
 KILL_TERM		?= pkill -9 -f "$(COM_PORT) $(COM_SPEED_SERIAL)" || exit 0
+ifdef WSL_ROOT
+TERMINAL		?= powershell.exe -Command "python -m serial.tools.miniterm $(COM_OPTS) $(COM_PORT) $(COM_SPEED_SERIAL)"
+else
 TERMINAL		?= $(PYTHON) -m serial.tools.miniterm $(COM_OPTS) $(COM_PORT) $(COM_SPEED_SERIAL)
+endif
 
 
 ##@Tools

Sming/build.mk

@@ -38,6 +38,11 @@ else ifneq ($(filter MSYS%WOW,$(UNAME)),)
 	UNAME		:= Linux
 else ifeq ($(UNAME), Linux)
  	#Linux
+ 	# Detect WSL (Windows Subsystem for Linux)
+ 	ifdef WSL_DISTRO_NAME
+        DEBUG_VARS += WSL_ROOT
+        WSL_ROOT := //wsl$$/$(WSL_DISTRO_NAME)
+ 	endif
 else ifeq ($(UNAME), Darwin)
  	#OS X
 else ifeq ($(UNAME), Freebsd)

docs/source/getting-started.rst

@@ -19,6 +19,10 @@ Sming supports multiple architectures. Choose the architecture of your choice to
 You can also try Sming without installing anything locally.
 We have an  `interactive tutorial <https://www.katacoda.com/slaff/scenarios/sming-host-emulator>`__ that can be run directly from your browser.
 
+Windows users may also wish to consider building in a Linux virtual environment.
+See :doc:`Windows WSL <windows-wsl>` for further details.
+
+
 Documentation
 -------------
 

docs/source/windows-wsl.rst

@@ -0,0 +1,103 @@
+Windows WSL
+===========
+
+.. highlight:: batch
+
+Building under Windows is generally slower than in Linux.
+This is because the current build system requires a Posix emulation layer (MinGW).
+However, it does offer the simplest way to use Sming on a Windows PC and does not affect the quality
+or functionality of your applications.
+
+However, there are situations where it is highly desirable to build Sming in a Linux environment, such as:
+
+-  Making use of linux-only development tools, such as valgrind (dynamic bug detection system)
+-  Integration building/testing prior to submitting a PR to the Sming repository
+-  Need/want faster builds
+
+Whilst a Windows computer can be configured to dual-boot with Linux, this is generally inconvenient
+for day-to-day use. A better solution is to run Linux inside a virtual machine environment such as
+`VirtualBox <https://www.virtualbox.org/>`__,
+`VmWare <https://www.vmware.com/>`__ or
+`Hyper-V <https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/about/>`__.
+
+Note that `Docker <https://www.docker.com/resources/what-container>`__ is not a virtual
+environment but can in fact be run inside a virtual machine to take advantage of the
+process isolation and security benefits.
+
+
+Windows Subsystem for Linux
+---------------------------
+
+https://docs.microsoft.com/en-us/windows/wsl/
+
+"The Windows Subsystem for Linux lets developers run a GNU/Linux environment
+-- including most command-line tools, utilities, and applications --
+directly on Windows, unmodified, without the overhead of a traditional virtual machine or dual-boot setup."
+
+There are currently two versions of WSL, dubbed WSL1 and WSL2.
+Either is fine, and you can switch between versions but WSL2 is recommended. 
+
+.. note::
+
+   WSL2 uses Hyper-V so may conflict with other virtual machines you may be using.
+
+
+Installing WSL
+--------------
+
+See instructions here https://docs.microsoft.com/en-us/windows/wsl/install-win10#manual-installation-steps.
+
+.. note::
+
+   You may encounter an error message similar to this during installation::
+
+      WslRegisterDistribution failed with error: 0x80370102
+      Error: 0x80370102 The virtual machine could not be started because a required feature is not installed.
+
+   One thing not mentioned in the instructions is to check that the hypervisor is set to auto-start at system boot.
+   This is the default but for various reasons it can get disabled.
+
+   To check, type::
+   
+      bcdedit
+
+   At an administrative command prompt. Under the ``Windows Boot Loader`` entry you should see an entry like this::
+
+      hypervisorlaunchtype    Auto
+
+   If it's missing or set to another value (e.g. ``off``) then change it as follows::
+
+      bcdedit /set {current} hypervisorlaunchtype auto
+
+   After a system reboot you should be able to continue with the installation.
+
+
+
+Installing Sming
+----------------
+
+Open a WSL command prompt and follow the :doc:`Linux installation instructions <../linux>`.
+
+
+Flashing devices
+----------------
+
+WSL2 does not currently support access to USB serial devices, so the Sming build system incorporates a workaround
+which runs the appropriate application (esptool) directly under Windows (via powershell).
+
+Therefore, use the normal Windows COM port name rather than the linux ones (such as /dev/ttyUSB0).
+
+For example::
+
+   make flash COM_PORT=COM4
+
+
+Valgrind
+--------
+
+You may get an error running ``make valgrind`` advising that ``libc6-dbg:i386`` be installed. Here's how::
+
+   sudo dpkg --add-architecture i386
+   sudo apt-get update
+   sudo apt-get install libc6-dbg:i386
+