Merge pull request #1 from dabecart/dev

Merge dev to main for alpha release

.gitignore

@@ -0,0 +1,2 @@
+__pycache__/
+dist/

.vscode/launch.json

@@ -0,0 +1,13 @@
+{
+    "version": "0.2.0",
+    "configurations": [
+      {
+        // Press F5 to launch the project.
+        "name": "Python Debugger: Project",
+        "type": "debugpy",
+        "request": "launch",
+        "program": "src/Main.py",
+        "console": "integratedTerminal"
+      }
+    ]
+  }

.vscode/settings.json

@@ -0,0 +1,16 @@
+{
+    "cSpell.languageSettings": [
+        {
+          "languageId": "python",
+          "includeRegExpList": [
+              "/#.*/",
+             "/('''|\"\"\")[^\\1]+?\\1/g",
+              "strings"
+          ]
+        },
+    ],
+
+    "cSpell.words": [
+        "Pixmap"
+    ]
+}

README.md

@@ -1,15 +1,44 @@
-# Verification and Validation Toolkit 
+# <img src="res/creationFiles/Logo.png" width="50"/> Verification and Validation Toolkit 
 
 *By [@dabecart](https://www.dabecart.net/en/).*
 
-An easy to use application to define a series of test cases to verify and validate your project, be it hardware or software. Each test case will be comprised of a call to your tool to test a part of your project. When your tool runs, it's expected to generate some sort of output (at the moment, only console output is supported). On the *V&V toolkit* you will select the "important" parts of this output, and that will be the **result** of the test case. When all test cases are run, you'll be able to generate a report of the device. This will be the expected output to be generated by other copies/versions of your project.
+An easy to use application to define a series of test cases to verify and validate your project, be it hardware or software. 
+
+![](example/images/setupMode.png)
+
+Each test case will be comprised of a call to your tool to test a part of your project. When your tool runs, it's expected to generate some sort of output (at the moment, only console output is supported). On the *V&V toolkit* you will select the "important" parts of this output, and that will be the **result** of the test case. When all test cases are run, you'll be able to generate a report of the device. This will be the expected output to be generated by other copies/versions of your project.
+
+![](example/images/testWindow3.png)
 
 Finally, when you want to automate the V&V process and run these test cases on bulk on other devices/software versions of your project, this toolkit will compare the results of the current run with the original, it will hint at the differences and will generate a report that will be used to validate your new version of software or your newly manufactured copy of your hardware.
 
 Based on this, the program has three modes:
 
-- **SETUP** mode. On setup mode you define all the test cases. You can order them, give them a short description, a category, a number of repetitions, you may enable or disable it.
-- **BUILD** mode. On build mode you can run all test cases at once or one at time. For each one, you will define what conditions are to be satisfied to considerate the result as successful. In example: you may specify that the test is successful if and only if the output when run again is exactly the same; or, you can specify which parts are to be the same. Once done, you'll be able to generate a report of the expected behavior of your device.
-- **TEST** mode. On test mode, all test cases are run and they get filtered by the rules defined on BUILD mode. After that, you may generate a report with the obtained results and it will point out the reasons that justify if each test case is successful or not.
+- <img src="example/images/mode-setup.png" width="20"/> **SETUP** mode. On setup mode you define all the test cases. You can order them, give them a short description, a category, a number of repetitions, you may enable or disable it.
+- <img src="example/images/mode-build.png" width="20"/> **BUILD** mode. On build mode you can run all test cases at once or one at time. For each one, you will define what conditions are to be satisfied to considerate the result as successful. In example: you may specify that the test is successful if and only if the output when run again is exactly the same; or, you can specify which parts are to be the same. Once done, you'll be able to generate a report of the expected behavior of your device.
+- <img src="example/images/mode-test.png" width="20"/> **TEST** mode. On test mode, all test cases are run and they get filtered by the rules defined on BUILD mode. After that, you may generate a report with the obtained results and it will point out the reasons that justify if each test case is successful or not.
+
+# Manual of this program
+
+At the moment, you can check an example with photos and steps taken [here](example/)!
+
+# External libraries
+
+This project uses the following libraries:
+
+- The UI is made with [PyQt6](https://pypi.org/project/PyQt6/), version 6.7.1.
+- For the color scheme: [qdarktheme](https://pyqtdarktheme.readthedocs.io/en/stable/), version 2.1.0.
+- To export test cases to `.xlsl` files I use [openpyxl](https://openpyxl.readthedocs.io/en/stable/), version 3.1.5.
+- To bundle the program as an executable file: [PyInstaller](https://pyinstaller.org/en/stable/), version 6.10.0.
+
+The project was built and tested on Python 3.10.8 64-bits.
+
+# To bundle as an executable
+
+Run the following: 
+
+```
+pyinstaller --onefile -w -i res/Logo.ico -n VVT src/Main.py
+```
 
 *This is a work in progress.*

TODO

@@ -0,0 +1,15 @@
++ Add icons with the especial TrackableIcon class.
++ Add formatting rules to the model excel so that there is some more color and bold stuff.
++ Expand justification sentences.
++ Add time deltas for the results.
++ Add a rotating circle bellow CollapsibleBoxes while running (?)
++ Add help window.
++ Add fields to enter the test fields (name, project...)
+
+- Add undo/redo of single cells in table.
+- Add a loading bar with an stop button to stop the build and test process.
+- Change the save icon (maybe add a dot) when changes are unsaved.
+- Be able to import a folder of scripts and add them automatically to the table.
+- Add autosave feature on temporary files.
+- Add a test run on BUILD mode to test the verification.
+- Add multiple conditions for tests.

example/Fibonacci.py

@@ -0,0 +1,27 @@
+import sys
+
+# Function for nth fibonacci number 
+def fibonacci(n):
+    a = 0
+    b = 1
+
+    # Check is n is less than 0
+    if n < 0:
+        print("Incorrect input")
+
+    # Check is n is equal to 0
+    elif n == 0:
+        return 0
+
+    # Check if n is equal to 1
+    elif n == 1:
+        return b
+    else:
+        for _ in range(1, n):
+            c = a + b
+            a = b
+            b = c
+        return b
+
+if __name__ == "__main__":
+   print(fibonacci(int(sys.argv[1])))

example/README.md

@@ -0,0 +1,116 @@
+# How to use the VVT (Validation and Verification Toolkit) program
+
+## <img src="images/Logo.png" width="50"/> First steps
+
+On execution, you'll see something similar to this.
+
+![Open Window](images/openWindow.png)
+
+From here, you can do several things:
+
+- Start a **new** file on File > New (`Ctrl + N`). 
+- **Open** a saved file from your computer on File > Open (`Ctrl + O`). The saved files have extension `.vvf`.
+- **Import** a test result by clicking File > Import test results. The test results have extension `.vvt`.
+
+If you created a new file or opened an already existing one, you'll be greeted to the [SETUP mode](#setup-mode) window.
+
+If you chose to import a test result, you'll enter [TEST mode](#importing-test-results).
+
+You can also change the **color theme** of the app by clicking Settings > Program settings (`Ctrl + R`) and selecting `Light` mode. 
+
+![Light mode](images/lightmode.png)
+
+
+## <img src="images/mode-setup.png" width="50"/> SETUP Mode 
+
+On **SETUP** mode you can create a list of all the individual tests that will conform your project.
+
+If you chose to open this [example](test.vvf) file, you would see the following: 
+
+![Setup Mode](images/setupMode.png)
+
+On the left table, you'll see the list of tests to run for your project. By clicking on any of the cells, the *Item Details* pane will pop up on the right. You may modify all tests on either the table or the details pane.
+
+Each test has the following fields on this mode:
+
+- `ID`. This is will be the order of execution on [TEST mode](#test-mode). It's always a positive number and each test has a different one.
+- `Name`. A brief description of what this test does or what it will accomplish.
+- `Category`. For an easier arrangement of the tests on the other modes, you may group your tests in categories.
+- `Repetitions`. The number of times this test will be run. This will also be the number of outputs the test will generate.
+- `Enabled`. This specifies if the test will be run on [TEST mode](#test-mode) or not.
+- `Command`. This is the code or command to run on a shell to launch this test. Bear in mind the `cwd` of all commands **will be the directory of the file you opened**.
+
+On this mode you can choose to either add new items by pressing the button `Add items` (`Alt + N`), remove them by first clicking on them and pressing `Remove item` (`Del`) or duplicate items by selecting and then pressing `Alt + D`.
+
+Once done, the next logic step would be to enter [BUILD mode](#build-mode) by clicking its icon on the top right corner.
+
+You may save your progress anytime by clicking File > Save or `Ctrl + S`.
+
+
+
+
+## <img src="images/mode-build.png" width="50"/> BUILD Mode
+
+On **BUILD** mode you specify which conditions are to be met for each test to be satisfactory.
+
+![Build Mode](images/buildWindow.png)
+
+Each test defined on [SETUP mode](#setup-mode), will be shown here on its own *box*. If you open a lot of test, it can get quite cumbersome to traverse the list. That's why you can filter the tests by choosing its category on the combo box on the top right. You may also show or hide disabled tests with the eye button on the right.
+
+If you click on any of the boxes, you'll see something similar to this:
+
+![Box on build mode](images/boxBuildMode.png)
+
+Each box has the following fields:
+- `Input`. It's the same as in [SETUP mode](#setup-mode).
+- `Verification Mode`. At the time, you may choose the following:
+  - `Same output`. The output(s) on TEST mode must be the same as in BUILD mode.
+  - `Conditional output`. You may choose what type of **operation** and **operand** to apply on the output to consider it valid. 
+
+    The following operations are available: `==`, `<>`, `<`, `>`, `<=`, `>=`, `contain` and `not contain`. 
+
+    The program will try to first operate numerically if both the output and operand are numbers. If not, the operations will be carried out as strings.
+- `Iteration output`. You may get the output of the functions by either running all tests (pressing the `Run all` button) or by running each test individually (the green play button on the top right corner of the box). This output is mainly needed for the `Same output` verification mode, but on next versions it will be needed for new incoming modes.
+
+    Each output also stores its `execution time` and `return code`.
+
+All **enabled tests must be run** on [BUILD mode](#build-mode) before passing to [TEST mode](#test-mode). 
+
+If an error were to occur, you'll get a warning window with the error code. Try to get one yourself by enabling the test "4 - Error function".
+
+
+
+## <img src="images/mode-test.png" width="50"/> TEST Mode
+
+On **TEST** mode you will run all tests on sequence and without interruption. This will generate a report you'll be able to export to a save file (`.vvt`) or to an Excel file (`.xlsl`). The last one can be printed or exported to PDF format.
+
+This mode is done so that **all tests have to pass first try**; in other words, there must not be any exception or fatal error, that would invalidate the tests results. If a single test were to fail, you can retry it individually, but would affect the final result.
+
+![Test1](images/testWindow1.png)
+
+Press the `New test` button to start a test. 
+
+![Test2](images/testWindow2.png)
+
+If you want to start another test, first [export it](#exporting-test-results) and then press `Clear test` or press `New test` again (a pop-up will appear).
+
+![Test3](images/testWindow3.png)
+
+Each box will have a symbol indicating the result of the test. An extensive explanation can be found by opening the box. You can also check the result of each iteration by selecting its number of the `Iteration output` combo. The little colored dot on the left of the number indicates the result of the iteration.
+
+![Test4](images/testWindow4.png)
+
+### Importing test results
+
+This mode is read-only. It's only used to visualize on the program a previous test result. It still allows you to export it again, as either a `.vvt` or `.xlsx` file.
+
+## Exporting test results
+
+Once your test has been run, **export the report** (saving won't work!) by pressing File >  Export test results.
+
+Before that you can set some extra fields for the report by clicking on Edit > Project settings (`Alt + .`). This fields are:
+
+- `Name`. Name for the report.
+- `Project` name.
+- `Author` of the VVT project.
+- `Conductor` of the test.

example/Random.py

@@ -0,0 +1,10 @@
+import sys
+import random
+
+if __name__ == "__main__":
+    if len(sys.argv) != 3:
+        print("Error: two integer arguments are needed!")
+        exit(-1)
+
+    val = random.randint(int(sys.argv[1]), int(sys.argv[2]))
+    print(val)

example/results/results.vvt

@@ -0,0 +1 @@
+[{"name": "VVT Test", "project": "VVT", "date": "12/08/2024 15:09:42.953404", "testCount": 2, "author": "dabecart", "conductor": "dabecart"}, [{"id": 0, "name": "Calculate 10th Fibonacci", "category": "Calculations", "repetitions": 1, "enabled": true, "runcode": "python Fibonacci.py 10", "result": [{"output": "55\r\n", "returnCode": 0, "executionTime": 1.2913614999997662, "timeOfExecution": "11/08/2024 20:42:34.467499", "result": 0}], "validationCmd": {"operation": 1, "operator": "contain", "operatorVal": "5"}, "testResult": 1, "testOutput": [{"output": "55\r\n", "returnCode": 0, "executionTime": 1.2908721000130754, "timeOfExecution": "12/08/2024 15:09:42.982955", "result": 1}], "wasTestRepeated": 0}, {"id": 1, "name": "Calculate 100th Fibonacci", "category": "Calculations", "repetitions": 2, "enabled": false, "runcode": "python Fibonacci.py 100", "result": [{"output": "354224848179261915075\r\n", "returnCode": 0, "executionTime": 1.2464685999984795, "timeOfExecution": "", "result": 0}, {"output": "354224848179261915075\r\n", "returnCode": 0, "executionTime": 1.2334131000025081, "timeOfExecution": "", "result": 0}], "validationCmd": {"operation": 1, "operator": "<", "operatorVal": "8888aa"}, "testResult": 0, "testOutput": [], "wasTestRepeated": 0}, {"id": 2, "name": "Calculate 200th Fibonacci", "category": "Calculations", "repetitions": 3, "enabled": false, "runcode": "python Fibonacci.py 200", "result": [{"output": "280571172992510140037611932413038677189525\r\n", "returnCode": 0, "executionTime": 1.2717756000001827, "timeOfExecution": "", "result": 0}, {"output": "280571172992510140037611932413038677189525\r\n", "returnCode": 0, "executionTime": 1.2711927000000287, "timeOfExecution": "", "result": 0}, {"output": "280571172992510140037611932413038677189525\r\n", "returnCode": 0, "executionTime": 1.2579394000003958, "timeOfExecution": "", "result": 0}], "validationCmd": {"operation": 0, "operator": "==", "operatorVal": ""}, "testResult": 0, "testOutput": [], "wasTestRepeated": 0}, {"id": 3, "name": "Random integer from -10 to 10", "category": "Random", "repetitions": 3, "enabled": true, "runcode": "python Random.py -10 10", "result": [{"output": "-2\r\n", "returnCode": 0, "executionTime": 1.293488699997397, "timeOfExecution": "11/08/2024 20:42:35.761405", "result": 0}, {"output": "-7\r\n", "returnCode": 0, "executionTime": 1.3129139000011492, "timeOfExecution": "11/08/2024 20:42:37.055391", "result": 0}, {"output": "7\r\n", "returnCode": 0, "executionTime": 1.2862715999981447, "timeOfExecution": "11/08/2024 20:42:38.368194", "result": 0}], "validationCmd": {"operation": 1, "operator": ">", "operatorVal": "0"}, "testResult": 4, "testOutput": [{"output": "-1\r\n", "returnCode": 0, "executionTime": 1.2555126999941422, "timeOfExecution": "12/08/2024 15:09:44.320151", "result": 2}, {"output": "-9\r\n", "returnCode": 0, "executionTime": 1.2387114000011934, "timeOfExecution": "12/08/2024 15:09:45.575924", "result": 2}, {"output": "10\r\n", "returnCode": 0, "executionTime": 1.2790373000025284, "timeOfExecution": "12/08/2024 15:09:46.814773", "result": 1}], "wasTestRepeated": 0}, {"id": 4, "name": "Error function", "category": "Error", "repetitions": 1, "enabled": false, "runcode": "python error", "result": [], "validationCmd": {"operation": 0, "operator": "==", "operatorVal": ""}, "testResult": 0, "testOutput": [], "wasTestRepeated": 0}]]

example/test.vvf

@@ -0,0 +1 @@
+[{"name": "VVT Test", "project": "VVT", "date": "12/08/2024 19:05:49.319509", "testCount": 4, "author": "dabecart", "conductor": "dabecart"}, [{"id": 0, "name": "Calculate 10th Fibonacci", "category": "Calculations", "repetitions": 1, "enabled": true, "runcode": "python Fibonacci.py 10", "result": [{"output": "55\r\n", "returnCode": 0, "executionTime": 1.2913614999997662, "timeOfExecution": "11/08/2024 20:42:34.467499", "result": 0}], "validationCmd": {"operation": 1, "operator": "contain", "operatorVal": "5"}, "wasTestRepeated": 0}, {"id": 1, "name": "Calculate 100th Fibonacci", "category": "Calculations", "repetitions": 2, "enabled": true, "runcode": "python Fibonacci.py 100", "result": [{"output": "354224848179261915075\r\n", "returnCode": 0, "executionTime": 1.2464685999984795, "timeOfExecution": "", "result": 0}, {"output": "354224848179261915075\r\n", "returnCode": 0, "executionTime": 1.2334131000025081, "timeOfExecution": "", "result": 0}], "validationCmd": {"operation": 1, "operator": "<", "operatorVal": "8888aa"}, "wasTestRepeated": 0}, {"id": 2, "name": "Calculate 200th Fibonacci", "category": "Calculations", "repetitions": 3, "enabled": true, "runcode": "python Fibonacci.py 200", "result": [{"output": "280571172992510140037611932413038677189525\r\n", "returnCode": 0, "executionTime": 1.2717756000001827, "timeOfExecution": "", "result": 0}, {"output": "280571172992510140037611932413038677189525\r\n", "returnCode": 0, "executionTime": 1.2711927000000287, "timeOfExecution": "", "result": 0}, {"output": "280571172992510140037611932413038677189525\r\n", "returnCode": 0, "executionTime": 1.2579394000003958, "timeOfExecution": "", "result": 0}], "validationCmd": {"operation": 1, "operator": "<", "operatorVal": "5"}, "wasTestRepeated": 0}, {"id": 3, "name": "Random integer from -10 to 10", "category": "Random", "repetitions": 3, "enabled": true, "runcode": "python Random.py -10 10", "result": [{"output": "-2\r\n", "returnCode": 0, "executionTime": 1.293488699997397, "timeOfExecution": "11/08/2024 20:42:35.761405", "result": 0}, {"output": "-7\r\n", "returnCode": 0, "executionTime": 1.3129139000011492, "timeOfExecution": "11/08/2024 20:42:37.055391", "result": 0}, {"output": "7\r\n", "returnCode": 0, "executionTime": 1.2862715999981447, "timeOfExecution": "11/08/2024 20:42:38.368194", "result": 0}], "validationCmd": {"operation": 1, "operator": ">", "operatorVal": "0"}, "wasTestRepeated": 0}, {"id": 4, "name": "Error function", "category": "Error", "repetitions": 1, "enabled": false, "runcode": "python error", "result": [], "validationCmd": {"operation": 0, "operator": "==", "operatorVal": ""}, "wasTestRepeated": 0}]]