Fuzzer coverage statistics / elm-explorations/test 2.0.0 (#614)

Co-authored-by: Simon Lydell <[email protected]>
rtfeldman · Oct 11, 2022 · 84cc928 · 84cc928
1 parent 83f1d84
commit 84cc928
Show file tree

Hide file tree

Showing 57 changed files with 833 additions and 438 deletions.
diff --git a/.gitignore b/.gitignore
@@ -11,7 +11,7 @@ pids
 lib-cov
 
 # Coverage directory used by tools like istanbul
-coverage
+/coverage
 
 # Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)
 .grunt

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,7 +2,23 @@
 
 Notable changes to this project will be documented in this file.
 
-The format is based on [Keep a Changelog](http://keepachangelog.com/). This project mirrors major Elm versions. So version 0.18.\* of this project will be compatible with Elm 0.18.\*.
+The format is based on [Keep a Changelog](http://keepachangelog.com/). This project mirrors the Elm version. So version 0.19.1-revisionX of this project will be compatible with Elm 0.19.1.
+
+## 0.19.1-revision10 - 2022-10-11
+
+⚠️ Updating to this version also requires upgrading to the just released [elm-explorations/test 2.0.0](https://github.com/elm-explorations/test/blob/master/CHANGELOG.md#changes-in-200)!
+
+### Breaking
+
+- This version of `elm-test` _only_ works with elm-explorations/test 2.x (2.0.0 <= v < 3.0.0), which was just released. The [versions table](https://github.com/rtfeldman/node-test-runner#versions) should make it clear which versions work together.
+
+- Removed `elm-test install-unstable-test-master` and `elm-test uninstall-unstable-test-master`. They are no longer needed since elm-explorations/test 2.0.0 has been released.
+
+### Added
+
+- Fuzzer distribution statistics report. [Fuzzer distribution](https://github.com/elm-explorations/test/blob/master/CHANGELOG.md#2-testdistribution) is new in elm-explorations/test 2.0.0, and it required a change in how things are reported from Elm to the test runner, which is why elm-test 0.19.1-revision10 is not compatible with older elm-explorations/test versions.
+
+- The `--no-clear-console` flag. By default, `elm-test --watch` clears the screen on every re-run, so you only see up-to-date output. With `--no-clear-console`, the console is _not_ cleared and a separator is instead printed between the old and new output instead (similar to how [elm-test-rs](https://github.com/mpizenberg/elm-test-rs) works). This is useful if you are running several commands in the same terminal and don’t want `elm-test --watch` to clear away output from other commands.
 
 ## 0.19.1-revision9 - 2022-07-03
 
@@ -19,7 +35,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/). This proj
 ### Added
 
 - `elm-test install-unstable-test-master`
-  - which installs the `master` version of the [elm-explorations/test library](https://github.com/elm-explorations/test/) in place of the `1.2.2` version in your `ELM_HOME`
+  - which installs the `master` version of the [elm-explorations/test] library in place of the `1.2.2` version in your `ELM_HOME`
 - `elm-test uninstall-unstable-test-master`
   - which undoes that
 

diff --git a/README.md b/README.md
@@ -9,6 +9,21 @@ When people say “elm-test” they usually refer to either:
 
 [elm-explorations/test]: https://package.elm-lang.org/packages/elm-explorations/test/latest
 
+## Versions
+
+Not all versions of [elm-explorations/test] and this CLI tool work together!
+
+| elm-explorations/test | elm-test CLI         |
+| --------------------- | -------------------- |
+| >= 2.0.0              | >= 0.19.1-revision10 |
+| <= 1.2.2              | <= 0.19.1-revision9  |
+
+> **Unfortunate behavior of 0.19.1-revision9 and older**
+>
+> - `elm-test init` always installs the latest [elm-explorations/test]. This means that if you run `elm-test init` on version 0.19.1-revision9 or older, you will get elm-explorations/test 2.0.0 or later, which don’t work 100 % together (see the next point).
+> - elm-test 0.19.1-revision9 or older do _not_ validate that [elm-explorations/test] in your elm.json has a compatible version. If you upgrade to elm-explorations/test 2.0.0 or later but forget to upgrade the elm-test CLI, most things will still work, but test distribution diagrams (new in elm-explorations/test 2.0.0) won’t show up. So if you use `Test.fuzzWith` and wonder why distribution diagrams never show up – check your elm-test CLI version!
+> - There exists an elm-test CLI version called just "0.19.1". It should have been called "0.19.1-revision1", but unfortunately isn’t. Don’t make the mistake thinking it’s the latest version! You always want "0.19.1-revisionX".
+
 ## Installation
 
 ```

diff --git a/elm/elm.json b/elm/elm.json
@@ -9,17 +9,17 @@
             "elm/core": "1.0.5",
             "elm/json": "1.1.3",
             "elm/random": "1.0.0",
-            "elm/time": "1.0.0"
-        },
-        "indirect": {}
-    },
-    "test-dependencies": {
-        "direct": {
-            "elm-explorations/test": "1.2.0"
+            "elm/time": "1.0.0",
+            "elm-explorations/test": "2.0.0"
         },
         "indirect": {
+            "elm/bytes": "1.0.8",
             "elm/html": "1.0.0",
-            "elm/virtual-dom": "1.0.2"
+            "elm/virtual-dom": "1.0.3"
         }
+    },
+    "test-dependencies": {
+        "direct": {},
+        "indirect": {}
     }
 }
diff --git a/elm/review/elm.json b/elm/review/elm.json
@@ -27,9 +27,7 @@
         }
     },
     "test-dependencies": {
-        "direct": {
-            "elm-explorations/test": "1.2.2"
-        },
+        "direct": {},
         "indirect": {}
     }
 }
diff --git a/elm/src/Test/Reporter/Console.elm b/elm/src/Test/Reporter/Console.elm
@@ -2,6 +2,7 @@ module Test.Reporter.Console exposing (reportBegin, reportComplete, reportSummar
 
 import Console.Text as Text exposing (..)
 import Json.Encode as Encode exposing (Value)
+import Test.Distribution exposing (DistributionReport)
 import Test.Reporter.Console.Format exposing (format)
 import Test.Reporter.Console.Format.Color as FormatColor
 import Test.Reporter.Console.Format.Monochrome as FormatMonochrome
@@ -35,6 +36,19 @@ pluralize singular plural count =
     String.join " " [ String.fromInt count, suffix ]
 
 
+passedToText : List String -> String -> Text
+passedToText labels distributionReport =
+    Text.concat
+        [ passedLabelsToText labels
+        , dark <| plain <| "\n" ++ indent distributionReport ++ "\n\n"
+        ]
+
+
+passedLabelsToText : List String -> Text
+passedLabelsToText =
+    formatLabels (dark << plain << withChar '↓') (green << withChar '✓') >> Text.concat
+
+
 todosToText : ( List String, String ) -> Text
 todosToText ( labels, failure ) =
     Text.concat [ todoLabelsToText labels, todoToChalk failure ]
@@ -50,7 +64,7 @@ todoToChalk message =
     plain ("◦ TODO: " ++ message ++ "\n\n")
 
 
-failuresToText : UseColor -> List String -> List Failure -> Text
+failuresToText : UseColor -> List String -> List ( Failure, DistributionReport ) -> Text
 failuresToText useColor labels failures =
     Text.concat (failureLabelsToText labels :: List.map (failureToText useColor) failures)
 
@@ -60,8 +74,8 @@ failureLabelsToText =
     formatLabels (dark << plain << withChar '↓') (red << withChar '✗') >> Text.concat
 
 
-failureToText : UseColor -> Results.Failure -> Text
-failureToText useColor { given, description, reason } =
+failureToText : UseColor -> ( Failure, DistributionReport ) -> Text
+failureToText useColor ( { given, description, reason }, distributionReport ) =
     let
         formatEquality =
             case useColor of
@@ -71,18 +85,23 @@ failureToText useColor { given, description, reason } =
                 UseColor ->
                     FormatColor.formatEquality
 
+        distributionText =
+            distributionReportToString distributionReport
+                |> Maybe.map (\str -> dark (plain ("\n" ++ indent str ++ "\n")))
+
+        givenText =
+            given
+                |> Maybe.map (\str -> dark (plain ("\nGiven " ++ str ++ "\n")))
+
         messageText =
-            plain ("\n" ++ indent (format formatEquality description reason) ++ "\n\n")
+            plain <| "\n" ++ indent (format formatEquality description reason) ++ "\n\n"
     in
-    case given of
-        Nothing ->
-            messageText
-
-        Just givenStr ->
-            [ dark (plain ("\nGiven " ++ givenStr ++ "\n"))
-            , messageText
-            ]
-                |> Text.concat
+    [ distributionText
+    , givenText
+    , Just messageText
+    ]
+        |> List.filterMap identity
+        |> Text.concat
 
 
 textToValue : UseColor -> Text -> Value
@@ -103,30 +122,64 @@ reportBegin useColor { globs, fuzzRuns, testCount, initialSeed } =
                 ++ " --seed "
                 ++ String.fromInt initialSeed
     in
-    (String.join " " (prefix :: globs) ++ "\n")
-        |> plain
-        |> textToValue useColor
+    Encode.object
+        [ ( "type", Encode.string "begin" )
+        , ( "output"
+          , (String.join " " (prefix :: globs) ++ "\n")
+                |> plain
+                |> textToValue useColor
+          )
+        ]
         |> Just
 
 
-reportComplete : UseColor -> Results.TestResult -> Value
-reportComplete useColor { labels, outcome } =
+getStatus : Outcome -> String
+getStatus outcome =
     case outcome of
-        Passed ->
-            -- No failures of any kind.
-            Encode.null
-
-        Failed failures ->
-            -- We have non-TODOs still failing; report them, not the TODOs.
-            failures
-                |> failuresToText useColor labels
-                |> textToValue useColor
+        Failed _ ->
+            "fail"
 
-        Todo str ->
-            Encode.object
-                [ ( "todo", Encode.string str )
-                , ( "labels", Encode.list Encode.string labels )
-                ]
+        Todo _ ->
+            "todo"
+
+        Passed _ ->
+            "pass"
+
+
+reportComplete : UseColor -> Results.TestResult -> Value
+reportComplete useColor { labels, outcome } =
+    Encode.object <|
+        ( "type", Encode.string "complete" )
+            :: ( "status", Encode.string (getStatus outcome) )
+            :: (case outcome of
+                    Passed distributionReport ->
+                        -- No failures of any kind.
+                        case distributionReportToString distributionReport of
+                            Nothing ->
+                                []
+
+                            Just report ->
+                                [ ( "distributionReport"
+                                  , report
+                                        |> passedToText labels
+                                        |> textToValue useColor
+                                  )
+                                ]
+
+                    Failed failures ->
+                        [ ( "failure"
+                          , -- We have non-TODOs still failing; report them, not the TODOs.
+                            failures
+                                |> failuresToText useColor labels
+                                |> textToValue useColor
+                          )
+                        ]
+
+                    Todo str ->
+                        [ ( "todo", Encode.string str )
+                        , ( "labels", Encode.list Encode.string labels )
+                        ]
+               )
 
 
 summarizeTodos : List ( List String, String ) -> Text
@@ -182,16 +235,21 @@ reportSummary useColor { todos, passed, failed, duration } autoFail =
             else
                 summarizeTodos (List.reverse todos)
     in
-    [ headline
-    , stat "Duration: " (formatDuration duration)
-    , stat "Passed:   " (String.fromInt passed)
-    , stat "Failed:   " (String.fromInt failed)
-    , todoStats
-    , individualTodos
-    ]
-        |> Text.concat
-        |> Text.render useColor
-        |> Encode.string
+    Encode.object
+        [ ( "type", Encode.string "summary" )
+        , ( "summary"
+          , [ headline
+            , stat "Duration: " (formatDuration duration)
+            , stat "Passed:   " (String.fromInt passed)
+            , stat "Failed:   " (String.fromInt failed)
+            , todoStats
+            , individualTodos
+            ]
+                |> Text.concat
+                |> Text.render useColor
+                |> Encode.string
+          )
+        ]
 
 
 stat : String -> String -> Text
@@ -205,3 +263,22 @@ stat label value =
 withChar : Char -> String -> String
 withChar icon str =
     String.fromChar icon ++ " " ++ str ++ "\n"
+
+
+distributionReportToString : DistributionReport -> Maybe String
+distributionReportToString distributionReport =
+    case distributionReport of
+        Test.Distribution.NoDistribution ->
+            Nothing
+
+        Test.Distribution.DistributionToReport r ->
+            Just (Test.Distribution.distributionReportTable r)
+
+        Test.Distribution.DistributionCheckSucceeded _ ->
+            {- Not reporting the table although the data is technically there.
+               We keep the full data dump for the JSON reporter.
+            -}
+            Nothing
+
+        Test.Distribution.DistributionCheckFailed r ->
+            Just (Test.Distribution.distributionReportTable r)