Skip to content

Commit

Permalink
Merge pull request #24 from bsless/sci-comparison
Browse files Browse the repository at this point in the history 
[#23] Rework SCI comparison
  • Loading branch information
@bsless
bsless authored Oct 15, 2021
2 parents c50bf1c + f3af7c8 commit cbb0e08
Show file tree
Hide file tree
Showing 3 changed files with 62 additions and 273 deletions.
53 changes: 21 additions & 32 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -172,45 +172,34 @@ Otherwise, a symbol will be interpreted as an environment lookup.

## Performance vs. SCI

Since contextual's model is compile-once run-many, invoke is
significantly faster than sci's eval:
Contextual optimizes for evaluation and string generation, being a template engine.

```clojure
(require '[sci.core :as sci])

(def scitx (sci/init {}))

(sci/eval-form scitx '(let [x 1 y 2] (+ x y)))
SCI's analysis is faster than Contextual which has several optimization passes.

(def c (-compile '(let [x 1 y 2] (+ x y))))
Besides let forms, Contextual's evaluation is faster.

(-invoke c {})
### Analysis

(require '[criterium.core :as cc])
| Expression | SCI | Contextual |
| (+ 1 2) | 964.928006158732 ns | 2.079940179533958 µs |
| (let [x 1 y 2] (+ x y)) | 4.053300853147959 µs | 10.428633451833264 µs |
| (let [x 1] (let [y 2] (+ x y))) | 5.014739082063033 µs | 15.419620810055868 µs |
| (str 1) | 940.5541048261234 ns | 2.462470343113364 µs |
| (str 1 2 3) | 978.0085949105783 ns | 3.3992453241441143 µs |
| (str 1 2 3 4 5 6 7 8 9 10) | 1.1567249302535807 µs | 6.230790269538975 µs |
| (str 1 (str 2 (str 3))) | 2.7034542882428667 µs | 6.367253313202483 µs |

(cc/quick-bench
(sci/eval-form scitx '(let [x 1 y 2] (+ x y))))
### Evaluation of analyzed expression

;;; Evaluation count : 20016 in 6 samples of 3336 calls.
;;; Execution time mean : 31.412883 µs
;;; Execution time std-deviation : 1.088819 µs
;;; Execution time lower quantile : 30.478367 µs ( 2.5%)
;;; Execution time upper quantile : 33.040048 µs (97.5%)
;;; Overhead used : 9.329803 ns

(cc/quick-bench
(-invoke c {}))

;;; Evaluation count : 543534 in 6 samples of 90589 calls.
;;; Execution time mean : 1.118617 µs
;;; Execution time std-deviation : 37.443201 ns
;;; Execution time lower quantile : 1.088897 µs ( 2.5%)
;;; Execution time upper quantile : 1.179156 µs (97.5%)
;;; Overhead used : 9.414056 ns
```
| Expression | SCI | Contextual |
| (+ 1 2) | 35.082971812986024 ns | 8.572977104636808 ns |
| (let [x 1 y 2] (+ x y)) | 296.1578519129252 ns | 239.01144451640243 ns |
| (let [x 1] (let [y 2] (+ x y))) | 448.73383802345813 ns | 503.64078619481376 ns |
| (str 1) | 25.416337334204073 ns | 19.849291428915592 ns |
| (str 1 2 3) | 117.64023555640503 ns | 45.32831079503547 ns |
| (str 1 2 3 4 5 6 7 8 9 10) | 409.09662405378765 ns | 124.85953596525712 ns |
| (str 1 (str 2 (str 3))) | 187.72209701166938 ns | 50.720662555853266 ns |

In most cases, compiling + invoking contextual code will also be faster
than sci.

## Status

Expand Down
277 changes: 37 additions & 240 deletions dev/contextual/bench.clj
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,10 @@
[contextual.impl.protocols :as p]
[sci.core :as sci]
[criterium.core :as cc]
[sci.impl.analyzer :as ana]
[sci.impl.evaluator :as eval]
[clj-async-profiler.core :as prof]))


(def simple-expr '(+ 1 2))
(def simple-let '(let [x 1 y 2] (+ x y)))
(def nested-let '(let [x 1] (let [y 2] (+ x y))))
Expand All @@ -15,246 +16,42 @@
(def string-10 '(str 1 2 3 4 5 6 7 8 9 10))
(def nested-str3 '(str 1 (str 2 (str 3))))

(def scitx (sci/init {}))

(defn bench-full-vs
[expr]
(println "Benchmarking expr:" expr)
(println "\nContextual")
(cc/quick-bench
(p/-invoke (c/-compile expr) {}))
(println "\nSCI")
(cc/quick-bench
(sci/eval-form scitx expr))
(println))
(def lookup {'+ +})

(defn bench-vs
[expr]
(println "Benchmarking expr:" expr)
(println "\nContextual")
(let [e' (c/-compile expr)]
(cc/quick-bench
(p/-invoke e' {})))
(println "\nSCI")
(cc/quick-bench
(sci/eval-form scitx expr))
(println))

(defn all-benchmarks
[]
(bench-vs simple-expr)
(bench-vs simple-let)
(bench-vs nested-let)
(bench-vs string-1)
(bench-vs string-3)
(bench-vs string-10)
(bench-vs nested-str3)
)
(def scitx (sci/init {}))

(defn all-full-benchmarks
[]
(bench-full-vs simple-expr)
(bench-full-vs simple-let)
(bench-full-vs nested-let)
(bench-full-vs string-1)
(bench-full-vs string-3)
(bench-full-vs string-10)
(bench-full-vs nested-str3)
)
(defn report-mean
[result]
(let [{:keys [mean]} result
[mean] mean
[factor unit] (cc/scale-time mean)]
[(* mean factor) unit]))

(defmacro bench
[& expr]
`(let [res# (cc/quick-benchmark (do ~@expr) nil)]
(report-mean res#)))

(defn bench-analysis [expr]
(println "ANALYSIS:" expr)
(let [[mean unit] (bench (ana/analyze scitx expr))]
(println "SCI:" mean unit))
(let [[mean unit] (bench (c/-compile expr lookup))]
(println "CONTEXTUAL:" mean unit)))

(defn bench-eval [expr]
(println "EVAL:" expr)
(let [ana (ana/analyze scitx expr)
[mean unit] (bench (eval/eval scitx {} ana))]
(println "SCI:" mean unit))
(let [c (c/-compile expr lookup)
[mean unit] (bench (p/-invoke c {}))]
(println "CONTEXTUAL:" mean unit)))

(comment
(all-full-benchmarks)
(all-benchmarks))

;;; Benchmarking expr: (+ 1 2)
;;; Contextual
;;; Evaluation count : 1120118280 in 60 samples of 18668638 calls.
;;; Execution time mean : 44.699888 ns
;;; Execution time std-deviation : 0.898814 ns
;;; Execution time lower quantile : 43.714308 ns ( 2.5%)
;;; Execution time upper quantile : 47.216891 ns (97.5%)
;;; Overhead used : 9.615146 ns
;;;
;;; Found 5 outliers in 60 samples (8.3333 %)
;;; low-severe 3 (5.0000 %)
;;; low-mild 2 (3.3333 %)
;;; Variance from outliers : 9.3559 % Variance is slightly inflated by outliers
;;; SCI
;;; Evaluation count : 6264540 in 60 samples of 104409 calls.
;;; Execution time mean : 9.762819 µs
;;; Execution time std-deviation : 334.990842 ns
;;; Execution time lower quantile : 9.405957 µs ( 2.5%)
;;; Execution time upper quantile : 10.590253 µs (97.5%)
;;; Overhead used : 9.615146 ns
;;;
;;; Found 5 outliers in 60 samples (8.3333 %)
;;; low-severe 4 (6.6667 %)
;;; low-mild 1 (1.6667 %)
;;; Variance from outliers : 20.6283 % Variance is moderately inflated by outliers
;;;
;;; Benchmarking expr: (let [x 1 y 2] (+ x y))
;;; Contextual
;;; Evaluation count : 40425060 in 60 samples of 673751 calls.
;;; Execution time mean : 1.493608 µs
;;; Execution time std-deviation : 28.630845 ns
;;; Execution time lower quantile : 1.462026 µs ( 2.5%)
;;; Execution time upper quantile : 1.571459 µs (97.5%)
;;; Overhead used : 9.615146 ns
;;;
;;; Found 7 outliers in 60 samples (11.6667 %)
;;; low-severe 6 (10.0000 %)
;;; low-mild 1 (1.6667 %)
;;; Variance from outliers : 7.8412 % Variance is slightly inflated by outliers
;;; SCI
;;; Evaluation count : 1895700 in 60 samples of 31595 calls.
;;; Execution time mean : 32.030505 µs
;;; Execution time std-deviation : 673.357839 ns
;;; Execution time lower quantile : 31.339457 µs ( 2.5%)
;;; Execution time upper quantile : 33.727690 µs (97.5%)
;;; Overhead used : 9.615146 ns
;;;
;;; Found 2 outliers in 60 samples (3.3333 %)
;;; low-severe 2 (3.3333 %)
;;; Variance from outliers : 9.4107 % Variance is slightly inflated by outliers
;;;
;;; Benchmarking expr: (let [x 1] (let [y 2] (+ x y)))
;;; Contextual
;;; Evaluation count : 31178340 in 60 samples of 519639 calls.
;;; Execution time mean : 1.948790 µs
;;; Execution time std-deviation : 33.639915 ns
;;; Execution time lower quantile : 1.907390 µs ( 2.5%)
;;; Execution time upper quantile : 2.034007 µs (97.5%)
;;; Overhead used : 9.615146 ns
;;;
;;; Found 5 outliers in 60 samples (8.3333 %)
;;; low-severe 5 (8.3333 %)
;;; Variance from outliers : 6.2782 % Variance is slightly inflated by outliers
;;; SCI
;;; Evaluation count : 1469340 in 60 samples of 24489 calls.
;;; Execution time mean : 41.373357 µs
;;; Execution time std-deviation : 957.141300 ns
;;; Execution time lower quantile : 40.438712 µs ( 2.5%)
;;; Execution time upper quantile : 44.146026 µs (97.5%)
;;; Overhead used : 9.615146 ns
;;;
;;; Found 5 outliers in 60 samples (8.3333 %)
;;; low-severe 2 (3.3333 %)
;;; low-mild 3 (5.0000 %)
;;; Variance from outliers : 11.0044 % Variance is moderately inflated by outliers
;;;
;;; Benchmarking expr: (str 1)
;;; Contextual
;;; Evaluation count : 417181620 in 60 samples of 6953027 calls.
;;; Execution time mean : 135.819063 ns
;;; Execution time std-deviation : 3.508690 ns
;;; Execution time lower quantile : 132.156693 ns ( 2.5%)
;;; Execution time upper quantile : 142.767954 ns (97.5%)
;;; Overhead used : 9.615146 ns
;;;
;;; Found 2 outliers in 60 samples (3.3333 %)
;;; low-severe 2 (3.3333 %)
;;; Variance from outliers : 12.6396 % Variance is moderately inflated by outliers
;;; SCI
;;; Evaluation count : 6627720 in 60 samples of 110462 calls.
;;; Execution time mean : 9.274174 µs
;;; Execution time std-deviation : 267.538610 ns
;;; Execution time lower quantile : 9.015513 µs ( 2.5%)
;;; Execution time upper quantile : 9.991612 µs (97.5%)
;;; Overhead used : 9.615146 ns
;;;
;;; Found 9 outliers in 60 samples (15.0000 %)
;;; low-severe 4 (6.6667 %)
;;; low-mild 5 (8.3333 %)
;;; Variance from outliers : 15.7974 % Variance is moderately inflated by outliers
;;;
;;; Benchmarking expr: (str 1 2 3)
;;; Contextual
;;; Evaluation count : 282818520 in 60 samples of 4713642 calls.
;;; Execution time mean : 206.056654 ns
;;; Execution time std-deviation : 2.778767 ns
;;; Execution time lower quantile : 202.658145 ns ( 2.5%)
;;; Execution time upper quantile : 211.704606 ns (97.5%)
;;; Overhead used : 9.615146 ns
;;;
;;; Found 2 outliers in 60 samples (3.3333 %)
;;; low-severe 2 (3.3333 %)
;;; Variance from outliers : 1.6389 % Variance is slightly inflated by outliers
;;; SCI
;;; Evaluation count : 5991120 in 60 samples of 99852 calls.
;;; Execution time mean : 10.084744 µs
;;; Execution time std-deviation : 287.457252 ns
;;; Execution time lower quantile : 9.754388 µs ( 2.5%)
;;; Execution time upper quantile : 10.832135 µs (97.5%)
;;; Overhead used : 9.615146 ns
;;;
;;; Found 5 outliers in 60 samples (8.3333 %)
;;; low-severe 2 (3.3333 %)
;;; low-mild 3 (5.0000 %)
;;; Variance from outliers : 15.7763 % Variance is moderately inflated by outliers
;;;
;;; Benchmarking expr: (str 1 2 3 4 5 6 7 8 9 10)
;;; Contextual
;;; Evaluation count : 126166620 in 60 samples of 2102777 calls.
;;; Execution time mean : 472.380161 ns
;;; Execution time std-deviation : 10.396655 ns
;;; Execution time lower quantile : 461.260238 ns ( 2.5%)
;;; Execution time upper quantile : 500.942020 ns (97.5%)
;;; Overhead used : 9.615146 ns
;;;
;;; Found 8 outliers in 60 samples (13.3333 %)
;;; low-severe 3 (5.0000 %)
;;; low-mild 5 (8.3333 %)
;;; Variance from outliers : 9.4624 % Variance is slightly inflated by outliers
;;; SCI
;;; Evaluation count : 4949760 in 60 samples of 82496 calls.
;;; Execution time mean : 12.283282 µs
;;; Execution time std-deviation : 269.201358 ns
;;; Execution time lower quantile : 11.971501 µs ( 2.5%)
;;; Execution time upper quantile : 12.884596 µs (97.5%)
;;; Overhead used : 9.615146 ns
;;;
;;; Found 7 outliers in 60 samples (11.6667 %)
;;; low-severe 6 (10.0000 %)
;;; low-mild 1 (1.6667 %)
;;; Variance from outliers : 9.4578 % Variance is slightly inflated by outliers
;;;
;;; Benchmarking expr: (str 1 (str 2 (str 3)))
;;; Contextual
;;; Evaluation count : 220127880 in 60 samples of 3668798 calls.
;;; Execution time mean : 267.387943 ns
;;; Execution time std-deviation : 4.567354 ns
;;; Execution time lower quantile : 259.143600 ns ( 2.5%)
;;; Execution time upper quantile : 276.919611 ns (97.5%)
;;; Overhead used : 9.615146 ns
;;;
;;; Found 3 outliers in 60 samples (5.0000 %)
;;; low-severe 3 (5.0000 %)
;;; Variance from outliers : 6.2700 % Variance is slightly inflated by outliers
;;; SCI
;;; Evaluation count : 2583300 in 60 samples of 43055 calls.
;;; Execution time mean : 23.643595 µs
;;; Execution time std-deviation : 479.989842 ns
;;; Execution time lower quantile : 23.165250 µs ( 2.5%)
;;; Execution time upper quantile : 24.715076 µs (97.5%)
;;; Overhead used : 9.615146 ns
;;;
;;; Found 5 outliers in 60 samples (8.3333 %)
;;; low-severe 3 (5.0000 %)
;;; low-mild 2 (3.3333 %)
;;; Variance from outliers : 9.3681 % Variance is slightly inflated by outliers

(comment)

;;; Machine

;;; OS: Ubuntu 20.04 focal
;;; Kernel: x86_64 Linux 5.4.0-52-generic
;;; CPU: Intel Core i7-6820HQ @ 8x 3.6GHz [60.0°C]

(comment)

;;; Java

;;; openjdk version "1.8.0_272"
;;; OpenJDK Runtime Environment (build 1.8.0_272-8u272-b10-0ubuntu1~20.04-b10)
;;; OpenJDK 64-Bit Server VM (build 25.272-b10, mixed mode)
(bench-analysis simple-expr)
(bench-eval simple-expr)
(doseq [expr [simple-expr simple-let nested-let string-1 string-3 string-10 nested-str3]]
(bench-analysis expr))
(doseq [expr [simple-expr simple-let nested-let string-1 string-3 string-10 nested-str3]]
(bench-eval expr)))
5 changes: 4 additions & 1 deletion project.clj
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,11 @@
["snapshots" :clojars]]
:dependencies [[org.clojure/clojure "1.10.1"]]
:profiles {:dev {:source-paths ["./dev"]
:jvm-opts ["-Djdk.attach.allowAttachSelf"
"-XX:+UnlockDiagnosticVMOptions"
"-XX:+DebugNonSafepoints"]
:dependencies [[org.clojure/tools.trace "0.7.10"]
[criterium "0.4.6"]
[com.clojure-goes-fast/clj-async-profiler "0.4.1"]
[borkdude/sci "0.1.1-alpha.9"]]}}
[borkdude/sci "0.2.6"]]}}
:repl-options {:init-ns contextual.core})

0 comments on commit cbb0e08

Please sign in to comment.