Skip to content

Commit c413e44

Browse files
jconningtreeder
authored andcommitted
Test harness to assess whether fnlb works properly (#573)
* Initial commit. * Update README.md * Update README.md * Update README.md. * Update README.md * Changes from PR code review.
1 parent d1ea037 commit c413e44

File tree

4 files changed

+280
-0
lines changed

4 files changed

+280
-0
lines changed

test/fnlb-test-harness/README.md

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# fnlb-test-harness
2+
Test harness that exercises the fnlb load balancer in order to verify that it works properly.
3+
## How it works
4+
This is a test harness that makes calls to an IronFunctions route through the fnlb load balancer, which routes traffic to multiple IronFunctions nodes.
5+
The test harness keeps track of which node each request was routed to so we can assess how the requests are being distributed across the nodes. The functionality
6+
of fnlb is to normally route traffic to the same small number of nodes so that efficiences can be achieved and to support reuse of hot functions.
7+
### Primes function
8+
The test harness utilizes the "primes" function, which calculates prime numbers as an excuse for consuming CPU resources. The function is invoked as follows:
9+
```
10+
curl http://host:8080/r/primesapp/primes?max=1000000&loops=1
11+
```
12+
where:
13+
- *max*: calculate all primes <= max (increasing max will increase memory usage, due to the Sieve of Eratosthenes algorithm)
14+
- *loops*: number of times to calculate the primes (repeating the count consumes additional CPU without consuming additional memory)
15+
16+
## How to use it
17+
The test harness requires running one or more IronFunctions nodes and one instance of fnlb. The list of nodes must be provided both to fnlb and to the test harness
18+
because the test harness must call each node directly one time in order to discover the node's container id.
19+
20+
After it has run, examine the results to see how the requests were distributed across the nodes.
21+
### How to run it locally
22+
Each of the IronFunctions nodes needs to connect to the same database.
23+
24+
STEP 1: Create a route for the primes function. Example:
25+
```
26+
fn apps create primesapp
27+
fn routes create primesapp /primes jconning/primes:0.0.1
28+
```
29+
STEP 2: Run five IronFunctions nodes locally. Example (runs five nodes in the background using Docker):
30+
```
31+
sudo docker run -d -it --name functions-8082 --privileged -v ${HOME}/data-8082:/app/data -p 8082:8080 -e "DB_URL=postgres://dbUser:dbPassword@dbHost:5432/dbName" iron/functions
32+
sudo docker run -d -it --name functions-8083 --privileged -v ${HOME}/data-8083:/app/data -p 8083:8080 -e "DB_URL=postgres://dbUser:dbPassword@dbHost:5432/dbName" iron/functions
33+
sudo docker run -d -it --name functions-8084 --privileged -v ${HOME}/data-8084:/app/data -p 8084:8080 -e "DB_URL=postgres://dbUser:dbPassword@dbHost:5432/dbName" iron/functions
34+
sudo docker run -d -it --name functions-8085 --privileged -v ${HOME}/data-8085:/app/data -p 8085:8080 -e "DB_URL=postgres://dbUser:dbPassword@dbHost:5432/dbName" iron/functions
35+
sudo docker run -d -it --name functions-8086 --privileged -v ${HOME}/data-8086:/app/data -p 8086:8080 -e "DB_URL=postgres://dbUser:dbPassword@dbHost:5432/dbName" iron/functions
36+
```
37+
STEP 3: Run fnlb locally. Example (runs fnlb on the default port 8081):
38+
```
39+
fnlb -nodes localhost:8082,localhost:8083,localhost:8084,localhost:8085,localhost:8086
40+
```
41+
STEP 4: Run the test harness. Note that the 'nodes' parameter should be the same that was used with fnlb. Example:
42+
```
43+
cd functions/test/fnlb-test-harness
44+
go run main.go -nodes localhost:8082,localhost:8083,localhost:8084,localhost:8085,localhost:8086 -calls 10 -v
45+
```
46+
STEP 5: Examine the output to determine how many times fnlb called each node. Assess whether it is working properly.
47+
48+
### Usage
49+
go run main.go -help
50+
51+
<i>Command line parameters:</i>
52+
- *-calls*: number of times to call the route (default 100)
53+
- *-lb*: host and port of load balancer (default "localhost:8081")
54+
- *-loops*: number of times to execute the primes calculation (ex: '-loops 2' means run the primes calculation twice) (default 1)
55+
- *-max*: maximum number to search for primes (higher number consumes more memory) (default 1000000)
56+
- *-nodes*: comma-delimited list of nodes (host:port) balanced by the load balancer (needed to discover container id of each) (default "localhost:8080")
57+
- *-route*: path representing the route to the primes function (default "/r/primesapp/primes")
58+
- *-v*: flag indicating verbose output
59+
60+
### Examples: quick vs long running
61+
62+
**Quick function:**: calculate primes up to 1000
63+
```
64+
go run main.go -nodes localhost:8082,localhost:8083,localhost:8084,localhost:8085,localhost:8086 -max 1000 -v
65+
```
66+
where *-max* is default of 1M, *-calls* is default of 100, *-route* is default of "/r/primesapp/primes", *-lb* is default localhost:8081
67+
68+
**Normal function**: calculate primes up to 1M
69+
```
70+
go run main.go -nodes localhost:8082,localhost:8083,localhost:8084,localhost:8085,localhost:8086 -v
71+
```
72+
where *-max* is default of 1M, *-calls* is default of 100, *-route* is default of "/r/primesapp/primes", *-lb* is default localhost:8081
73+
74+
**Longer running function**: calculate primes up to 1M and perform the calculation ten times
75+
```
76+
go run main.go -nodes localhost:8082,localhost:8083,localhost:8084,localhost:8085,localhost:8086 -loops 10 -v
77+
```
78+
where *-max* is default of 1M, *-calls* is default of 100, *-route* is default of "/r/primesapp/primes", *-lb* is default localhost:8081
79+
80+
**1000 calls to the route**: send 1000 requests through the load balancer
81+
```
82+
go run main.go -nodes localhost:8082,localhost:8083,localhost:8084,localhost:8085,localhost:8086 -calls 1000 -v
83+
```
84+
where *-max* is default of 1M, *-calls* is default of 100, *-route* is default of "/r/primesapp/primes", *-lb* is default localhost:8081
85+
86+
## Planned Enhancements
87+
- Create 1000 routes and distribute calls amongst them.
88+
- Use concurrent programming to enable the test harness to call multiple routes at the same time.

test/fnlb-test-harness/main.go

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
package main
2+
3+
import (
4+
"fmt"
5+
"net/http"
6+
"io/ioutil"
7+
"encoding/json"
8+
"time"
9+
"flag"
10+
"log"
11+
"strings"
12+
)
13+
14+
type execution struct {
15+
DurationSeconds float64
16+
Hostname string
17+
node string
18+
body string
19+
responseSeconds float64
20+
}
21+
22+
var (
23+
lbHostPort, nodesStr, route string
24+
numExecutions, maxPrime, numLoops int
25+
nodes []string
26+
nodesByContainerId map[string]string = make(map[string]string)
27+
verbose bool
28+
)
29+
30+
func init() {
31+
flag.StringVar(&lbHostPort, "lb", "localhost:8081", "host and port of load balancer")
32+
flag.StringVar(&nodesStr, "nodes", "localhost:8080", "comma-delimited list of nodes (host:port) balanced by the load balancer (needed to discover container id of each)")
33+
flag.StringVar(&route, "route", "/r/primesapp/primes", "path representing the route to the primes function")
34+
flag.IntVar(&numExecutions, "calls", 100, "number of times to call the route")
35+
flag.IntVar(&maxPrime, "max", 1000000, "maximum number to search for primes (higher number consumes more memory)")
36+
flag.IntVar(&numLoops, "loops", 1, "number of times to execute the primes calculation (ex: 'loops=2' means run the primes calculation twice)")
37+
flag.BoolVar(&verbose, "v", false, "true for more verbose output")
38+
flag.Parse()
39+
40+
if maxPrime < 3 {
41+
log.Fatal("-max must be 3 or greater")
42+
}
43+
if numLoops < 1 {
44+
log.Fatal("-loops must be 1 or greater")
45+
}
46+
47+
nodes = strings.Split(nodesStr, ",")
48+
}
49+
50+
func executeFunction(hostPort, path string, max, loops int) (execution, error) {
51+
var e execution
52+
53+
start := time.Now()
54+
resp, err := http.Get(fmt.Sprintf("http://%s%s?max=%d&loops=%d", hostPort, path, max, loops))
55+
e.responseSeconds = time.Since(start).Seconds()
56+
if err != nil {
57+
return e, err
58+
}
59+
defer resp.Body.Close()
60+
if resp.StatusCode != http.StatusOK {
61+
return e, fmt.Errorf("function returned status code: %d", resp.StatusCode)
62+
}
63+
64+
body, err := ioutil.ReadAll(resp.Body)
65+
if err != nil {
66+
return e, err
67+
}
68+
69+
err = json.Unmarshal(body, &e)
70+
if err != nil {
71+
e.body = string(body) // set the body in the execution so that it is available for logging
72+
return e, err
73+
}
74+
e.node = nodesByContainerId[e.Hostname]
75+
76+
return e, nil
77+
}
78+
79+
func invokeLoadBalancer(hostPort, path string, numExecutions, max, loops int) {
80+
executionsByNode := make(map[string][]execution)
81+
fmt.Printf("All primes will be calculated up to %d, a total of %d time(s)\n", maxPrime, numLoops)
82+
fmt.Printf("Calling route %s %d times (through the load balancer)...\n", route, numExecutions)
83+
84+
for i := 0; i < numExecutions; i++ {
85+
e, err := executeFunction(hostPort, path, max, loops)
86+
if err == nil {
87+
if ex, ok := executionsByNode[e.node]; ok {
88+
executionsByNode[e.node] = append(ex, e)
89+
} else {
90+
// Create a slice to contain the list of executions for this host
91+
executionsByNode[e.node] = []execution{e}
92+
}
93+
if verbose {
94+
fmt.Printf(" %s in-function duration: %fsec, response time: %fsec\n", e.node, e.DurationSeconds, e.responseSeconds)
95+
}
96+
} else {
97+
fmt.Printf(" Ignoring failed execution on node %s: %v\n", e.node, err)
98+
fmt.Printf(" JSON: %s\n", e.body)
99+
}
100+
}
101+
102+
fmt.Println("Results (executions per node):")
103+
for node, ex := range executionsByNode {
104+
fmt.Printf(" %s %d\n", node, len(ex))
105+
}
106+
}
107+
108+
func discoverContainerIds() {
109+
// Discover the Docker hostname of each node; create a mapping of hostnames to host/port.
110+
// This is needed because IronFunctions doesn't make the host/port available to the function (as of Mar 2017).
111+
fmt.Println("Discovering container ids for every node (use Docker's HOSTNAME env var as a container id)...")
112+
for _, s := range nodes {
113+
if e, err := executeFunction(s, route, 100, 1); err == nil {
114+
nodesByContainerId[e.Hostname] = s
115+
fmt.Printf(" %s %s\n", s, e.Hostname)
116+
} else {
117+
fmt.Printf(" Ignoring host %s which returned error: %v\n", s, err)
118+
fmt.Printf(" JSON: %s\n", e.body)
119+
}
120+
}
121+
}
122+
123+
func main() {
124+
discoverContainerIds()
125+
invokeLoadBalancer(lbHostPort, route, numExecutions, maxPrime, numLoops)
126+
}
127+
Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
package main
2+
3+
import (
4+
"fmt"
5+
"os"
6+
"strings"
7+
"strconv"
8+
"time"
9+
)
10+
11+
// return list of primes less than N
12+
// source: http://stackoverflow.com/a/21923233
13+
func sieveOfEratosthenes(N int) (primes []int) {
14+
b := make([]bool, N)
15+
for i := 2; i < N; i++ {
16+
if b[i] == true {
17+
continue
18+
}
19+
primes = append(primes, i)
20+
for k := i * i; k < N; k += i {
21+
b[k] = true
22+
}
23+
}
24+
return
25+
}
26+
27+
func main() {
28+
start := time.Now()
29+
maxPrime := 1000000
30+
numLoops := 1
31+
32+
// Parse the query string
33+
s := strings.Split(os.Getenv("REQUEST_URL"), "?")
34+
if len(s) > 1 {
35+
for _, pair := range strings.Split(s[1], "&") {
36+
kv := strings.Split(pair, "=")
37+
if len(kv) > 1 {
38+
key, value := kv[0], kv[1]
39+
if key == "max" {
40+
maxPrime, _ = strconv.Atoi(value)
41+
}
42+
if key == "loops" {
43+
numLoops, _ = strconv.Atoi(value)
44+
}
45+
}
46+
}
47+
}
48+
49+
// Repeat the calculation of primes simply to give the CPU more work to do without consuming additional memory
50+
for i := 0; i < numLoops; i++ {
51+
primes := sieveOfEratosthenes(maxPrime)
52+
_ = primes
53+
if i == numLoops - 1 {
54+
//fmt.Printf("Highest three primes: %d %d %d\n", primes[len(primes) - 1], primes[len(primes) - 2], primes[len(primes) - 3])
55+
}
56+
}
57+
fmt.Printf("{\"durationSeconds\": %f, \"hostname\": \"%s\", \"max\": %d, \"loops\": %d}", time.Since(start).Seconds(), os.Getenv("HOSTNAME"), maxPrime, numLoops)
58+
}
59+
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
name: jconning/primes
2+
version: 0.0.1
3+
runtime: go
4+
entrypoint: ./func
5+
path: /primes
6+
max_concurrency: 1

0 commit comments

Comments
 (0)