Performance
This section evaluates the performance and adaptability of Shawn. It first compares Shawn with Ns-2 and TOSSIM to demonstrate that Shawn can handle large networks at high speeds. Then Shawn’s adaptability and flexibility is demonstrated at the example of the available Edge Models.
Since the exchange of wireless messages is the key ingredient in wireless sensor networks, a simulator’s ability to dispatch messages to their recipients determines the speed of simulations. In the following, measurements are presented that show the amount of memory and CPU time required to simulate a simple application that broadcasts a message every 250ms of simulated time. The communication range of the sensor nodes is set to 50 length units and each simulation runs for 60 simulated time units. The size of the simulated area is 500x500 length units. A number of simulations with increasing node count were performed. Therefore, the network’s density increases steadily as more nodes are added to the scenario. This application has been implemented for Ns-2, TOSSIM and Shawn. All simulation tools were used as supplied by the source repositories with maximal compiler optimization enabled. The simulations were run on standard, state of the art i686 PCs.
The left figure below depicts the required CPU time in seconds and the figure on the right shows the maximally used amount of RAM for the three simulation tools at different node counts. It should be noted that this kind of comparison is biased in favor of Shawn because the two other simulators perform much more detailed computations to arrive at the same results. It is more to be seen as an indication how application developers can benefit from using Shawn when these detailed results are not in the focus of interest. When interpreting the results, please note that these figures as well as the following ones use a logarithmic scale on both axes.
{| align="center"
| 
| 
|}
The first thing to notice is that Shawn outperforms both other simulation tools by orders of magnitude. Ns-2 hits the one-day barrier where Shawn is still finishing in less than one minute with a considerably smaller memory footprint. As mentioned above, this is because Ns-2 performs a very detailed simulation of lower layers such as the physical and the data link layer while Shawn simply dispatches the messages using a simplified model. Nearly the same situation applies to TOSSIM that simulates an underlying TinyOS-supported hardware platform. This clearly shows that Shawn excels in its area of expertise – the simulation of large-scale sensor networks with a focus on abstract, algorithmic considerations and high-level protocol development.
As discussed in the introduction, Shawn’s runtime behavior is heavily influenced by choosing a distinct implementation of one of the models (e.g., the [Edge Model](Edge Model), [Communication Model](Communication Model) or the [Transmission Model](Transmission Model)). Depending on the simulated scenario, a different choice may substantially alter the performance and the resource consumption of the simulation. Since selecting a specific implementation simply requires changing a value in the configuration file, users can select the best implementation for each simulation.
To demonstrate the pros and cons of the different edge model implementations, the above-described simulations were repeated using the same underlying scenarios using different edge models. The figure below on the left shows the required CPU time for the different implementations and node counts. It is evident, that the List and the Fast List are faster than Grid and much faster than Simple. However, considering the memory consumption (the figure on the right), this performance gain comes at a certain price.
{| align="center"
| 
| 
|}
Because the size of the simulated world is fixed at 500x500 length-units with a communication range of 50 length-units, the density of the underlying graph – and therefore the amount of memory required for storing the neighbors of each node – increases significantly the more nodes are added to the scenario. A compromise between speed and memory consumption is offered by Grid.
However, this observation does not hold in general. Consider the situation where the number of nodes increases while the average density of the network remains constant. The figures below depict the result of such simulations with a constant density (the density corresponds to the one of a scenario with 100 nodes in the simulations presented above). When looking at simulations using Fast List, the runtimes are an order of magnitude lower than in previous case. However, the memory requirements differ only marginally (maximal difference 108.64 MByte, 9.94 MByte in average). This is because, in contrast to the previous scenario, the neighborhood sizes remain constant and the storage space required for keeping them in memory increases only slowly. As in the previous example, Simple is the slowest one and Fast List the fastest edge model implementation. In this example, Grid outperforms List only for node counts larger than 10,000. This is because of the high initial setup cost of the list edge model.
{| align="center"
| 
| 
|}
However, Fast List performs better for all node counts which is because it reduces the time required for construction of a List edge model by using a Grid internally to limit the search space for neighboring nodes.
The above-presented measurements show that Shawn’s central design goal (simulate the effect caused by a phenomenon, and not the phenomenon itself) indeed leads to a high scalability and performance compared to traditional approaches to simulation. Furthermore, the results show that Shawn’s runtime behavior can be custom-tailored simply by changing configuration parameters. This does not only apply to edge models, but also the other models used in Shawn have a major impact on the performance and resource consumption. Hence, an optimal selection of model implementations must take the properties of the scenario into account.
As a result, developers must carefully select the simulation tool depending on the application area. When detailed simulations of issues such as radio propagation properties or low-layer issues should be considered, Shawn is obviously not the perfect choice. This is where Ns-2 and TOSSIM offer the desired granularity. However, when developing algorithms and high-level protocols for WSNs, this level of detail often limits the expressiveness of simulations and blurs the view on the actual research problem. This is where Shawn provides the required abstractions and performance.