Comparison to native code alternatives

As discussed in Section 3.7, several non-VM approaches have been proposed to guarantee safety on a sensor node. Two of these, t-kernel [35] and Harbor [49], allow the node to guarantee safety independent of the host. Both target the Mica family of sensor nodes, which use the same ATmega128 CPU used by CapeVM. In this section we compare them to CapeVM and consider the question whether a VM is a good way to provide safety.

t-kernel reports a slowdown of between 50 and 200%, which is roughly in the same

Table 7.10: Comparison of overhead in Harbor and CapeVM

Benchmark CapeVM overhead Harbor overhead

VM + safety checks = safe VM current hypothetical

Array writes 182.8% 268.7% 451.5% 1230% 416%

Outlier 75.7% 52.7% 128.4% 690% 234%

16-bit FFT 17.7% 12.6% 30.3% 380% 129%

range as CapeVM. However both t-kernel and CapeVM provide additional advantages.

In t-kernel’s case a form of virtual memory, and for the VM platform independence. This makes them hard to compare, but we note that while the performance of both systems is similar, t-kernel’s code size overhead is much higher at a 6-8.5x increase, limiting the size of programmes that can be loaded onto a device.

A better comparison is possible for Harbor, which only provides safety. Harbor uses three benchmarks to evaluate performance: writing arbitrary data to an array to mimic copying a buffer of sensor data, and the outlier detection and FFT benchmarks also used in the rest of CapeVM’s evaluation. The size of the data used for the first two is not specified in the paper, but since it mentions they work on sensor data and the ATmega CPU has 10-bit analogue-to-digital converters, we use 16-bit data for these benchmarks as well.

As mentioned before, the FFT benchmark is taken from the Harbor sources [94], and outlier detection implemented as described in the paper. The array writes benchmark is implemented as a loop that fills an array of 256 elements with an arbitrary number, as shown in Listing 7.2.

1 for (short i = 0; i < NUMBERS; i++) {

2 numbers[i] = (short)1;

3 }

Listing 7.2: Array writes benchmark (8-bit version)

The resulting overhead is shown in Table 7.10. Filling an array is a hard case for safe CapeVM since consecutive array writes are expensive for two reasons: (i) it results in repeated executions of the PUTARRAY instruction, which calculates the target address for each write, while native code can slide a pointer over the array, and (ii) each of these writes

will trigger a call to heapcheck.

CapeVM incurs overhead both related to the VM, and because of the added run-time safety checks, while for Harbor all overhead is due to safety checks. Still, CapeVM’s total overhead of 451.5% is much lower than Harbor’s 1230%.

While CapeVM is more than twice as fast as Harbor for this benchmark, the compari-son is not entirely fair. Harbor lists the cycle overhead for all of its 5 run-time protection primitives. We assume that without any function calls, only the ’Write access check’ is relevant to this benchmark, which takes 65 cycles. In contrast, CapeVM’s heapcheck routine only takes 22 cycles.

The difference is due to Harbor’s more fine grained protection, which allows it to grant access to any aligned block of 8 bytes to the application, while CapeVM’s protection of the entire heap as a single block is more coarse. If Harbor could be modified to use a similar check as CapeVM’s, its overhead for the array writes benchmark could potentially be reduced to 1230/65∗22 ≈ 416%, slightly faster than CapeVM’s total overhead. However, it is not clear from the paper whether Harbor’s architecture could support such a coarse-grained check since it requires all application data that needs run-time write checks to be in a single block of memory.

While this shows CapeVM achieves a performance comparable to even a hypothetical optimised version of Harbor, the array writes benchmark does not highlight the advantage of using a VM to provide safety because it spends much of its time writing to an array, for which both approaches insert a run-time check. However, the VM can verify writes to local and static variables to be safe at translation time, while the Harbor sources [94]

show that its verifier requires all stores to go through the run-time write access check. The authors do note that static analysis of the code could reduce the number of checks, but that this would come at the cost of a significantly more complex verifier.

CapeVM’s total overhead for the array writes benchmark is slightly higher than the hypothetical optimised Harbor, but the overhead due to safety checks is lower because CapeVM does not need to check writes to the index variable i. This advantage should be more pronounced in code with more frequent writes to local variables, which is exactly

Table 7.11: Number or registers and word size for the ATmega, MSP430, and Cortex-M0

ATmega MSP430 Cortex-M0 [65, 64] [88, 87] [4]

Number of general purpose registers 32 12 13

Word size 8-bit 16-bit 32-bit

Total register file size (bytes) 32 24 52

the case for the more realistic outlier detection and FFT benchmarks.

The reported overhead is 690% and 380%, which would result in 234% and 129%

resp., when using the faster memory access check. For these benchmarks, CapeVM is significantly faster at only 128.4% and 30.3% overhead.