Replacement Policies for Code Buffer Management

When a method gets compiled by the JIT compiler, the JIT-compiled code is stored into the code buffer for future utilization. If the code buffer doesn’t have enough free space for a newly compiled method, one or several compiled methods in the code buffer must be replaced to make room for the newly compiled method. The replacement policy for deciding which method(s) to evict from the code buffer is of crucial importance since evicting frequently executed methods may result in performance degradation.

Besides, the fragmentation problem and the maintenance overhead of the replacement policy also have influences on the program execution performance. The fragmentation problem means that even if the total capacity of the free space in the code buffer is adequate for an incoming method, the method still cannot be inserted into the code buffer because the free space is fragmented. For an example of a fragmented code buffer, see Figure 2-3, where the white blocks indicate free space. Although the total size of the free space is larger than the size of the incoming method, each segment of contiguous free space is too small to fit the method. The defragmentation operation may be employed to deal with this problem, but the overheads of most defragmentation techniques are too high to be applicable to Java environments. Such a cause necessitates a replacement policy for code buffer management that scarcely brings about the fragmentation problem.

14

Figure 2-3 An Example of a Fragmented Code Buffer

The following are the six most common ways to decide which method(s) to evict from the code buffer:

 Least-Recently Accessed (LRA)

When the code buffer doesn’t have sufficient free space for a newly compiled method, the JIT-compiled method in the code buffer that hadn’t been accessed for the longest time is the first candidate for replacement. In case removing only this JIT-compiled method cannot free an adequately large space to place the incoming method, the subsequent one or more methods in the code buffer are also removed until the space freed is larger than the incoming method. This policy has the benefit of exploiting the temporal locality, yet it has the drawback of deleting innocent victim methods in order to make a contiguous space large sufficiently for the incoming method. This policy would suffer from the effect of fragmentation.

 Least-Frequently Accessed (LFA)

In the event that there isn’t adequate room in the code buffer for an incoming method, the JIT-compiled method accessed the fewest times among all the methods in the code buffer is the first candidate for replacement. Like the LRA policy, if evicting the method doesn’t generate enough free space, the subsequent one or more methods are evicted as well. Even though this policy can identify hot methods effectively, it has the possibility of deleting a method compiled just lately, whose access count has not been high yet. Additionally, this policy also has the disadvantage of giving rise to the fragmentation problem.

15

 Least-Recently Created (LRC)

This policy treats the code buffer as a circular buffer, and the compiled methods are replaced in the same order as they were inserted. In the event that replacing a method does not produce adequate free space for the incoming method, the subsequent one or more methods are also replaced till the space freed is enough. If the space freed is larger than needed by the incoming method, the remainder free space can be used by the next method inserted into the code buffer. This policy completely avoids the generation of fragmentation, and the overhead is rather low because it merely needs a pointer to be updated when a method is inserted.

 Largest Element (LE)

When a JIT-compiled method cannot be put into the code buffer owing to lack of free space, the method of the largest size in the code buffer is selected as the first victim to discard.

Likewise, if discarding the method cannot free adequate space, the subsequent one or more methods are also discarded. The purpose of this policy is to minimize the number of replacements that occur within the code buffer, but no attention is paid to the temporal locality.

This policy would lead to the fragmentation problem as well.

 Best-Fit Element (BFE)

In case the free space in the code buffer is not sufficient to store a newly compiled method, the entire code buffer is scanned in search of the method of the smallest size that is larger or equal to the size of the incoming method. If there is no method in the code buffer large enough for the incoming method, all the methods and their next ones are grouped into pairs of two, and the best-fit search resumes. Although this policy attempts to minimize the number of times the fragmentation problem arises, it incurs very high overhead from the search for the best-fit element.

 Full Cache Flush

As soon as a JIT-compiled method cannot be inserted into the code buffer, all compiled methods in the code buffer are flushed, and the incoming method becomes the first element

16

placed in the code buffer. Although this is a replacement policy with very low overhead and no fragmentation problem, it has the adverse side effect of evicting frequently executed methods from the code buffer and thus potentially degrades the program execution performance.

The above six replacement policies for code buffer management have been evaluated in a previous study [7]. The experimental result reveals that the LRA policy outperforms the other policies across various code buffer sizes in terms of the code buffer miss rate as shown in Figure 2-4. The code buffer miss rate of the LRC policy is also fairly low, although it is a little higher than that of the LRA policy.

Figure 2-4 Code Buffer Miss Rate of Each Code Buffer Replacement Policy [7]

In addition to the code buffer miss rate, the fragmentation problem and the maintenance overhead should also be taken into account for the evaluation of these policies. As summarized in Table 2-1, only two policies, LRC and Flush, do not suffer from the fragmentation problem at all, and each of them simply needs a pointer for management, incurring the lowest overhead among all the policies. After combining the factors of code buffer miss rate, fragmentation problem, and overhead, the LRC policy, with a low code buffer miss rate, no fragmentation problem, and the lowest maintenance overhead, appears to be the most desirable replacement policy for code buffer management.

17

Table 2-1 Summary of Code Buffer Replacement Policies [7]

Policy Fragmentation Code Buffer Miss Rate Management

Flush None 4.61% Pointer

LRA Yes 2.48% Priority Queue

LFA Yes 9.11% Priority Queue

LE Yes 13.91% Priority Queue

BFE Minimal 20.77% Multiple Sorted Lists

LRC None 2.88% Pointer