THE LINUX SYSTEM
Chapter A discussed the internals of the 4.3BSDoperating system in detail. BSDis just one of theUNIX-like systems. Linux is anotherUNIX-like system that has gained popularity in recent years. In this chapter, we look at the history and development of Linux, and cover the user and programmer interfaces that Linux presents interfaces that owe a great deal to theUNIX tradition. We also discuss the internal methods by which Linux implements these interfaces.
However, since Linux has been designed to run as many standardUNIXapplications as possible, it has much in common with existingUNIXimplementations. We do not duplicate the basic description ofUNIXgiven in the previous chapter.
Linux is a rapidly evolving operating system. This chapter describes specifically the Linux 2.0 kernel, released in June 1996.
Answers to Exercises
20.1 Linux runs on a variety of hardware platforms. What steps must the Linux developers take to ensure that the system is portable to different processors and memory-management architectures, and to minimize the amount of architecture-specific kernel code?
Answer: The organization of architecture-dependent and architecture-independent code in the Linux kernel is designed to satisfy two design goals: to keep as much code as possible common between architectures and to provide a clean way of defining architecture-specific properties and code. The solution must of course be consistent with the overriding aims of code maintainability and performance.
There are different levels of architecture dependence in the kernel, and different tech-niques are appropriate in each case to comply with the design requirements. These levels include:
CPU word size and endianness These are issues that affect the portability of all software written in C, but especially so for an operating system, where the size and alignment of data must be carefully arranged.
CPU process architecture Linux relies on many forms of hardware support for its pro-cess and memory management. Different propro-cessors have their own mechanisms for
93
94 Chapter 20 The Linux System
changing between protection domains (e.g., entering kernel mode from user mode), rescheduling processes, managing virtual memory, and handling incoming inter-rupts.
The Linux kernel source code is organized so as to allow as much of the kernel as pos-sible to be independent of the details of these architecture-specific features. To this end, the kernel keeps not one but two separate subdirectory hierarchies for each hardware ar-chitecture. One contains the code that is appropriate only for that architecture, including such functionality as the system call interface and low-level interrupt management code.
The second architecture-specific directory tree contains C header files that are descriptive of the architecture. These header files contain type definitions and macros designed to hide the differences between architectures. They provide standard types for obtaining words of a given length, macro constants defining such things as the architecture word size or page size, and function macros to perform common tasks such as converting a word to a given byte-order or doing standard manipulations to a page table entry.
Given these two architecture-specific subdirectory trees, a large portion of the Linux ker-nel can be made portable between architectures. An attention to detail is required: when a 32 bit integer is required, the programmer must use the explicit int32 type rather than as-sume than an int is a given size, for example. However, as long as the architecture-specific header files are used, then most process and page-table manipulation can be performed using common code between the architectures. Code that definitely cannot be shared is kept safely detached from the main common kernel code.
20.2 Dynamically loadable kernel modules give flexibility when drivers are added to a system, but do they have disadvantages too? Under what circumstances would a kernel be com-piled into a single binary file, and when would it be better to keep it split into modules?
Explain your answer.
Answer: There are two principal drawbacks with the use of modules. The first is size:
module management consumes unpageable kernel memory, and a basic kernel with a number of modules loaded will consume more memory than an equivalent kernel with the drivers compiled into the kernel image itself. This can be a very significant issue on machines with limited physical memory.
The second drawback is that modules can increase the complexity of the kernel bootstrap process. It is hard to load up a set of modules from disk if the driver needed to access that disk as itself a module that needs to be loaded. As a result, managing the kernel boot-strap with modules can require extra work on the part of the administrator: the modules required to bootstrap need to be placed into a ramdisk image that is loaded alongside the initial kernel image when the system is initialized.
In certain cases it is better to use a modular kernel, and in other cases it is better to use a kernel with its device drivers prelinked. Where minimizing the size of the kernel is im-portant, the choice will depend on how often the various device drivers are used. If they are in constant use, then modules are unsuitable. This is especially true where drivers are needed for the boot process itself. On the other hand, if some drivers are not always needed, then the module mechanism allows those drivers to be loaded and unloaded on demand, potentially offering a net saving in physical memory.
Where a kernel is to be built which must be usable on a large variety of very different machines, then building it with modules is clearly preferable to using a single kernel with dozens of unnecessary drivers consuming memory. This is particularly the case for commercially distributed kernels, where supporting the widest variety of hardware in the simplest manner possible is a priority.
Answers to Exercises 95
However, if a kernel is being built for a single machine whose configuration is known in advance, then compiling and using modules may simply be an unnecessary complexity.
In cases like this, the use of modules may well be a matter of taste.
20.3 Multithreading is a commonly used programming technique. Describe three different ways that threads could be implemented. Explain how these ways compare to the Linux clonemechanism. When might each alternative mechanism be better or worse than using clones?
Answer: Thread implementations can be broadly classified into two groups: kernel-based threads and user-mode threads. User-mode thread packages rely on some kernel support-they may require timer interrupt facilities, for example-but the scheduling be-tween threads is not performed by the kernel but by some library of user-mode code.
Multiple threads in such an implementation appear to the operating system as a single execution context. When the multithreaded process is running, it decides for itself which of its threads to execute, using non-local jumps to switch between threads according to its own preemptive or non-preemptive scheduling rules.
Alternatively, the operating system kernel may provide support for threads itself. In this case, the threads may be implemented as separate processes that happen to share a com-plete or partial common address space, or they may be implemented as separate execu-tion contexts within a single process. Whichever way the threads are organized, they appear as fully independent execution contexts to the application.
Hybrid implementations are also possible, where a large number of threads are made available to the application using a smaller number of kernel threads. Runnable user threads are run by the first available kernel thread.
In Linux, threads are implemented within the kernel by a clone mechanism that creates a new process within the same virtual address space as the parent process. Unlike some kernel-based thread packages, the Linux kernel does not make any distinction between threads and processes: a thread is simply a process that did not create a new virtual address space when it was initialized.
The main advantage of implementing threads in the kernel rather than in a user-mode library are that:
kernel threaded systems can take advantage of multiple processors if they are avail-able; and
if one thread blocks in a kernel service routine (for example, a system call or page fault), other threads are still able to run.
A lesser advantage is the ability to assign different security attributes to each thread.
User-mode implementations do not have these advantages. Because such implementa-tions run entirely within a single kernel execution context, only one thread can ever be running at once, even if multiple CPUs are available. For the same reason, if one thread enters a system call, no other threads can run until that system call completes. As a re-sult, one thread doing a blocking disk read will hold up every thread in the application.
However, user-mode implementations do have their own advantages. The most obvious is performance: invoking the kernel’s own scheduler to switch between threads involves entering a new protection domain as the CPU switches to kernel mode, whereas switch-ing between threads in user-mode can be achieved simply by savswitch-ing and restorswitch-ing the main CPU registers. User-mode threads may also consume less system memory: most UNIXsystems will reserve at least a full page for a kernel stack for each kernel thread, and this stack may not be pageable.
96 Chapter 20 The Linux System
The hybrid approach, implementing multiple user threads over a smaller number of ker-nel threads, allows a balance between these tradeoffs to be achieved. The kerker-nel threads will allow multiple threads to be in blocking kernel calls at once and will permit run-ning on multiple CPUs, and user-mode thread switching can occur within each kernel thread to perform lightweight threading without the overheads of having too many ker-nel threads. The downside of this approach is complexity: giving control over the tradeoff complicates the thread library’s user interface.
20.4 What are the extra costs incurred by the creation and scheduling of a process, as compared to the cost of a cloned thread?
Answer: In Linux, creation of a thread involves only the creation of some very simple data structures to describe the new thread. Space must be reserved for the new thread’s execution context its saved registers, its kernel stack page and dynamic information such as its security profile and signal state but no new virtual address space is created.
Creating this new virtual address space is the most expensive part of the creation of a new process. The entire page table of the parent process must be copied, with each page being examined so that copy-on-write semantics can be achieved and so that reference counts to physical pages can be updated. The parent process’s virtual memory is also affected by the process creation: any private read/write pages owned by the parent must be marked read-only so that copy-on-write can happen (copy-on-write relies on a page fault being generated when a write to the page occurs).
Scheduling of threads and processes also differs in this respect. The decision algorithm performed when deciding what process to run next is the same regardless of whether the process is a fully independent process or just a thread, but the action of context-switching to a separate process is much more costly than switching to a thread. A process requires that the CPU’s virtual memory control registers be updated to point to the new virtual address space’s page tables.
In both cases—creation of a process or context switching between processes the extra virtual memory operations have a significant cost. On many CPUs, changing page tables or swapping between page tables is not cheap: all or part of the virtual address translation look-aside buffers in the CPU must be purged when the page tables are changed. These costs are not incurred when creating or scheduling between threads.
20.5 The Linux scheduler implements soft real-time scheduling. What features are missing that are necessary for some real-time programming tasks? How might they be added to the kernel?
Answer: Linux’s “soft” real-time scheduling provides ordering guarantees concerning the priorities of runnable processes: real-time processes will always be given a higher priority by the scheduler than normal time-sharing processes, and a real-time process will never be interrupted by another process with a lower real-time priority.
However, the Linux kernel does not support “hard” real-time functionality. That is, when a process is executing a kernel service routine, that routine will always execute to comple-tion unless it yields control back to the scheduler either explicitly or implicitly (by waiting for some asynchronous event). There is no support for preemptive scheduling of kernel-mode processes. As a result, any kernel system call that runs for a significant amount of time without rescheduling will block execution of any real-time processes.
Many real-time applications require such hard real-time scheduling. In particular, they often require guaranteed worst-case response times to external events. To achieve these guarantees, and to give user-mode real time processes a true higher priority than kernel-mode lower-priority processes, it is necessary to find a way to avoid having to wait for low-priority kernel calls to complete before scheduling a real-time process. For example,
Answers to Exercises 97
if a device driver generates an interrupt that wakes up a high-priority real-time process, then the kernel needs to be able to schedule that process as soon as possible, even if some other process is already executing in kernel mode.
Such preemptive rescheduling of kernel-mode routines comes at a cost. If the kernel cannot rely on non-preemption to ensure atomic updates of shared data structures, then reads of or updates to those structures must be protected by some other, finer-granularity locking mechanism. This fine-grained locking of kernel resources is the main requirement for provision of tight scheduling guarantees.
Many other kernel features could be added to support real-time programming. Deadline-based scheduling could be achieved by making modifications to the scheduler. Prioriti-zation of IO operations could be implemented in the block-device IO request layer.
20.6 The Linux kernel does not allow paging out of kernel memory. What effect does this restriction have on the kernel’s design? What are two advantages and two disadvantages of this design decision?
Answer: The primary impact of disallowing paging of kernel memory in Linux is that the non-preemptability of the kernel is preserved. Any process taking a page fault, whether in kernel or in user mode, risks being rescheduled while the required data is paged in from disk. Because the kernel can rely on not being rescheduled during access to its primary data structures, locking requirements to protect the integrity of those data structures are very greatly simplified. Although design simplicity is a benefit in itself, it also provides an important performance advantage on uni-processor machines due to the fact that it is not necessary to do additional locking on most internal data structures.
There are a number of disadvantages to the lack of pageable kernel memory, however.
First of all, it imposes constraints on the amount of memory that the kernel can use. It is unreasonable to keep very large data structures in non-pageable memory, since that represents physical memory that absolutely cannot be used for anything else. This has two impacts: first of all, the kernel must prune back many of its internal data structures manually, instead of being able to rely on a single virtual memory mechanism to keep physical memory usage under control. Second, it makes it infeasible to implement certain features that require large amounts of virtual memory in the kernel, such as the /tmp-filesystem (a fast virtual memory based file-system found on someUNIXsystems).
Note that the complexity of managing page faults while running kernel code is not an issue here. The Linux kernel code is already able to deal with page faults: it needs to be able to deal with system calls whose arguments reference user memory which may be paged out to disk.
20.7 In Linux, shared libraries perform many operations central to the operating system. What is the advantage of keeping this functionality out of the kernel? Are there any drawbacks?
Explain your answer.
Answer: There are a number of reasons for keeping functionality in shared libraries rather than in the kernel itself. These include:
Reliability. Kernel-mode programming is inherently higher risk than user-mode pro-gramming. If the kernel is coded correctly so that protection between processes is enforced, then an occurrence of a bug in a user-mode library is likely to affect only the currently executing process, whereas a similar bug in the kernel could conceiv-ably bring down the entire operating system.
Performance. Keeping as much functionality as possible in user-mode shared libraries helps performance in two ways. First of all, it reduces physical memory consump-tion: kernel memory is non-pageable, so every kernel function is permanently
res-98 Chapter 20 The Linux System
ident in physical memory, but a library function can be paged in from disk on de-mand and does not need to be physically present all of the time. Although the library function may be resident in many processes at once, page sharing by the virtual memory system means that at most once it is only loaded into physical memory.
Second, calling a function in a loaded library is a very fast operation, but calling a kernel function through a kernel system service call is much more expensive. Enter-ing the kernel involves changEnter-ing the CPU protection domain, and once in the kernel, all of the arguments supplied by the process must be very carefully checked for cor-rectness: the kernel cannot afford to make any assumptions about the validity of the arguments passed in, whereas a library function might reasonably do so. Both of these factors make calling a kernel function much slower than calling the same function in a library.
Manageability. Many different shared libraries can be loaded by an application. If new functionality is required in a running system, shared libraries to provide that func-tionality can be installed without interrupting any already-running processes. Sim-ilarly, existing shared libraries can generally be upgraded without requiring any system down time. Unprivileged users can create shared libraries to be run by their own programs. All of these attributes make shared libraries generally easier to man-age than kernel code.
There are, however, a few disadvantages to having code in a shared library. There are obvious examples of code which is completely unsuitable for implementation in a li-brary, including low-level functionality such as device drivers or file-systems. In gen-eral, services shared around the entire system are better implemented in the kernel if they are performance-critical, since the alternative—running the shared service in a separate process and communicating with it through interprocess communication—requires two context switches for every service requested by a process. In some cases, it may be ap-propriate to prototype a service in user-mode but implement the final version as a kernel routine.
Security is also an issue. A shared library runs with the privileges of the process calling the library. It cannot directly access any resources inaccessible to the calling process, and the calling process has full access to all of the data structures maintained by the shared library. If the service being provided requires any privileges outside of a normal process’s, or if the data managed by the library needs to be protected from normal user processes, then libraries are inappropriate and a separate server process (if performance permits) or a kernel implementation is required.
20.8 What are three advantages of dynamic (shared) linkage of libraries compared to static linkage? What are two cases where static linkage is preferable.
Answer: The primary advantages of shared libraries are that they reduce the memory and disk space used by a system, and they enhance maintainability.
When shared libraries are being used by all running programs, there is only one instance of each system library routine on disk, and at most one instance in physical memory.
When the library in question is one used by many applications and programs, then the disk and memory savings can be quite substantial. In addition, the startup time for run-ning new programs can be reduced, since many of the common functions needed by that program are likely to be already loaded into physical memory.
Maintainability is also a major advantage of dynamic linkage over static. If all running programs use a shared library to access their system library routines, then upgrading those routines, either to add new functionality or to fix bugs, can be done simply by replacing that shared library. There is no need to recompile or relink any applications;