2.1.1 Processes and Messages
We abstract the units that are able to perform computations in a distributed sys-tem through the notion of a process. We consider that the syssys-tem is composed of N different processes, named p, q, r, s, and so on. The set of processes in the system is denoted by Π. Unless stated otherwise, this set is static and does not change, and every process knows the identities of all processes. Sometimes, a func-tionrank: Π → {1, . . . , N} is used to associate every process with a unique index between1 and N. In the description of an algorithm, the special process name self denotes the name of the process that executes the code. Typically, we will assume that all processes of the system run the same local algorithm. The sum of these copies constitutes the actual distributed algorithm.
We do not assume any particular mapping of our abstract notion of process to the actual processors or threads of a specific computer machine or operating sys-tem. The processes communicate by exchanging messages and the messages are uniquely identified, say, by their original sender process using a sequence number or a local clock, together with the process identifier. In other words, we assume that all messages that are ever exchanged by some distributed algorithm are unique.
Messages are exchanged by the processes through communication links. We will capture the properties of the links that connect the processes through specific link abstractions, which we will discuss later.
2.1.2 Automata and Steps
A distributed algorithm consists of a distributed collection of automata, one per process. The automaton at a process regulates the way the process executes its com-putation steps, i.e., how it reacts to a message. Every process is implemented by the same automaton, as illustrated in Fig.2.1; they interact through some means of communication that will be introduced later. The execution of a distributed algo-rithm is represented by a sequence of steps executed by the processes. The elements of the sequences are the steps executed by the processes involved in the algorithm.
A partial execution of the algorithm is represented by a finite sequence of steps, an infinite execution by an infinite sequence of steps.
Communication abstraction Processes
p q r z
Figure 2.1:A distributed algorithm consisting of processes that are implemented by identical automata
It is convenient for presentation simplicity to assume the existence of a global clock, outside the control of the processes. This clock provides a global and linear notion of time that regulates the execution of the algorithms. The steps of the pro-cesses are executed according to ticks of the global clock: one step per clock tick.
Even if two steps are executed at the same physical instant, we view them as if they were executed at two different times of our global clock. A correct process executes an infinite number of steps of its automaton, i.e., every such process has an infinite share of time units (we come back to this notion in the next section) and follows the specified algorithm. In a sense, there is some entity, sometimes called a global scheduler, that assigns time units to processes, though the very notion of time is outside the control of the processes.
A process step consists of receiving (sometimes we also say delivering) a mes-sage from another process (global event), executing a local computation (local event), and sending a message to some process (global event) (Fig. 2.2). The execution of the local computation and the sending of a message is determined by the process automaton, i.e., by the local algorithm. Local events are typically those exchanged between modules of the same process at different layers.
Sometimes a process has no message to receive or send, but has some local com-putation to perform; this is captured simply by assuming that messages can be nil, in the sense that the process receives or sends a special nil message. Of course, a process might not have any local computation to perform either, in which case it simply does not touch any of its local variables. In this case, the local computation is also nil.
It is important that the interaction between the components of one process is viewed as local computation and not as communication, although they look syntac-tically the same. When an event is exchanged between two modules of the same process, the algorithm performs a computation step that is local. In contrast, a com-munication stepof the algorithm occurs when a process sends a message to another process, and the latter receives this message, through events occuring at different processes. The process is the unit of communication, just like it is the unit of failure, as we will discuss. As the transmission delay of a network is typically much larger than the local computation delay, the number of communication steps of an algo-rithm has a significant impact on the latency and the performance of a distributed
Process
Modules of the process internal computation
(receive)
incoming
message outgoing
message (send)
Figure 2.2:Step of a process
algorithm. Needless to say, the number of computation steps may also affect the per-formance of the algorithm, especially when computationally expensive operations are involved, such as cryptographic operations.
An important parameter of the process abstraction is the restriction imposed on the speed at which local steps are performed and messages are exchanged. We will come back to this aspect when discussing timing assumptions later in this chapter.
Unless specified otherwise, we will consider deterministic algorithms. That is, for every step performed by any given process, the local computation executed by the process, the local state after the computation, and the message sent by this pro-cess are uniquely determined by the message received by the propro-cess and its local state prior to executing the step.
In specific situations, we will also discuss randomized (or probabilistic) algo-rithms, where every process may use a local random source. The output of the random source determines the choice of the local computation to perform or the next message to send, according to a probability distribution over the set of possible values output by the source.
2.1.3 Safety and Liveness
When we devise a distributed algorithm to implement a distributed programming abstraction, we seek to satisfy the properties of the abstraction in all possible exe-cutions of the algorithm, covering all possible sequences of steps executed by the processes according to the algorithm. The scheduling of these steps remains outside the control of the processes and depends on the global scheduler. The properties of the abstraction to be implemented needs to be satisfied for a large set of pos-sible interleavings of these steps. These properties usually fall into two classes:
safety and liveness. Having in mind the distinction between these classes usually helps to understand the two complementary faces of the abstraction and to devise an adequate algorithm for implementing it.
Safety. Basically, a safety property is a property of a distributed algorithm that can be violated at some time t and never be satisfied again after that time. Roughly
speaking, safety properties state that the algorithm should not do anything wrong.
To illustrate this, consider a property of perfect links (which we will discuss in more detail later in this chapter) stating that no process should receive a message unless this message was indeed sent. In other words, communication links should not invent messages out of thin air. To state that this property is violated in some execution of an algorithm, we need to determine a time t at which some process receives a message that was never sent. This observation helps devise a correctness argument (by contradiction) for an algorithm presumably satisfying the property.
More precisely, a safety property is a property such that, whenever it is violated in some execution E of an algorithm, there is a partial execution E′of E such that the property will be violated in any extension of E′. This means that safety properties prevent a set of unwanted execution prefixes from occurring.
Of course, safety properties are not enough. Sometimes, a good way of prevent-ing bad thprevent-ings from happenprevent-ing consists in simply doprevent-ing nothprevent-ing. In many countries, some public administrations seem to understand this rule quite well and, hence, have an easy time ensuring safety.
Liveness. In order to define a useful abstraction, it is therefore necessary to add some liveness properties. They ensure that eventually something good happens. For instance, to define a meaningful notion of perfect links, we require that if a cor-rect process sends a message to a corcor-rect destination process, then the destination process should eventually deliver the message (besides the safety property which stipulates that messages should not be invented out of thin air and only be delivered if priorly sent). To state that such a liveness property is violated in a given execu-tion, we need to show that there is an infinite scheduling of the steps of the algorithm where the message is never delivered.
More precisely, a liveness property is a property of a distributed system execution such that, for any time t, there is some hope that the property can be satisfied at some time t′ ≥ t. It is a property for which, quoting Cicero, “while there is life there is hope.”
Combining them. The challenge is to guarantee both liveness and safety. (The difficulty is not in talking, or not lying, but in telling the truth.) Indeed, useful distributed services are supposed to provide both liveness and safety properties.
Formulating an abstraction with only one kind of property is usually a sign for a flawed specification.
Consider, for instance, the traditional interprocess communication service of a reliable, ordered data stream: it ensures that messages exchanged between two pro-cesses are neither lost nor duplicated, and are received in the order in which they were sent. As we pointed out, requiring that messages are not lost is a liveness prop-erty. Requiring that messages are not duplicated and that they are received in the order in which they were sent are safety properties.
As another example, the soundness property of the job handler abstraction in Module1.2from Sect.1.4represents a safety property. Moreover, Modules1.1and 1.2 in the same section both contain a guaranteed response property, which is a liveness property.
Crash
Omission Crash with Recovery
Eavesdropping Arbitrary
Figure 2.3:Types of process failures
It is usually better, for modularity purposes, to separate the safety and liveness properties of an abstraction specification into disjoint classes. However, we will sometimes for the sake of conciseness consider properties that are neither pure liveness nor pure safety properties, but rather a union of both.