1.2 Information Causality
1.3.2 The noisy computation
1.3.2.1 The model of noisy computation
Before studying how to implement the new result for signal propagation in noisy computa-tion, we need to discuss the model of noisy computation in a more precise way. Modeling the computation as a sequence of steps is more convenient for us to connect the compu-tation and the communication but is not precise enough. The more precise model was proposed by von Neumann in 1952 [47]. It is described in Fig.1.3.
The model of noisy computation is a noisy circuit. The circuit has n Boolean inputs and one Boolean output. Note that, the entire circuit is used to compute the Boolean function f : {0, 1}n → {0, 1} and is constructed by many gates which correspond to the vertices in Fig.1.3. Each gate has a fixed number of Boolean inputs and one Boolean output. In Fig.1.3, the edge from vertex a to b corresponds to the variable as the output of gate a and the input of gate b. Assuming these connections are directional and do not allow any feedback. Therefore, the entire circuit is a directional and acyclic graph. Note that, the
Figure 1.3: The model of noisy computation
number of the outgoing edges for each gate does not have to be one. If each gate only has only one outgoing edge, the circuit then has the tree structure. One may call the circuit a formula. The following jargons are used to characterize the model of computation.
• Depth: The depth is the number of gates in the longest path of the circuit.
• Size: The size is the number of the gates in the entire circuit.
In this model, for each gate, the unsuccessful probability for obtaining the Boolean func-tion is . This means that, we cannot always have the perfect computafunc-tion for the entire circuit. However, we can study the reliable computation. The circuit is the reliable com-putation, if we can obtain the Boolean function of the n inputs f : {0, 1}n → {0, 1} with the (1 − δ) ≥ 12 successful probability.
1.3.2.2 The tolerable error rate and the depth for a reliable computation.
For the reliably computational circuit, the error rate for each gate and the depth of the circuit are the essential values because both of them will affect the successful probability for the entire computation.
In [47], von Neumann calculated the tolerable error rate for the reliable computation. In von Neumann’s computation circuit, each gate has three inputs. The reliable computation can be achieved by the noisy gates when the error rate of each gate is independent and less than 16. In this case, the error rate could not be arbitrary for reliable computation.
On the other hand, one may increases the depth of the circuit to control the successful probability for the entire computation. Since the computational time is increased with bigger depth of the circuit, the price to pay for the reliable noisy computation is the computational time.
Moreover, in [48] Pippenger used the information-theoretic method to prove that the limit of and the longer time are required for the reliably noisy formulas (tree structure of the circuits). Let x1, x2, ..., xn be the Boolean inputs of the circuit. In Pippenger’s proof, for any input xm, one can obtain that the output of the noiseless circuit is equal to xm or the complement of xm while the other n − 1 inputs correspond to a specific setting so that the circuit is noiseless. Therefore, with this specific setting, the output O of the reliable circuit should be strongly correlated with the input xm. In this information theoretical setup, it implies that the mutual information between xm and the output O of the reliable circuit should be high enough. Since we know that the structure of the noisy gates affects the mutual information between the input xm and the output O, the total information flow from the input xm to the output O is bounded by the summation of information flow for each path p from xm to the output O. According to this condition, Pippenger obtained the upper bound of the mutual information
I(xm; O) ≤ Σpξ|p|, (1.60)
where |p| is the number of the gates in the path p from xm to O and the error rate for each gate is = 1−ξ2 .
With this inequality, Pippenger then obtained the lower bound of the depth for the reliable circuit. Assuming n inputs of the circuit and that each gate has at most k inputs.
Note that, the error rate for each gate = 1−ξ2 . Pippenger then obtained the constraint for the depth c of the 1 − δ ≥ 12-reliable circuit as follows:
• (i) if ξ > k1 then c ≥ log(n∆)/ log(kξ) ,
• (ii) if ξ ≤ k1 then n ≤ 1/∆,
where ∆ := 1 + δ log δ + (1 − δ) log(1 − δ). For k = 3 case, one may find the tolerable error rate 13 is higher than the one in the von Neumann’s argument. On the other hand, the lower bound of the depth must increase. Here, the lower bound of the noiseless circuit is logkn and one may multiply it by at least (1+log1
kξ) to obtain the lower bound of the depth for the noisy circuit. Moreover, when the error rate is higher than the tolerable one, the number of inputs cannot be arbitrary and it will be limited by ∆. However, since Pippenger’s argument only works for the formulas, we are not sure if the above bounds on the error rate and the depth also hold for the other circuits.
Pippenger used three inequalities to obtain the upper bound (1.60) on the mutual information. The first one is the data processing inequality. Assuming the gate G produces the output Y with inputs Yi for i ∈ {1, ..., k}. The mutual information between the essential input xm and Y is smaller than the one between xm and Y1, Y2, ..., Yk, i.e., I(xm; Y ) ≤ I(xm; Y1, Y2, ..., Yk). The second one is I(xm; Y1, Y2, ..., Yk) ≤ P
iI(xm; Yi).
This inequality holds when we can obtain the information about xmfrom Yiindependently.
Therefore, the second inequality holds for the formulas but may not be satisfied by other circuits. In [51], one example (in Fig.1.4) can show that the inequality does not hold any more. In this example, Y2 is the input for both of the gates which are noiseless NOR gates 1. Therefore, one can know xm by Y1 and Y2, i.e., I(xm; Y1, Y2) = 1. Note that, xm could not be obtained when one only knows Y1 or Y2, i.e., I(xm; Y1) = I(xm; Y2) = 0.
Obviously, in this case, I(xm; Y1, Y2) > P2
i=1I(xm; Yi).
The third ingredient is I(xm, Z) ≤ ξI(xm; Y ), where Z is the output for the other gate with the input Y . One can image Z = Y + V , where V is a Boolean variable and its value is 1 with probability 1−ξ2 and 0 with probability 1+ξ2 [48]. With this assumption, one may obtain the third inequality. Note that, this inequality also implies I(xI(xm,Z)
m;Y ) ≤ ξ.
In order to obtain the lower bound of the depth for more general circuits, Evans and Schulman overcame the difficulty for the second inequality and modified the third inequality [50, 51]. Instead of summing over the individual mutual information between xm and Yi, Evans and Schulman directly calculated the bound of the mutual information between xm and a set of random variables. On the other hand, for the third inequality, they had an impressive modification. They found that the noisy gate could correspond a binary symmetric channel given by (1.58). According to the proof in the previous
1When both the inputs of the NOR gate are 0, the output will be 1, otherwise the output is 0
XOR1
XOR2
Figure 1.4: The example for the failure of inequality I(xm; Y1, Y2, ..., Yk) ≤P
iI(xm; Yi)
subsection, instead of ξ implied by the third inequality, one could obtain the upper bound of the ratio I(xI(xm;Z)
m;Y ) to be ξ2. Therefore, one can modify the third inequality to be I(xm, Z) ≤ ξ2I(xm; Y ).
Using these modifications, one can then obtain the upper bound of the mutual infor-mation, i.e.,
I(xm; W ) ≤ Σpξ2|p|, (1.61)
where W is used to denote the set of variables and |p| is the number of the gates on the path p from xm to W . Note that, if W is the output O, one may find this upper bound is tighter than (1.60). Similar to Pippenger’s argument, one may use this upper bound to obtain the lower bound of the depth c for the 1 − δ ≥ 12-reliable circuit using the gates with at most k inputs. The results are as follows:
• (i) if ξ2> 1k then c ≥ log(n∆)/ log(kξ2) ,
• (ii) if ξ2≤ 1k then n ≤ 1/∆,
where ∆ := 1 + δ log δ + (1 − δ) log(1 − δ), and the error rate for each gate is = 1−ξ2 . Compare this result with the ones from von Neumann’s and Pippenger’s arguments, one can find two things. The first one is that, the threshold of the error rate for each gate becomes looser. For the reliable computation with k = 3 gates, von Neumann showed it can be achieved when < 16, and the error rate requires < 13 for Pippenger’s argument. For the same case, Evans and Schulman showed that the threshold error rate is 12(1 − √1
3) which is bigger than the ones from von Neumann and Pippenger’s arguments. The second one is that, the lower bound of the depth is bigger than the one from Pippenger’s arguments. As known that in order to achieve the reliable computation using the noisy gates, one needs to increase the depth of the noisy circuit. Here, the lower bound of the depth for the noiseless circuit is logkn. From Pippenger’s arguments, one needs to multiply it by (1+log1
kξ) to obtain the lower bound of the depth for the noisy circuit. On the other hand, Evans and Schulman obtained that one needs to multiply it by (1+log1
kξ2). Thus, one may need more time to realize the reliably noisy computation.
In Chapter 2, we will use the bound on the error rate to check if IC allows the 1−δ ≥ 12 -reliable computation.