Fault Tolerance In A Distributed System

Published Date: 02 Nov 2017

Abstract—The essential problem in distributed computing is to achieve overall system reliability in the presence of a number of faulty processes. Maximizing fault tolerance is the important for message exchanges in distributed environment. To achieve this point, solving consensus problem that requires agreement among a number of processors for a single data value is important one. This system proposed the faster consensus agreement approach to solve the consensus problem in simple and better ways. This system uses as few as two rounds of message exchange. The network partitioning is used to reduce the message exchange overhead.

Keywords— Distributed Computing, Fault Tolerance, Consensus Problem, Byzantine Agreement Problem, Network Partitioning

Introduction

In distributed computing, making agreement among the non-faulty processes is most important because the failed component and traitors may send conflicting information to different parts of system. Therefore, distributed agreement problems are essential today. The general goal of the distributed agreement algorithm is to have all non faulty processes reach the agreement. All the processes including the system have to collaborate for that agreement.

One of the famous agreement problems is Byzantine General Problem [4]. In this problem, the initiator broadcasts a value to all other processes. These processes make the message exchange each other. Then, this system tried to detect the traitors and removed them from the group. Finally, the rest all non-faulty processes are agree on the value of initiator.

If there are many processes and their behaviours are different, the operation becomes more difficult. There are some solution to reach this agreement with the maximum fault tolerance and the minimum message exchange. But, the solutions are not optimal in both cases and they have some assumptions [8]. The proposed system in this paper can tolerate the maximum fault tolerance and it can exchange message only in two rounds. This can reduce the time to take an agreement among the processes.

Related Works

The conventional Byzantine agreement problem [1] with n processors requires many numbers of rounds to exchange the messages. It has the upper bound of n>=3t+1 where t is the maximum number of tolerable faulty processes. It is proved that all the conventional byzantine agreement problems require t+1 rounds for sending message among the processors.

Synchronous consensus with Mortal Byzantines approach [5] reduces the number of rounds for sending messages up to n>=2t. However, it has an assumption that faulty process takes a finite number of arbitrary steps before they eventually crash.

Quick consensus algorithm [7] gets the agreement only in two rounds of message exchange. It uses the network partitioning scheme to reduce the message exchange overhead and network traffic. Each process exchanges the initial value in first rounds. In the second round, the processes get the vector of all others to calculate the decision vector. But, that algorithm can handle the faulty processes which happened only in the first round. However, that algorithm did not consider for the faulty processes in the second round of message exchange.

The next solution for consensus problem proposed in [8] also has only two rounds of message exchange. It can solve the assumption proposed in Quick consensus algorithm. The algorithm can detect the faulty processes in the second round. But that has some delay to take the agreement value.

To save the time, a solution that reaches the agreement in quick and simple way is proposed. This can get the maximum fault tolerance and minimum message exchange overhead only in two rounds.

Proposed Algorithm

A large network with many processors may include the number of message exchanged overhead. The failed components and traitors may send conflicting information to different parts of the system. The main point is to reach all non-faulty processes to an agreement. The proposed system can provide the decision value in two rounds of message exchange when the system tries to reach the agreement.

To reduce the message exchange overhead, the network partitioning scheme is used. It can reduce the network traffic in a network with the large number of processors. The following section is the propose scheme.

3.1 Reaching Consensus among a Group of Processors

In this system, all the process sites exchange their initial values with each other as shown in Figure 1. The proposed scheme uses exactly two rounds of message exchange. Every process site must possess an initial decision value (0/1) before participating in the consensus procedure. A system of five processes (P1, P2, P3, ..., P5) is considered among which P1 and P3 are assumed faulty.

Round 1:

All process sites exchange their initial values.

Round 2:

At the beginning of round two, each process Pi is aware of the initial value of every other. These values are composed as the initial vector Fi. The vector Fi stored at each process site.

Each process receives all vectors of other processes to be two dimension matrixes Si.

There are four cases depending on the various natures of the faulty processes. These cases are:

Case 1: If there is no faulty process in the system, all the processes receive the initial value of other processes correctly. This case is shown in Figure 2. It is clearly we can see that all rows values are identical. It is easy to compute the majority value.

Case 2: In this case, the faulty process generates wrong value in round one. It is assumed that there is not happened in round 2. The Matrix Si at each process is as shown in Figure 3. The row values for faulty processes are different from others.

Case 3: The faulty processes send the wrong messages to other processes only in round two. In round one, each process can receive the correct message shown in Figure 4(a). After round two, the row values of received matrix Si for each process site is different as shown in Figure 4(b).

Case 4: This is the big case. The faulty processes send the wrong value in both rounds. Since round one is finished, the row values for faulty processes are different from others as shown in Figure 5(a). After both round, different row values for each matrix Si in every sites are shown in Figure 5(b).

The next section is described the solution to reach a consensus among the non-faulty processes. To reach this solution, the following assumptions are needed.

All the processors in the network are fully connected. The entire links among the processors are reliable and any delay does not be introduced.

Assume two types of fault can be included. First is dormant faulty process 'd' that may be caused from traitors. The other is crash faulty process 'c' that cannot send back any message when a process site crashes permanently.

t = d + c is the total number of faulty processes.

Algorithm I:

co = Coordinator processor

Pi = Process ID at each processor site, i=1, 2, …, n. Where n= total number of processor

Fi = Initial Vector for each site after first round

Si = n x n Matrix for each site after second round

D = Decision Vector

A = Agreement Result Vector

Input : Process ID Pi and initial value (0/1) for each process

Output : Decision and Agreement Result

Step 1:

Pco = randomize (Pi)

Each process sends its initial value to all other processes

If a process Pi does not receive message from Pco,

Then Pco = randomize(Pi)

cv = i

where Pco {Pi}

If a process Pi does not receive message from any other process Pk

Then Fi[k] = C

Step 2:

Each process sends its vector Fi to every other processor

If process Pi does not receive any vector from any other process Pk, Then

for i= 1, 2, .., n of Fi

{S[k][i]=C}

Step 3:

For coordinator Pco compare with each processor Pi

For j=1 to n

Begin

Di= true;

For each column k=1 to n

Begin

If Si[j][k] contain C

Then Dk=false and break.

Else if (Sco[j][k] != Si[j][k])

Then Dj=false and break.

End if

End

End

Step 4:

For i=1 to n

Begin

If Di=true

Then Ai= Sco[cv][i]

Else Ai=Di

End

{1}

{0}

{1}

P1

P2

P3

P4

P5

{0}

{1}

Figure 1. Exchanging initial value in round one

Case 1:

1

0

1

0

1

1

0

1

0

1

1

0

1

0

1

1

0

1

0

1

1

0

1

0

1

Figure 2. 2x2 matrix of the state at no one is faulty(S1=S2=S3=S4=S5)

Case 2:

1

0

1

0

C

1

0

1

0

C

1

0

1

0

C

1

0

1

0

C

C

C

C

C

C

Figure 3. The faulty process sends wrong message since round one is executed (assume P5 is crashed)

Case 3:

P1

P2

P3

P4

P5

1

0

1

0

1

F1

1

0

1

0

1

F2

1

0

1

0

1

F3

1

0

1

0

1

F4

1

0

1

0

1

F5

Figure 4(a). Correct vectors received after 1st round

1

1

0

1

0

1

0

0

1

0

1

0

1

0

1

1

0

1

0

1

0

0

1

0

1

1

1

0

0

1

1

0

1

0

1

1

0

1

0

1

1

0

1

0

1

1

0

1

0

1

S2

S1

1

1

0

1

0

1

1

0

1

1

1

0

1

0

1

1

0

1

0

1

0

0

1

0

1

1

0

1

0

1

1

0

1

0

1

1

0

1

0

1

1

0

S31

0

1

1

0

S41

0

1

1

0

1

1

1

1

0

1

0

1

0

1

0

0

1

1

0

1

0

1

1

0

S51

0

1

Figure 4(b). Wrong vectors received in second round (assume P1 and P3 are faulty processes)

Case 4:

P1

P2

P3

P4

P5

1

0

1

0

1

F1

0

0

0

0

1

F2

0

0

1

0

1

F3

0

0

1

0

1

F4

1

0

0

0

1

F5

Figure 5(a). Vectors of each site when faulty processes are found in round one

0

1

0

0

1

1

0

1

1

1

0

0

0

0

1

0

0

0

0

1

1

0

0

0

1

1

1

1

0

1

0

0

1

0

1

0

0

1

0

1

1

0

S10

0

1

1

0

S20

0

1

1

0

0

0

0

1

1

0

1

0

0

0

0

0

1

0

0

0

0

1

1

1

0

0

1

1

0

0

0

1

0

0

1

0

1

0

0

1

0

1

1

0

S30

0

1

1

0

S40

0

1

1

0

0

1

1

0

0

0

0

1

0

0

1

0

1

0

0

1

0

1

1

0

S50

0

1

Figure 5(b). 2x2 Matrix of each process while P1 and P3 are faulty process in both rounds

0

1

0

0

1

1

0

1

1

1

0

0

0

0

1

0

0

0

0

1

1

0

0

0

1

1

1

1

0

1

0

0

1

0

1

0

0

1

0

1

1

0

S10

0

1

1

0

S20

0

1

0

1

0

0

1

1

0

1

1

1

0

0

0

0

1

0

0

0

0

1

1

0

0

0

1

1

1

1

0

1

0

0

1

0

1

0

0

1

0

1

1

0

S30

0

1

1

0

S40

0

1

1

0

0

1

1

0

0

0

0

1

0

0

1

0

1

0

0

1

0

1

1

0

0

0

1

S5

Figure 5(c). Detection the faulty processes during round two

Decision Vector

F

T

F

T

T

Result Vector

F

0

F

0

1

Figure 6. Final Decision Vector and Result Vector

3.2 Reaching Consensus among entire Networks

It is sure that the network traffic is huge in a large network. Assume that G number of groups is partitioned in a large network and N processors are included in each group. Each group has a coordinator Ci to decide the consensus agreement among its members. After completing the above algorithm I, the result vector is decided for that particular group. Once the local round for all the groups are completed, the coordinator from each group participates in a global decision making round. After executing the above algorithm I, each processor possesses a value of decision vector F and number of non-faulty processes W for its group Gi.

At the initialization phase of network partitioning, a coordinator Ci for each group is selected randomly and they broadcast an initialization message [2][3]. After receiving it, each process of the system initializes a counter value to 1 and starts incrementing. Then Ci broadcasts G tokens. Each token can be received by one and only one process. The processors holding the token are leaders. Each leader logically forms a group of N/G-1 processes.

Algorithm II:

Input: leaders of the groups and their weighted local decision value (F, W)

Output: Global decision value

if for any k, 1<k<G, local round for Group Gk is finished. Ck sets random timer T[C] and starts decrementing it

If C doesn't receive any advertisement from an initiator of global round and T[C]=0, then C send local decision and weight to all leaders

Else go to step 2

If a leader Q  Gi receives initialization message from C then Q resets T[Q]=0

C initiates Algorithm I considering the set of leaders as a group and decision value computed is the global decision value

The leader of each group conveys global decision value to all processes belonging to its group Gi

Return global decision value

If weighted local decision value is sent by the faulty leader of group G, there may be a chance of mishap. However, the proposed scheme can mask off such faults as the faulty leader is detected in the local round. Once the faulty leader is detected, a new leader is selected from G. The new leader participates in global round by sending a reply to the initiation message.

Performance Evaluation

The following is the analytical results of our proposed algorithm in terms of number of message exchanges to reach consensus.

N = number of processors in a network

G = number of partitioned groups in a network

n = number of processors in each group that means n=

Tf = total number of faulty processes in each group

m = number of messages

Message exchange for local round,

TmL = n (n - 1)

TmL represents the number of message exchange required to get for local consensus. In each group, there is assumed that n processors. Each processor i send its own message to all others and others sent back.

Message exchange for global round,

TmG = G (G-1) (1)

TmG represents the number of message exchange required to get global consensus.

Message exchange for coordinator,

TmC = n - Tf (2)

TmC represents the number of message exchange required by the coordinator of each groups to inform the global decision to all members except the faulty processes belonging to that group.

For m = 1,

Total message exchange,

TmT = TmL + TmG + TmC (3)

= n(n - 1) + G (G-1) + n- Tf

= n2 + G2 - Tf

For m = q,

Total message exchange,

TmT = n2 + G2 - Tf + q (4)

One important impact for this consensus approach is the number of partitions describe in following Table 1.

It describes that different number of processors N is partitioned into different number of Groups G. In our experiment, we assumed that the partitioned group is started from 3 and initialized number of processors in a network is 40.

We found that the number of message exchanges reduces with the increase in number of partitions. This continues up to a threshold limit. If the number of partitions is close to the number of processes, the partitioning can't reduce the message exchange overhead. According to the analysis results, the threshold value for the number of group for partitions is .

G

Table 1. Performance Evaluation

Total Message Exchange (TmT)

N

G

40

50

60

70

80

90

100

110

120

130

140

150

3

268

296

407

536

736

907

1096

1376

1607

1856

2216

2507

4

114

183

239

338

414

543

639

798

914

1103

1239

1458

5

87

123

167

219

279

347

423

507

599

699

807

923

6

83

98

134

178

203

259

323

358

434

518

563

659

7

83

96

111

147

168

216

243

303

336

408

447

488

8

87

98

134

178

203

259

323

358

434

318

386

423

9

95

115

128

143

162

183

231

258

287

275

335

368

10

114

123

134

147

162

179

198

219

242

267

294

323

11

135

144

144

155

168

183

200

219

240

263

288

315

12

151

158

167

178

191

206

206

223

242

263

286

311

13

176

183

194

192

203

216

231

231

242

267

288

311

14

203

210

210

210

230

230

243

258

275

275

294

315

15

232

232

239

248

248

259

272

272

287

304

304

323

Conclusion

This faster consensus agreement approach is highly fault tolerant. It can be great used in any distributed system. The two round of message exchange is sufficient to reach an agreement in a fully connected network when the total number of process failure is less than or equal to (n-1)/2. Where ‘n’ is the total number of processors in a given network. This proposed system can detect the traitor in short visit. Moreover, they can detect the faulty process in both rounds when their messages are exchanged. Then, consensus agreement value can be quickly decide. A network partitioning scheme reduces message exchange overhead. However, we need to emphasis the threshold value for partitioning the members. According to analysis results the number of message exchanges can reduce only in limited number of partitions.