Access the full text.
Sign up today, get DeepDyve free for 14 days.
Federated Byzantine Agreement Systems (FBASs) are a fascinating new paradigm in the context of consensus protocols. Originally proposed for powering the Stellar payment network, FBASs can instantiate Byzantine quorum systems without requiring out-of-band agreement on a common set of validators; every node is free to decide for itself with whom it requires agreement. Sybil-resistant and yet energy-efﬁcient consensus protocols can therefore be built upon FBASs, and the “decentral- ity” possible with the FBAS paradigm might be sufﬁcient to reduce the use of environmentally unsustainable proof-of-work protocols. In this paper, we ﬁrst demonstrate how the robustness of individual FBASs can be determined, by precisely deter- mining their safety and liveness buffers and therefore enabling a comparison with threshold-based quorum systems. Using simulations and example node conﬁguration strategies, we then empirically investigate the hypothesis that while FBASs can be bootstrapped in a bottom-up fashion from individual preferences, strategic considerations should additionally be applied by node operators in order to arrive at FBASs that are robust and amenable to monitoring. Finally, we investigate the reported “open-membership” property of FBASs. We observe that an often small group of nodes is exclusively relevant for deter- mining liveness buffers and prove that membership in this top tier is conditional on the approval by current top tier nodes if maintaining safety is a core requirement. Keywords Byzantine quorum systems · Asymmetric trust · Byzantine faults · Consensus · Stellar · Blockchain 1 Introduction suggest a middle way: Each node deﬁnes its own rules about which groups of nodes it will consider as sufﬁcient validators. We study Federated Byzantine Agreement Systems (FBASs), If the sum of all such conﬁgurations fulﬁlls a set of properties, as originally proposed by Mazières [16]. FBASs are con- protocols like the Stellar Consensus Protocol (SCP) [16] can ceptually related to Asymmetric Quorum Systems [2] and be deﬁned that leverage the resulting structure for establish- Personal Byzantine Quorum Systems [14]. While research ing a live and safe consensus system [3,8,9,13,14]. on consensus protocols has accelerated in the wake of In the original FBAS model [16], which this paper is global blockchain enthusiasm, developments still mostly based on, these properties are foremost quorum availabil- fall in two extreme categories: permissionless, i.e., open- ity despite faulty nodes, which enables liveness, and quorum membership, as exempliﬁed by Bitcoin’s notoriously energy- intersection despite faulty nodes, which makes it possible for hungry “Nakamoto consensus” [17], and permissioned, with consensus protocols to prevent forks and thus enables safety. a closed group of validators, as assumed both in the classi- In a practical deployment, it is seldom clear which nodes are cal Byzantine fault tolerance (BFT) literature (e.g., [4]) and faulty, and in this way the level of risk w.r.t. to liveness and many state-of-the art protocols from the blockchain world safety is uncertain. We propose an intuitive and yet precise (e.g., [22]). The FBAS paradigm and the works it has inspired analysis approach for determining the level of risk, based on enumerating minimal blocking sets and minimal splitting sets—minimal sets of nodes that, if faulty, can by themselves B Martin Florian martin.ﬂorian@hu-berlin.de compromise liveness and safety. We provide algorithms for determining these sets in arbitrary FBASs and make available Humboldt-Universität zu Berlin / Weizenbaum Institute, Berlin, Germany Technische Universität Darmstadt, Darmstadt, Germany 123 400 M. Florian et al. an efﬁcient software-based analysis framework . To the best network (Appendix B) and the current Stellar network of our knowledge, we are the ﬁrst to propose and implement (Appendix C). an analysis methodology for the assessment of the liveness and safety guarantees of FBAS instances that yields precise results as opposed to heuristic estimations. As previously 2 Related work shown in [8], FBASs induce Byzantine quorum systems as per Malkhi and Reiter [15]—hence our results might be of Federated Byzantine Agreements Systems were ﬁrst proposed interest to more classical formalizations as well. For exam- in [16], together with the Stellar Consensus Protocol (SCP), ple, we explicitly distinguish between sets of nodes that can a ﬁrst protocol for this setting. The viability of SCP has been undermine liveness and such sets that can undermine safety, proven formally [8,9,13] and the protocol is in active use in highlighting that in an actual system the threat to liveness and two large-scale payment networks [13,18]. The FBAS notion the threat to safety can differ both in structure and in severity. has furthermore been generalized and reformulated in dif- We apply our analysis approach and tooling in an empir- ferent ways, creating bridges to more classical models and ical study that investigates the emergence of FBASs from enabling the development of additional protocols [2,3,14]. existing inter-node relationships, as encoded in, e.g., trust Among other things, as shown by García-Pérez and Gots- graphs. Based on example conﬁguration policies, we demon- man [8], FBASs with “safe” conﬁgurations induce Byzantine strate that while FBASs can be bootstrapped in a bottom-up quorum systems [15]. In this work, we are less interested fashion from individual preferences, strategic considerations in the mechanics of speciﬁc protocols for the FBAS set- should additionally be applied by node operators in order to ting but instead investigate the conditions they require for arrive at FBASs that are robust and amenable to monitoring. achieving safety, liveness and performance. We investigate Strategic considerations can increase centralization, on how many node failures (and of which nodes) an FBAS can top of what is already implied by individual preferences. We tolerate before the conditions to safety and liveness are com- observe that centralization manifests as a top tier of nodes promised, and how individual node conﬁguration policies that is solely relevant when determining liveness buffers. We inﬂuence these “buffers”. contribute a proof that if maintaining basic safety guarantees Previously, consensus protocols relevant in practice (such is a minimal strategic requirement of node operators, top as PBFT [4]) have relied on a symmetric threshold model. tiers are effectively “closed-membership” in the sense that In a typical instantiation with 3 f + 1 nodes that can toler- a top tier’s composition can only change with cooperation ated up to f Byzantine node failures, each 2 f + 1 nodes of current top tier nodes. This casts doubt on the reported form a (minimal) quorum. This model naturally gives rise to “open-membership” property of FBASs—while any node quorum systems that are trivial to analyze, i.e., for which it can become part of the FBAS, our results show that only is trivial to determine under which maximal fail-prone sets nodes approved by the current top tier can become relevant [15] consensus is still possible. The possibility for quorum for consensus. systems that lack symmetry (that is opened up by the FBAS Following an overview of related work (Sec. 2) and the paradigm and related notions) makes the investigation of a formal introduction of the FBAS model and its interpretation more general analysis approach necessary. in practical deployments (Sec. 3), we structure our paper A heuristics-based methodology for analyzing FBAS around our main original contributions: instances was previously proposed in [11], focusing on the identiﬁcation of central nodes and threats to FBAS liveness. – An analysis framework for reasoning about safety and We propose a novel analysis approach that is not heuristics- liveness guarantees in concrete FBASs (Sec. 4). based and hence yields precise insights, based on a solid – Algorithms for efﬁciently performing the proposed anal- theoretic foundation. As in [11], we apply our methodology yses (Sec. 5). to snapshots of the live Stellar network (cf. Appendix C). – A simulation-based exploration of possible conﬁguration Bracciali et al. [1] explore fundamental bounds on the policies and their effects (Sec. 6). decentrality in open quorum systems. One of their cen- – Formal proof that membership in an FBAS’ top tier is tral arguments with regards to the FBAS paradigm is that only “open” if a violation of safety is considered accept- quorum intersection, a crucial requirement to guaranteeing able (Sec. 7). safety in protocols like SCP, is computationally intractable to determine and maintain, necessitating centralization if safety As appendices, we prove a number of additional corol- is a requirement. The NP-hardness of determining quorum laries and theorems (Appendix A) and present results from intersection was previously also proven by Lachowski [12], applying our analysis methodology to an interesting toy together, however, with practical algorithms for nevertheless determining safety-critical properties of non-trivial FBASs. https://github.com/wiberlin/fbas_analyzer We develop new algorithms that incorporate the possibility 123 The sum of its parts: Analysis of federated... 401 that some nodes may fail, enumerating minimal blocking sets FBAS must exhibit in order to enable liveness and safety and minimal splitting sets. We evaluate their performance for guarantees. different FBAS sizes, providing insights into the computa- tional limitations that are relevant in practice. While, based on 3.1 Quorum slice and FBAS our analysis approach and its application to speciﬁc FBASs, we can conﬁrm that nodes of higher inﬂuence (top tier nodes In an FBAS, each node (respectively its human administra- according to our choice of words) naturally emerge, we argue tor) individually conﬁgures which other nodes’ opinions it that it is not only the existence and size of such a group should consider when participating in consensus. Conﬁgu- that determines “centralization” but also the ﬂuidity of that rations can express individual expectations, such as “out of group’s membership (which we explicitly investigate). these n nodes, at most f will simultaneously cooperate to An alternative analysis methodology and software frame- attack the system”, and can be used to strategically inﬂuence work has recently been presented in [10]. Among other global system parameters. On a conceptual level, the conﬁg- things, the authors provide algorithms for determining the uration of an FBAS node consists in the deﬁnition of quorum consequences of speciﬁc sets of nodes becoming faulty, slices. whereas we propose and implement approaches for identify- ing all minimal sets of nodes that need to become faulty for Deﬁnition 3.1 (FBAS; adapted from [16]) A Federated Byzan- an FBAS to lose safety and liveness guarantees. tine Agreement System (FBAS) is a pair (V, Q) comprising a set of nodes V and a quorum function Q : V → 2 specify- ing quorum slices for each node, where a node belongs to all of its own quorum slices—i.e., ∀v ∈ V, ∀q ∈ Q(v), v ∈ q. 3 Federated byzantine agreement Informally, each quorum slice of a node v describes a set In the following, we introduce core concepts of the FBAS of nodes that, should they all agree to externalize a value in paradigm that form our basis for reasoning about speciﬁc a given slot, is sufﬁcient to also cause v to externalize that FBAS instances. We use terminology based on [12,13,16] value. and the Stellar codebase (stellar-core). Clearly, an FBAS cannot be modeled as a regular graph Our FBAS model is based on the concept of nodes. (with FBAS nodes as graph edges) without losing informa- Whereas nodes usually represent individual machines, for tion. Graph-based analyses as in [11] can therefore result only the purposes of this paper we typically assume that each node in heuristic insights. An FBAS can be modeled as a directed represents a distinct entity or organization. We will illustrate hypergraph [7]. However, we ﬁnd the quorum set abstraction introduced concepts using examples, with nodes represented (presented next) more suitable for subsequent analysis. In as integers. For example, {0, 1, 2} denotes a set of three dis- Sec. 6, we explore strategies for bootstrapping robust FBASs tinct nodes. We will occasionally also use established terms from graphs. in the context of consensus protocols, such as “slot”, “exter- nalize” and “faulty”, without formally introducing them. As 3.2 Quorum set an informal and approximate adaptation to the blockchain setting, a slot is a block of a given height, to externalize a While a useful abstraction for formally describing protocols value is to decide the contents of a block , and a faulty node is for the FBAS setting, quorum slices are an unwieldy format one that violates protocol rules in arbitrary ways, e.g., assum- for describing concrete FBAS instances. In Stellar, the cur- ing the worst-case scenario, via being under the control of an rently most relevant practical deployment of an FBAS, nodes attacker that also controls all other faulty nodes. are conﬁgured not via quorum slices but via quorum sets [13]. We ﬁrst introduce the formal foundation of the FBAS Each quorum set deﬁnes a set of validator nodes U ⊆ V,a paradigm as originally proposed in [16]. Following that, set of inner quorum sets I and a threshold value t. Intuitively, we formally deﬁne the quorum set conﬁguration format this representation enables the encoding of notions such as for FBAS nodes that was previously only used in a prac- “out of these nodes U, at least t must agree” (satisfying the tical implementation (of the Stellar network software) but quorum set) or “the sum of agreeing nodes in U and satisﬁed whose convenience for deﬁning speciﬁc FBAS instances also inner quorum sets in I must be at least t”. beneﬁts the theoretical discussion. Based on the introduced foundations, we ﬁnally derive the necessary properties an Deﬁnition 3.2 (quorum set; adapted from Stellar codebase) A quorum set is a recursive tuple (U , I, t ) ∈ D, D := V D + 2 × 2 × Z . For quorum sets of the form D = (U , I, t ), Consensus protocols for the FBAS setting typically provide immedi- we recursively deﬁne that a set of nodes q ⊆ V satisﬁes D ate ﬁnality, in the sense that once the value for a slot has been externalized, it cannot be reverted or changed. iff (|q ∩ U|+|{I ∈ I : q satisﬁes I }|) ≥ t. 123 402 M. Florian et al. For example, ({0, 1}, ∅, 1) encodes that agreement is Deﬁnition 3.4 (quorum [16]) A set of nodes U ⊆ V in FBAS required from either node 0 or node 1, whereas ({0}, I, 1) (V, Q) is a quorum iff U =∅ and U contains a quorum slice with I ={({1, 2, 3}, ∅, 2)} encodes that either node 0 or two for each member—i.e., ∀v ∈ U ∃q ∈ Q(v) : q ⊆ U. out of {1, 2, 3} must agree. Inner quorum sets (members of This is equivalent to stating that U satisﬁes the quorum I) are often used for grouping nodes belonging to the same sets of all v ∈ U. Quorums are therefore determined by the entity (respectively organization), so that the importance of sum of all individual quorum set conﬁgurations. Continu- an entity can be decoupled from the number of nodes it con- ing the previous example with nodes V ={0, 1, 2}, we get trols. the quorums U ={{0, 2}, {0, 1, 2}}. We capture part of the Quorum sets are useful for deﬁning the quorum slices of semantics behind quorums by deﬁning what it means for a a node. To ease notation, we deﬁne the formalism qset(v, D) consensus protocol to honor a given FBAS —namely that that expresses the set of quorum slices of a node v ∈ V based whenever values are externalized for a slot, at least one quo- on a quorum set D ∈ D. rum of nodes must eventually externalize values as well. Deﬁnition 3.3 (quorum set → quorum slices) For a node v ∈ Deﬁnition 3.5 (protocol that honors an FBAS)Let (V, Q) V and a quorum set D ∈ D,qset(v, D) maps to the set of be an FBAS such that V contains only non-faulty nodes, P all valid quorum slices for v that satisfy D, i.e., qset(v, D) : a consensus protocol, and N ⊆ V the set of all nodes that, V i V × D → 2 := {q ⊆ V | v ∈ q ∧ q satisﬁes D}. following P, eventually externalize a value for a given slot i. We say that P honors (V, Q) iff any nonempty N contains a Via the qset notation, quorum sets and quorum slices quorum, i.e., ∀i : N =∅∨∃U ⊆ N such that U is a quorum become equivalent representations that can be transformed for (V, Q). into one another. A straightforward (but generally not space- efﬁcient) way to express any k quorum slices {q ∈ 2 |} i We say that (V, Q) has quorum availability despite faulty i ∈[0, k), v ∈ q of a node v ∈ V via a quorum set is nodes iff there exists a U ⊆ V that is a quorum in (V, Q) qset(v, (∅, I, 1)), with I ={(q , ∅, |q |) | i ∈[0, k)}. Quo- i i and consists of only non-faulty nodes. Quorum availability rum sets are translated to quorum slices (values of Q)by despite faulty nodes is a necessary condition to achieving applying the qset function. For example (with V ={0, 1, 2}): liveness in an FBAS, i.e., ensuring that non-faulty nodes can externalize new values independently of the behavior Q(0) = qset(0,({1, 2}, ∅, 1)) ={{0, 1}, {0, 2}, {0, 1, 2}} of faulty nodes [16]. Q(1) = qset(1,({0, 2}, ∅, 2)) ={{0, 1, 2}} Theorem 3.1 (quorum availability ⇐ liveness) Let (V, Q) Q(2) = qset(2,({0, 1, 2}, ∅, 2)) ={{0, 2}, {1, 2}, {0, 1, 2}} be an FBAS and P a consensus protocol that honors (V, Q). If P can provide liveness for (V, Q) independently of the In the above example, V ={0, 1, 2} and their quorum sets behavior of faulty nodes, then (V, Q) enjoys quorum avail- (as per Q) form the FBAS (V, Q). As a way to visualize ability despite faulty nodes. (V, Q), it can heuristically be represented as a graph where Proof Let F ⊆ V be the set of all faulty nodes and (V \F , Q ) the existence of an edge (v ,v ) implies that v is included i j j a sub-FBAS that contains all non-faulty nodes, with Q (v) := in at least one of v ’s quorum slices: {q ∈ Q(v) | q ⊆ V \F } for ∀v ∈ V \F. P honors (V, Q) and can provide liveness independently of the behavior of nodes in F, therefore there must exist a protocol P that can provide 1 2 liveness while honoring (V \F , Q ). Based on Def. 3.5, there is therefore at least one U ⊆ V \F that is a quorum for (V \F , Q ). U is, trivially, also a quorum for (V, Q). 3.3 Preconditions to liveness Given quorum availability despite faulty nodes, proto- A consensus system is live if it can externalize new values . cols like SCP can provide liveness [16]. In the case of SCP, A consensus system built upon an FBAS is live if the FBAS this was previously demonstrated through correctness proofs contains an intact quorum— a group of FBAS nodes that can [9] as well as formal veriﬁcation and practical deployment externalize new values by itself. experience [13]. Additional conditions to achieving live- ness include the reaction (via quorum set adaptations, i.e., We content ourselves with a weak notion of liveness whereby a changes to Q) to (detectable) timing attacks [13]. We defer system is live as long as it is non-blocking [9] for one or more non- to works such as [2,3,14,16] for an in-depth exploration of faulty nodes, i.e., as long as an execution path exists that allows one the mechanics and guarantees of consensus protocols for the or more non-faulty nodes to make progress. This can also be called plausible liveness. FBAS setting. 123 The sum of its parts: Analysis of federated... 403 3.4 Preconditions to safety A set of nodes in an FBAS enjoy safety if no two of them ever externalize different values for the same slot [16]. In a blockchain context, a lack of safety guarantees translates into the possibility of forks and double spends. Protocols that honor an FBAS can only guarantee safety if the FBAS enjoys Fig. 1 Example FBAS (V, Q) quorum intersection. Deﬁnition 3.6 (quorum intersection [16]) A given FBAS (U , Q ) and (U , Q ) can externalize values for the same 1 1 2 2 enjoys quorum intersection iff any two of its quorums share slots without any communication taking place between nodes a node—i.e., for all quorums U and U , U ∩ U =∅. 1 2 1 2 in U and nodes in U . 1 2 As P is non-trivial, the externalized values can differ, i.e., For example, the set of quorums {{0, 2}, {0, 1, 2}} inter- safety cannot be guaranteed. sects, whereas introducing an additional quorum {1, 4} would break quorum intersection. In the latter scenario, {0, 2} and As formally proven by García-Pérez and Gotsman [8], {1, 4} could induce two new, separated FBASs [14]. We say an FBAS that enjoys quorum intersection induces a Byzan- that an FBAS enjoys quorum intersection despite faulty nodes tine quorum system [15], and an FBAS that enjoys quorum if every two quorums that contain non-faulty nodes intersect intersection despite faulty nodes can induce a dissemina- in at least one non-faulty node, even if all faulty nodes change tion quorum system [15]. These results are independent of their quorum sets in arbitrary ways or report different quorum attempts by faulty nodes to lie about their quorum set conﬁg- sets to different peers. Formally, quorum intersection despite uration [8]. There is strong evidence that protocols like SCP faulty nodes is deﬁned via a delete operation that transforms can guarantee safety in any FBAS with quorum intersection an FBAS based on the assumption that a given set of nodes despite faulty nodes [2,9,13,14]. is acting in the most harmful (to safety) way possible. Deﬁnition 3.7 (delete [16]) If (V, Q) is an FBAS and F ⊆ V 4 Concepts for further analysis a set of nodes, then to delete F from (V, Q), written (V, Q) , means to compute the modiﬁed FBAS (V \F , Q ) where In the following, we deﬁne new concepts for capturing rel- Q (v) ={q \ F , q ∈ Q(v)}. evant properties of concrete FBAS instances. While it is typical in the BFT literature to construct proofs based on If F ⊆ V is the set of all faulty nodes, then an FBAS assuming which sets of nodes can fail simultaneously (i.e., (V, Q) enjoys quorums intersection despite faulty nodes iff which are the fail-prone sets [15]), we instead investigate (V, Q) enjoys quorum intersection. If quorum intersection which sets of nodes have to fail in order for global live- despite faulty nodes is not given, safety cannot be guaranteed ness and safety guarantees to become void. This perspective (although it can be maintained by chance). uncovers the liveness and safety buffers a given (potentially non-trivial) quorum system has and is thus highly relevant for Theorem 3.2 (quorum intersection ⇐ guaranteed safety) the monitoring and evaluation of systems deployed in prac- Let (V, Q) be an FBAS and P a consensus protocol that tice. While deﬁned based on the FBAS model, the proposed can provide liveness for any FBAS with quorum availability concepts are readily transferable to more general quorum sys- despite faulty nodes, while honoring the respective FBAS. tem formalizations (e.g., recall that safety-enabling FBASs Let P furthermore be non-trivial, in the sense that external- induce Byzantine quorum systems [8]). ized values are non-deterministic and depend on user input. For illustration, we will be using the example FBAS If P can guarantee safety for all non-faulty nodes in V, then deﬁned via Fig. 1. An analysis of a slightly larger example (V, Q) enjoys quorum intersection despite faulty nodes. FBAS is presented in Appendix B. Appendix A contains for- mal write-ups and proofs of various corollaries and theorems Proof Let F ⊆ V be the set of all faulty nodes and relevant to this section. (V , Q ) := (V, Q) .If (V , Q ) does not enjoy quorum intersection, then there are two quorums U , U ⊂ V so 1 2 that U ∩ U =∅.For i ∈{1, 2},let Q be deﬁned such 4.1 Starting point: Minimal quorums 1 2 i that ∀v ∈ U : Q (v) := {q ∈ Q (v) | q ⊆ U }. Then both i i i (U , Q ) and (U , Q ) form FBASs with quorum availabil- As a prerequisite to subsequent analyses, it is helpful to 1 1 2 2 ity. As P can provide liveness for any FBAS with quorum understand which quorums (cf. Def. 3.4) exist in an FBAS. availability, We will be focusing on minimal quorums, i.e., quorums 123 404 M. Florian et al. ˆ ˆ U ⊆ V for which there is no proper subset U ⊂ U that is in the context of Stellar, the blocking set could block the also a quorum. Informally, the set of all minimal quorums ratiﬁcation of transactions involving speciﬁc accounts. We U carries sufﬁcient information for precisely determining chose the term blocking in analogy to the v-blocking sets FBAS-wide liveness properties, while being of signiﬁcantly introduced in [16]. As an important distinction, we use the smaller size than the set of all quorums U. term blocking set to refer to a property of the whole FBAS (V, Q), as opposed to a property of an individual node v ∈ V. Deﬁnition 4.1 (minimal node set) Within the set of node sets In the above example, {0} and {1, 3} are not only blocking N ⊆ 2 , a member set N ∈ N is minimal iff none of its sets with respect to U,theyare minimal blocking sets, i.e., proper subsets is included in N —i.e., ∀N ∈ N , N ⊂ N . 4 none of their proper subsets is a blocking set . In essence, minimal blocking sets describe minimal threat (respectively, The FBAS depicted in Fig. 1 has the quorums U = fail) scenarios w.r.t. liveness. {{0, 1, 2}, {0, 3, 4}, {0, 1, 2, 3, 4}} and consequently the min- imal quorums U ={{0, 1, 2}, {0, 3, 4}}. 4.3 Minimal splitting sets The notion of minimal quorums is helpful, among other things, for efﬁciently determining whether an FBAS enjoys As per Thm. 3.2, an FBAS can only be considered safe (as quorum intersection [12]: it can be shown that an FBAS one coherent system) as long as it enjoys quorum intersection enjoys quorum intersection iff every two of its minimal quo- despite faulty nodes, i.e., as long as each two of its quorums rums intersect (Cor. A.1). intersect even after all faulty nodes have been deleted (as per Def. 3.7). For practical purposes, quorum intersection 4.2 Minimal blocking sets despite faulty nodes is furthermore a sufﬁcient condition for achieving safety in an FBAS, considering protocols like SCP As per Thm. 3.1,anFBAS (V, Q) cannot enjoy liveness if it and the correctness proofs surrounding them (s.a. Sec. 3.4). doesn’t contain at least one non-faulty quorum. Considering Hence, for assessing the risk to safety, it is interesting to iden- the state of the art in consensus protocols for the FBAS setting tify sets of nodes that can cause an FBAS to effectively lose and their formal veriﬁcation (s.a. Sec. 3.3), quorum availabil- quorum intersection. We call such a set of nodes a splitting ity despite faulty nodes is furthermore the only precondition set, as it can, if faulty, cause at least two quorums to diverge, to achieving liveness that depends on (V, Q) and arguably the splitting the FBAS. most difﬁcult to satisfy in a practical deployment. However, while quorum availability can easily be checked based on Deﬁnition 4.3 (splitting set) We denote the set S ⊆ V asplit- Q, faulty nodes are usually not readily identiﬁable as such ting set iff (V, Q) lacks quorum intersection—i.e., there are in practice. We therefore propose, as a means to grasping distinct quorums U and U of (V, Q) so that U ∩ U =∅. 1 2 1 2 liveness risks, to look at sets of nodes that, if faulty, can undermine quorum availability. In the above example with U ={{0, 1, 2}, {0, 3, 4}}, {0} {0} is already a splitting set, as (V, Q) induces the two non- Deﬁnition 4.2 (blocking set)Let U ⊆ 2 be the set of all intersecting quorums {1, 2} and {3, 4}. Intuitively, {0} is a quorums of the FBAS (V, Q). We denote the set B ⊆ V as splitting set of (V, Q) because it forms the intersection of blocking iff it intersects every quorum of the FBAS—i.e., the quorums {0, 1, 2} and {0, 3, 4}. ∀U ∈ U , B ∩ U =∅ The existence of a faulty splitting set violates quo- For example: {0} and {1, 3} are both blocking sets for rum intersection despite faulty nodes and therefore, as per U ={{0, 1, 2}, {0, 3, 4}, {0, 1, 2, 3, 4}}. Thm. 3.2, threatens safety. Informally, the members of a splitting set can perform two types of actions to compromise Corollary 4.1 (blocking sets and liveness) Control over any safety in practice (s.a. Thm. A.1). On the one hand, they can blocking set B is sufﬁcient for compromising the liveness of change their quorum conﬁgurations (or lie about them) to an FBAS (V, Q). cause existing quorums to shrink or new quorums to emerge, both with the goal of reducing the overlap between quorums. Proof As B intersects all quorums of the FBAS, there is no On the other hand, whenever the intersection of two (min- quorum that can be formed without cooperation by B. With- imal) quorums is comprised entirely of faulty nodes, these out at least one non-faulty quorum, liveness is not possible nodes can agree to different statements in each quorum, caus- as per Thm 3.1. ing the quorums to externalize conﬂicting values and in this way diverge. Notably, blocking sets can also block liveness selectively, enabling censorship. As nodes from the blocking set are present in every quorum, consensus will never be reached For completeness, the set of all minimal blocking sets w.r.t. U is on any value that the blocking set opposes to. For example, B ={{0}, {1, 3}, {1, 4}, {2, 3}, {2, 4}}. 123 The sum of its parts: Analysis of federated... 405 As with blocking sets, we are especially interested in ﬁnd- 5 Analysis algorithms V 5 ing the minimal splitting sets S ⊂ 2 of an FBAS (V, Q). Minimal splitting sets describe minimal threat scenarios w.r.t. In the following, we propose algorithms for performing the safety. analyses introduced in Sec. 4. We describe them as pseu- docode that necessarily abstracts away some implementation 4.4 Top tier details and optimizations. As a companion to this paper, we release a well-tested implementation of the presented algo- For narrowing down notions of “centralization” with respect rithms as open source (fbas_analyzer ). After outlining to FBASs, we propose the concept of a top tier. Informally, algorithms for enumerating minimal quorums (foundation the top tier is the set of nodes in the FBAS that is exclusively for further analyses), determining quorum intersection (nec- relevant when determining minimal blocking sets and hence essary condition for safety), enumerating minimal blocking the liveness buffers of an FBAS. sets (liveness “buffers”), enumerating minimal splitting sets (safety “buffers”), and efﬁciently dealing with symmetric top Deﬁnition 4.4 (top tier) The top tier of an FBAS (V, Q) is tiers, the section concludes with a short empirical study on the set of all nodes that are contained in one or more minimal analysis scalability. quorums—i.e., if U ⊆ 2 is the set of all minimal quorums of the FBAS, T = U is its top tier. 5.1 Minimal quorums In the above example, it in fact holds that T ={0, 1, 2, 3, 4} = V. Algorithm 1 describes a branch-and-bound algorithm for It can be shown that each minimal blocking set consists ﬁnding all minimal quorums. It is based on a quorum enu- exclusively of top tier nodes (Cor. A.5), and each top tier node meration procedure originally described in [12]. Previous is included in at least one minimal blocking set (Thm. A.2). algorithms did not rigorously ﬁlter out non-minimal quo- The FBAS (V, Q) with top tier T has therefore the same rums, which we realize through is_minimal_quorum. properties w.r.t. global liveness as the FBAS induced by T , The set of all minimal quorums of an FBAS deﬁnes its top i.e., the FBAS (T , Q ) with Q (v) := {q ∩ T | q ∈ Q(v)}. tier (cf. Sec. 4.4) and can be used for determining whether This observation has direct implications for the com- the FBAS enjoys quorum intersection. putational complexity of FBAS analysis (further discussed The keystone of the algorithm is the function fmq_step in Sec. 5), and for the performance of FBAS-based con- that takes a current quorum candidate U, a sorted list of yet- sensus protocols. A consensus round in SCP (the so far to-be-considered nodes V and a reference to Q for mapping only production-ready protocol for the FBAS setting, to the nodes to their quorum sets. The algorithm implements a clas- best of our knowledge) can demonstrably be completed in sical branching pattern: at each invocation of fmq_step O(|T | ) messages. While classical consensus protocols with in which U is not already a quorum, the next node in V quadratic message complexity (such as PBFT [4]) are notori- is taken out and, in one branch, added to U, and, in the ous for becoming unusable in larger validator groups, several other, not. Hopeless branches are identiﬁed early using the improved protocols have recently emerged that target the is_satisfiable function. blockchain use case and scenarios with 100 and more valida- As proposed in [12], we initially sort V using a heuristic tors [20,22]. As a possible avenue for future exploration— such as PageRank [19] which can improve the algorithm’s for FBASs with a symmetric top tier, existing permissioned performance in practice. Another important optimization protocols could be adapted without much modiﬁcation. from [12], that we leave out in our pseudocode for greater Deﬁnition 4.5 (symmetric top tier) The top tier T of an FBAS clarity, is the partitioning of V into strongly connected (V, Q) is a symmetric top tier iff all top tier nodes have components so that find_minimal_quorums must be identical quorum sets—i.e., ∃D ∈ D, ∀v ∈ T : Q(v) = applied only to (often signiﬁcantly smaller) subsets of V.Tar- qset(v, D). jan [21] gives an algorithm for performing this preprocessing step in linear time. Symmetric top tiers are also signiﬁcantly more amenable As noted in other works (e.g., [1,12]), determining to analysis. For example, in FBASs with a symmetric top tier quorum intersection, and hence also enumerating all min- T and a non-nested top tier quorum set (T , ∅, t ), it holds that any minimal blocking set has cardinality |B|=|T |− t + 1 (Thm. A.3) and any minimal splitting set that can cause two 6 https://github.com/wiberlin/fbas_analyzer; Our Rust-based library top tier nodes to diverge from each other has cardinality |S|= has been integrated into https://stellarbeat.io/ (a popular monitoring service for the Stellar network) and supports in-browser usage—cf. 2t −|T | (Thm. A.4). our interactive analysis website at https://trudi.weizenbaum-institut.de/ stellar_analysis/. In the above example, {0} is the only minimal splitting set w.r.t. U, i.e., the set of all minimal splitting sets is S ={{0}}. Based on the heuristic representation of the FBAS as a directed graph. 123 406 M. Florian et al. Algorithm 1: Find minimal quorums Algorithm 2: Checking for quorum intersection via approach from [12]. 1 Function find_minimal_quorums((V, Q)): Data: An FBAS (V, Q). 1 Function has_quorum_intersection((V, Q)): Result: U, the set of all minimal quorums of (V, Q). Data: An FBAS (V, Q). Result: true if the FBAS enjoys quorum intersection, false 2 V ← V sorted by, e.g., PageRank [19](cf.[12]); else. 3 return fmq_step(∅,V, Q); 2 U ← find_minimal_quorums((V, Q)); 4 Function fmq_step(U, V , Q): ˆ ˆ ˆ ˆ 5 if is_quorum(U, Q) then 3 return ∀U , U ∈ U : U ∩ U =∅; i j i j 6 if is_minimal_for_quorum(U, Q) then 7 return {U }; 8 else return ∅; never the case, the FBAS enjoys quorum intersection. This 9 else if is_satisfiable(U, V , Q) then approach for checking for quorum intersection has the ben- 10 v ← next in V ; 11 return fmq_step(U ∪{v},V \{v}, Q) ∪ eﬁt that only a constant number of node sets must be held in fmq_step(U, V \{v}, Q); memory at the same time, as opposed to all minimal quorum 12 else return ∅; sets as in Algorithm 2. The space complexity of the check is 13 Function is_quorum(U, Q): therefore reduced from exponential to linear. 14 return ∀v ∈ U ∃q ∈ Q(v) : q ⊆ U; 15 Function is_satisfiable(U, V , Q): 16 return ∀v ∈ U ∃q ∈ Q(v) : q ⊆ U ∪ V ; Algorithm 3: Checking for quorum intersection via 17 Function is_minimal_for_quorum(U, Q): alternative approach with linear space complexity. 18 for v ∈ U do 1 Function has_quorum_intersection((V, Q)): 19 if contains_quorum(U \{v}, Q) then Data: An FBAS (V, Q). 20 return false; Result: true if the FBAS enjoys quorum intersection, false 21 end else. 22 return true; 2 for U ∈ find_minimal_quorums((V, Q)) do 23 Function contains_quorum(U, Q): 3 if contains_quorum(V \U) then // remove non-satisfiable nodes 4 return false; 24 while ∃v ∈ U ∀q ∈ Q(v) : q U do 5 end 25 U ←{v ∈ U |∃q ∈ Q(v) : q ⊆ U }; 6 return true; 26 end 27 return U =∅; Our implementation of Algorithm 3 is also empiri- cally faster for many FBASs, probably because contains imal quorums, is NP-hard. Consequently, our algorithm _quorum scales better than iterating once over all minimal has exponential time complexity. For an FBAS with n = quorums, and because less data must be written to memory. |V| nodes and a top tier of size m =|T | we ﬁnd all For both algorithms, we leave out optimization details such as m n k ≤ minimal quorums in O(2 ). Note that in prac- leveraging the fact that quorum intersection is guaranteed to tice the number of de-facto considered nodes n is greatly ˆ ˆ hold if all minimal quorums U ∈ U have cardinality greater reduced through polynomial-time preprocessing steps such | U | than . In Algorithm 3, for example, it sufﬁces to check as strongly-connected-component analysis and heuristics- 2 | U | based sorting, yielding actual running times that are close only minimal quorums with fewer than members. to the O(2 ) bound. 5.3 Minimal blocking sets 5.2 Quorum intersection Algorithm 4 presents our algorithm for enumerating all min- Quorum intersection is a central property for being able to imal blocking sets based on a branch-and-bound strategy. guarantee safety in an FBAS (cf. Sec. 4.3). Quorum intersec- The check whether a given candidate set B is blocking is per- tion can be determined by checking the pairwise intersection formed by checking whether the FBAS contains any quorums of all minimal quorums (Cor. A.1). This straightforward after B is removed from the node population. If a blocking approach, that was also proposed in [12], is embodied in set can still be formed from B and the yet-to-be-considered Algorithm 2. nodes V (this is the pruning rule), the enumeration continues, In this paper, we propose an additional, alternative algo- branching via either adding the next node in V to the candi- rithm (Algorithm 3), that doesn’t check for pairwise inter- date set or discarding it altogether. The order in which nodes sections but instead checks whether the complement sets are visited can be tuned using a suitable heuristic—we sort of found quorums contain quorums themselves. If this is nodes using PageRank [19] (as for ﬁnding minimal quorums) 123 The sum of its parts: Analysis of federated... 407 in the example pseudocode and our current implementation. changing their quorum set, quorum expanders could reverse Like for Algorithm 1, the complexity of Algorithm 4 is in this effect, leading to smaller quorums and, accordingly, an O(2 ) (for an FBAS with n nodes) with a likely practical increased risk to quorum intersection. average case complexity of O(2 ) (m being the size of the The has_potential function embodies an explicit top tier). pruning condition for the branch-and-bound search. Here, we check whether a change in the FBAS’s minimal quo- rums is possible if some or all outstanding candidate nodes Algorithm 4: Find minimal blocking sets V are joined with the current candidate set S. As a heuris- 1 Function find_minimal_blocking_sets((V, Q)): tic to avoid actually calculating minimal quorums, we check Data: An FBAS (V, Q). whether the quorum-containing strongly connected compo- Result: B, the set of all minimal blocking sets of (V, Q). nents of the FBAS change after deleting V in addition to 2 V ← V sorted by, e.g., PageRank [19]; S. 3 return fmb_step(∅,V, Q); For improving readability and comprehension, we leave 4 Function fmb_step(B, V , Q): out various details and smaller optimizations from our pseu- 5 if is_blocking(B, V , Q) then 6 if is_minimal_for_blocking(B, V , Q) then docode listing for Algorithm 5. Among other things, we 7 return {B}; don’t include our full algorithms for enumerating quorum 8 else return ∅; _expanders and deliberately ignore opportunities for 9 else if is_blocking(B ∪ V, V, Q) then caching and reusing the results of costly operations. 10 v ← next in V ; 11 return fmb_step(B ∪{v},V \{v}, Q) ∪ fmb_step(B, V \{v}, Q); 12 else return ∅; Algorithm 5: Find minimal splitting sets 13 Function is_blocking(B, V , Q): 1 Function find_minimal_splitting_sets((V, Q)): Data: An FBAS (V, Q). 14 return ¬contains_quorum(V \ B, Q); Result: S, the set of all minimal splitting sets of (V, Q). 15 Function is_minimal_for_blocking(B, V , Q): 2 V ← find_minimal_quorums((V, Q)); 16 for v ∈ B do 3 V ← V ∪ quorum_expanders((V, Q)); 17 if is_blocking(B \{v},V, Q) then 18 return false; 4 V ← V sorted by, e.g., number of affected nodes; 5 A ← V; 19 end 6 S ← fs_step(∅,V, A, (V, Q)); 20 return true; 7 return reduce_to_minimal_sets(S); 8 Function fs_step(S, V , (V, Q)): 9 if ¬has_quorum_intersection((V, Q) ) then 10 return {S}; 5.4 Minimal splitting sets 11 else if has_potential(S, V , (V, Q)) then 12 v ← next in V ; 13 return fs_step(S ∪{v},V \{v}, (V, Q)) ∪ Algorithm 5 presents our algorithm for enumerating all min- 14 fs_step(S, V \{v}, (V, Q)); imal splitting sets. We again perform a branch-and-bound 15 else return ∅; search. The ﬁnal condition for accepting a candidate set S is 16 Function quorum_expanders((V, Q)): whether deleting it (cf. Def. 3.7) from the FBAS causes the 17 return FBAS to lose quorum intersection. {v ∈ V |∃v ∈ V, q ∈ Q(v ) : v ∈ q ∧ (∀q ∈ Q(v) : q q )}; This check is signiﬁcantly more expensive than the corre- sponding checks in Algorithm 1 and Algorithm 4. Addition- 18 Function has_potential(S, V , (V, Q)): ally, unlike the previously presented algorithms, Algorithm 5 19 return (∃v ∈ V ∃ ∈ Q(v) : q ⊆ S ∪ V ) ∨ S∪V also needs to consider non-top tier nodes as candidates. We 20 (quorum_clusters((V, Q) ) quorum_clusters((V, Q) )); incorporate the observation (from Thm. A.1) that a node can only be part of a minimal splitting set if it is part of a min- 21 Function quorum_clusters((V, Q)): 22 N ← strongly connected components of (V, Q); imal quorum (only then can it be part of an intersection of 23 return {N ∈ N | contains_quorum(N)}; minimal quorums) or if a change of its quorum set can poten- 24 Function reduce_to_minimal_sets(S): tially cause new, smaller quorums to emerge. Consequently, ˆ ˆ 25 return {S ∈ S |∀S ∈ S : S ⊂ S}; we consider as candidates all top tier nodes and all nodes that are quorum expanders: nodes that are part of a quorum slice of another node that is a not a quorum slice for themselves (formal deﬁnition in Def. A.1). Informally, by not sharing The asymptotic complexity of Algorithm 5 remains in n |T ∪X | a quorum slice with a node they affect, quorum expanders O(2 ), respectively O(2 ) where T is the top tier and X may force quorums to expand beyond this quorum slice. By the set of all quorum expanders. However, due to the costly 123 408 M. Florian et al. acceptance check for splitting sets and the larger number of nodes that need to be considered, the algorithm is signiﬁ- cantly slower than Algorithm 1 and Algorithm 4 in practice. 5.5 Symmetric clusters As a generalization of symmetric top tiers (Def. 4.5), we deﬁne symmetric clusters of an FBAS (V, Q) as groups of nodes Y ⊆ V such that ∃D ∈ D, ∀v ∈ Y : Q(v) = qset(v, D) and {Q(v), v ∈ Y}= Y . If an FBAS has one symmetric cluster Y and V \Y does not contain a quo- rum, Y is the symmetric top tier of (V, Q) . Symmetric clusters can be found in polynomial time, by grouping nodes with identical quorum set conﬁgurations (values for Q) and checking the above condition for each Fig. 2 Analysis duration for FBASs resembling classical 3 f + 1 quo- thus formed candidate set. rum systems. Analysis optimizations for symmetric top tiers were turned off Symmetric clusters can be analyzed signiﬁcantly more efﬁciently. For example, an FBAS with a non-nested sym- metric top tier is isomorphic to a classical, threshold-based quorum system (s.a. Thm. A.3 and A.4). For symmetric clus- ters formed around a nested quorum set, minimal quorums and minimal blocking sets can be enumerated without the overhead of checking candidate sets, by recursively listing combinations and forming their Cartesian product. If the interest is to ﬁnd only such splitting sets that can cause nodes within the symmetric cluster to diverge, then the same is true for minimal splitting sets. 5.6 Analysis performance Our analysis approach requires the enumeration of minimal quorums, minimal blocking sets and minimal splitting sets— which in all three cases is an NP-hard problem. It is unclear, Fig. 3 Analysis duration for FBASs resembling the structure of the Stellar network top tier. Analysis optimizations for symmetric top tiers however, what this means for the practical limitations of were turned off thoroughly determining the safety and liveness buffers of an FBAS. Practical limitations are difﬁcult to conclusively determine as the real-life performance of analyses depends zation is represented by (usually) 3 physical nodes arranged heavily on the topology of analyzed FBASs and the imple- in crash failure-tolerating 2 f + 1 inner quorum sets: mentation of the algorithms. In the following, we present a short exploratory study into V ={v ,v , ...v }, n = 3m 0 1 n−1 the scalability of our own implementation. We construct syn- I ={({v ,v ,v }, ∅, 2) | i ∈[0, m)} thetic FBASs of increasing size that consist of only a top tier. 3i 3i +1 3i +2 In the ﬁrst series of presented experiments (Fig. 2), we con- 2m + 1 ∀v ∈ V : Q(v) = qset(v, (∅, I, )) struct FBASs (V, Q) resembling classical 3 f + 1 quorum systems: We enumerate all minimal quorums, minimal blocking 2|V|+ 1 sets and minimal splitting sets of thus generated FBASs and ∀v ∈ V : Q(v) = qset(v, (V, ∅, )) record the time to completion of each of these operations. All analyses were single-threaded and performed on regular In a second series of experiments (Fig. 3), we approximate the server-class hardware. We explicitly deactivated all optimiza- structure of the Stellar network’s top tier where each organi- tions based on detecting and exploiting symmetric clusters, so that the results of this study reﬂect the performance of the If an FBAS has l > 1 symmetric clusters or V \Y does contain a quorum, (V, Q) does not enjoy quorum intersection. more expensive Algorithms 1, 4 and 5. 123 The sum of its parts: Analysis of federated... 409 Figures 2 and 3 depict the median measured times on a of cardinality 1—any node can block the single quorum in log scale, from a set of 10 measurements per FBAS size (we the induced FBAS. performed the same analysis 10 times, recording individual As an improvement, the threshold of the formed quorum times). As was expected, analysis durations raise exponen- sets can be set in resemblance to classical BFT protocols: tially with growing top tier sizes m. Analyses start requiring more than an hour to ﬁnish at m ≥ 23 for ﬂat symmetric top 2|V|+ 1 ∀v ∈ V : Q(v) = qset(v, (V, ∅, )) tiers and m ≥ 24 for Stellar-like topologies. This is a cau- tiously positive result—top tier sizes observed in practice (Ideal Open QSC) are currently in the range of 7 organizations (23 raw nodes) for the Stellar network (cf. Appendix C) and 7 organizations For |V|= 3 f + 1 with an f ∈ Z , setting the threshold (10 raw nodes) for the MobileCoin network [18]. It is likely 2|V|+1 to t = leads to FBASs in which any 2 f + 1 nodes that, for example through parallelization or the development form a (minimal) quorum. This results in both all minimal of additional optimizations for “almost symmetric” FBASs, blocking sets and all minimal splitting sets of the induced the analysis durations for naturally occurring FBASs can be FBAS having cardinality f + 1, i.e., both safety and liveness reduced further. can be maintained in the face of up to f node failures. 6 Bootstrapping FBASs 6.1.1 Choosing validators The reported openness enabled through the FBAS paradigm The preceding example policies construct non-nested quo- comes at the cost of increased conﬁguration responsibilities rum sets that use as validators U the set of all nodes in the for node operators. As discussed in Sec. 3, each node must FBAS (U = V). These are clearly toy examples—if anything become associated with a quorum set (respectively quorum else, without additional mechanisms to restrict or ﬁlter the slices) in order to become a useful part of an FBAS. We membership in V, V can easily become dominated by faulty will refer to this process as quorum set conﬁguration (QSC). Sybil [5] nodes. But how should a node operator go about QSC? Based on the In the scope of this work, and in line with the motivation analytical toolset introduced in Sec. 4, we can now investigate behind the FBAS paradigm, we consider V to enjoy open what kinds of QSC policies are plausible and in what kind of membership, with no universally trusted whitelist or ranking. FBASs they result. For arriving at sensible choices for U, QSC policies must Notably, we explore how individual preferences (such as therefore take individual knowledge into account. which nodes should be “trusted”) can be mapped to the quo- rum set formalism. Based on experiments that use Internet 6.1.2 Modeling individual preferences topology as a representative graph representation of interde- pendence and trust, we conclude that purely individualistic QSC policies based on individual preferences contribute conﬁguration policies can result in systems with low live- node-local knowledge to the collective FBAS conﬁguration. ness and high complexity. We outline possible directions for For example: future research by sketching policies with a strategic element and empirically demonstrating their effectiveness. – Which nodes are trusted to be (and stay) non-faulty. It 6.1 QSC policies and their evaluation is often implied that QSC should reﬂect some form of trust, e.g., in wordings such as “ﬂexible trust” [16]or “asymmetric distributed trust” [2]. While reasoning about A QSC policy is individually and repeatedly invoked for each node v ∈ V. It takes information about a current FBAS the future behavior of participants in a consensus proto- col might be an overwhelming task for node operators, instance (V, Q) as input and returns a quorum set for v, setting a new value for Q(v). We use the quorum set for- they may at least encode plausible beliefs about non- Sybilness [5] (i.e., which groups of nodes are (un)likely malization introduced in Sec. 3.2. For illustration, consider the following trivial policy: to be controlled by the same entity). – To which nodes do dependencies exist (e.g., for business ∀v ∈ V : Q(v) = qset(v, (V, ∅, |V|)) (Super Safe QSC) reasons). Adding nodes of organizations one interacts with to one’s If implemented by all nodes in V, Super Safe QSC leads to quorum sets might be necessary to maintain “sync” with each node having only one quorum slice—V itself (Q(v) = these organizations [13], as opposed to ending up with {V}). The policy maximizes safety but leads to blocking sets diverging ledgers in the event of a fork. 123 410 M. Florian et al. In the following discussion, we will use graph repre- that do not enjoy quorum intersection . The high prevalence sentations for modeling individual preferences. It is an of AS peering is a likely explanation for why sufﬁciently well intriguing hypothesis that the FBAS paradigm can enable intraconnected clusters can emerge outside of the “natural” Sybil-resistant and yet energy-efﬁcient permissionless con- top tier of the AS graph. sensus by bootstrapping quorum systems along existing trust A lack of quorum intersection implies that the induced graphs or interdependence graphs. In Sec. 3.1 we saw that FBASs may split into multiple sub-FBASs. This might be a transforming an FBAS into an equally sized regular graph desirable effect when bootstrapping from individual prefer- leads to a loss of information, i.e., can yield only heuristic ences. For example, separated communities with low levels representations. In the following sections we pose the inverse of inter-community interaction and trust might prefer the question: How can a “good” FBAS (V, Q) be instantiated added sovereignty of an “own” FBAS. We repeated the anal- from a given graph G = (V, E )? ysis for the respectively largest sub-FBASs, with an upper For evaluating example policies incorporating individual bound on top tier size of, respectively, 355 and 14339 preferences, we will use the autonomous system (AS) rela- nodes. Potential top tier sizes of this magnitude make a tionships graph inferred by the CAIDA project —a reﬂection complete analysis unfeasible (s.a. the discussion on analysis of the interdependence and trust between networks that form scalability in Sec. 5.6). This is problematic, as the robust- the Internet. The topological structure of the Internet has ness of the resulting FBASs, in terms of safety and liveness, repeatedly been cited as an argument for the viability of the cannot be reliably determined. Existing weaknesses in the FBAS model [13,16]. We discuss results based on two snap- global quorum structure cannot be identiﬁed and (strategi- shots of the AS relations graph: from January 1998—the cally) ﬁxed. Weaknesses, however, are likely to exist. For earliest available snapshot describing a younger Internet with example, preliminary analysis results for the FBAS instanti- 3233 ASs connected via 4921 (directed) customer/provider ated from G imply the existence of blocking sets with AS98 links and 852 (undirected) peering links—and from Jan- only 3 members. uary 2020—with 67308 ASs connected via 133864 cus- tomer/provider links and 312763 peering links. We will refer 6.3 Tier-based QSC to the graphs as G and G . AS98 AS20 Towards making resulting top tiers more focused (and hence, the resulting FBASs more efﬁcient and more amenable to 6.2 Naive individualistic QSC analysis), QSC policies can incorporate strategic consider- We consider a QSC policy naively individualistic if it is based ations in addition to individual preferences. We explore a entirely on individual preferences. We model “preference for prudent example strategy in the following: the weighing of a node” as edges in a graph G = (V, E ), with nodes being nodes based on tierness, or relative importance. Tierness aware only of their own graph neighborhood. is an established notion for ASs in the Internet graph. For Consider a simple representative of this class—forming FBASs, a tiered quorum structure with every node including quorum sets using the entire graph neighborhood of a node, only higher-tier neighbors in its quorum sets was proposed weighing each neighbor equally within a 3 f + 1 threshold (as an example) as early as in the original FBAS proposal logic (that models the assumption that strictly less than a [16]. Classifying nodes based on their tierness is also related third of all neighbors can be faulty): to the quality-based conﬁguration format currently used by the Stellar software [13]. Lastly, it is a plausible assumption ∀v ∈ V : U ={v}∪{v ∈ V | (v, v ) ∈ E } that the relative tierness of graph neighbors can be estimated locally, enabling QSC decisions that do not require a global 2|U|+ 1 Q(v) = qset(v, (U , ∅, )) view. We sketch an example QSC policy in which nodes use (All Neighbors QSC) only higher-tier nodes in their quorum sets, or same-tier If G is a complete graph, we get the same result as with nodes if none of their neighbor appears to be of higher Ideal Open QSC.If G is not connected, we cannot have quo- tier. We assume that nodes can infer the relative tierness of rum intersection (and hence safety). The latter is also true if their graph neighbors. Speciﬁcally, that they can determine G contains more than one cluster of sufﬁcient size and weak which of their neighbors are of a higher tier than themselves. (relative) connectedness to the rest of the graph. We can con- For simulation, we use the PageRank [19] score of nodes (calculated without dampening) as a proxy for their tier- ﬁrm that this is the case for the AS graph snapshots G AS98 and G . Using them, All Neighbors QSC induces FBASs AS20 As determined using fbas_analyzer (Sec. 5). 9 11 The CAIDA AS Relationships Dataset, 1998-01-01 (serial-1) and Based on the size of the largest quorum that is fully contained in a 2020-01-01 (serial-2), https://www.caida.org/data/as-relationships/ strongly connected component (which is the union of all such quorums). 123 The sum of its parts: Analysis of federated... 411 ness. Each simulated node considers a neighbor of higher (lower) tier if the neighbor’s PageRank score is twice as high (low) as its own. More formally, with R(v) denoting the PageRank score of node v, edges (v) the set of its neigh- bors (edges (v) := {v ∈ V | (v, v ) ∈ E }), H its higher-tier neighbors and P its same-tier neighbors (“peers”): H (v) ={v ∈ edges (v) | R(v ) ≥ 2R(v)} P(v) ={v ∈ edges (v) | R(v) < R(v )< 2R(v)} (Tierness Heuristics) Based on this heuristic, we can deﬁne the following QSC policy: (a) {v}∪ H (v) if H (v) =∅ ∀v ∈ V : U = {v}∪ P(v) else 2|U|+ 1 Q(v) = qset(v, (U , ∅, )) (Higher-Tier Neighbors QSC) Our results show that improvements to the naive case are possible when incorporating strategic considerations, despite the fact that the quorum structure is heavily inﬂuenced by individual preferences. More prominently—top tiers become of more manageable size (both for analysis and for consensus (b) protocols leveraging the FBAS). Fig. 4 Histogram of the cardinalities of relevant sets in FBASs resulting We simulated the application of Higher-Tier Neighbors from the application of Higher-Tier Neighbors QSC using snapshots of QSC using the AS graph snapshots G and G .The AS98 AS20 the AS relationship graph (G , G ) AS98 AS20 two thus induced FBASs contained, respectively, 2 and 6 nodes with one-node quorums sets which we ﬁlter our for top tiers (of 15 and 36 nodes, respectively), groups of only the subsequent analysis. We apply fbas_analyzer, our 2 nodes, and in the G case even one group of only one AS20 software-based analysis framework (cf. Sec. 5), to the result- node, exist that are sufﬁcient to completely block (or cen- ing FBASs. sor) the FBAS. For comparison, symmetric top tiers of the Figure 4 presents the analysis ﬁndings. It depicts his- same size would result in all minimal blocking sets having tograms of the relevant sets, i.e., how many minimal quo- sizes of, respectively, 5 and 12. This liveness-threatening dis- rums, minimal blocking sets or minimal splitting sets of a crepancy can be explained through cascading failures: If (for given size exist for the given FBAS. For the G case, we AS98 example) two nodes fail, this can result in a third node with restricted our minimal splitting sets analysis to the core of a “weak” quorum set becoming unsatisﬁable, so that three the FBAS, i.e., to its top tier and all nodes that are referenced 12 nodes have now de-facto failed, which can result in a fourth by top tier nodes either directly or transitively . We ﬁnd that node becoming unsatisﬁable, et cetera. It can be concluded doing so yields more informative results; the full FBAS con- that the composition and size of smallest blocking sets for tains a large number of splitting sets with cardinality 1 that an FBAS is heavily inﬂuenced by the “weakest” quorum sets only split off very small groups of nodes from the rest. Even in the FBAS’ top tier. An additional example for cascading when restricting the analysis to core nodes only, we were not failures is given Appendix B. able to fully enumerate the minimal splitting sets for G AS20 in reasonable time, due to the size and speciﬁc structure of 6.4 Symmetry enforcement the resulting FBAS. Strikingly, our analysis reveals that the liveness of both The graph-based QSC policies discussed so far easily result FBASs is easily compromised. Despite their relatively large in systems that are brittle (in the sense of small minimal blocking sets) and hard to analyze. Both of these character- This corresponds to the union of all strongly connected components that contain a quorum. istics are vastly improved, relative to top tier size, in FBASs 123 412 M. Florian et al. with symmetric top tiers. However, symmetric top tiers enjoys quorum intersection despite faulty nodes (a safe FBAS emerge organically from a preexisting relationship graph G as per the discussion in Sec. 3.4). only if the top tier nodes form a complete subgraph of G, which is not the case in the graphs investigated so far. As a 7.1 Top-down top tier change policy enhancement, nodes believing themselves to be top tier can mirror the quorum sets of other apparently top tier As a preliminary remark, recall that, as per Def. 4.4,we nodes, strategically including non-neighbors in their quorum deﬁne the top tier T of an FBAS (V, Q) as the union of sets for improving the global FBAS structure. A behavior all its minimal quorums. T is therefore also a quorum and along this lines can, in fact, be observed in the live Stellar intersects every quorum in (V, Q). network (s.a. Appendix C). Theorem 7.1 (top tier can safely change itself) Let T ⊂ V Yet, by making validator decisions independent of the be the top tier of an FBAS (V, Q) that enjoys quorum avail- local knowledge representation G, new assumptions become ability and quorum intersection. Then it is possible, without necessary to be able to rule out attacks. Mirroring makes it compromising neither quorum availability nor quorum inter- easier for malicious top tier nodes to introduce Sybil nodes section, to instantiate a new top tier T ⊆ V, T =∅ by into the top tier. The approach is therefore only secure (w.r.t. changing only the quorum sets of new and old top tier nodes both safety and liveness) if it can be assumed that nodes in v ∈ T ∪ T . T make plausibility checks before expanding their quorum sets, so that attempted (Sybil) attacks can be detected. Given Proof Let T ⊆ V, T =∅ be the target top tier. Let Q be the lack of explicit incentives for running validator nodes in a modiﬁcation of Q so that ∀v ∈ T ∪ T : Q (v) ={T } systems like Stellar, such a burden on the operators of top tier and ∀v/ ∈ T ∪ T : Q (v) = Q(v).As T is a quorum w.r.t. nodes might be viewed as problematic [11]. However, simi- Q , (T , Q ) enjoys quorum availability. Therefore, (V, Q ) lar critique can also be voiced against systems (like Bitcoin) enjoys quorum availability. (V \T , Q ) does not enjoy quo- that base their security arguments on notions of economic rum availability, because no node in T is satisﬁed without T rationality, as economic rationality can also be leveraged by and no node in V \T can form a quorum without a node from attackers [6]. T (otherwise T would not have been the top tier w.r.t. Q,cf. Def. 4.4). There are therefore no quorums w.r.t. Q that are disjoint of T . (V, Q ) therefore enjoys quorum intersection iff (T , Q ) enjoys quorum intersection, which it (trivially) 7 Limits on openness and top tier ﬂuidity does. The FBAS paradigm reportedly enables the instantiation The situation is less clear if some nodes T \ T do not of consensus systems with open membership [13,16]. And wish to leave T . Note, however, that single nodes can always clearly, arbitrary nodes can join an FBAS, causing new quo- endanger safety via trivial conﬁgurations such as Q(v) = rums to be formed that contain them. Based on the preceding {{v}}. If performed by one or more nodes in T , such an act of discussion, however, we recognize that without creating a sabotage can have an impact on the safety of large portions new, de-facto disjoint FBAS, or the active reconﬁguration of of the FBAS. existing nodes, new nodes cannot become part of minimal quorums and hence minimal blocking sets. Thereby, their 7.2 Bottom-up top tier change existence is irrelevant as far as the discussed liveness indica- tors are concerned, and their importance for safety is limited. In the following, we assume a “self-centered” top tier in the In Sec. 4 we deﬁned the notion of a top tier to reﬂect the set sense that all top tier nodes include only other top tier nodes of nodes in an FBAS that is central to liveness, i.e., the set in quorum sets. Symmetric top tiers (Def. 4.5) have this prop- of nodes from which all minimal quorums and blocking sets erty, as do top tiers observed in the wild in the Stellar network are formed. The top tier wields absolute power to censor and (cf. Appendix C). block the whole FBAS. In the following, we investigate the question to what extent Theorem 7.2 (no safe top tier change with uncooperative this top tier can be considered a group with open membership. top tier) Let (V, Q) be an FBAS that enjoys quorum intersec- How can its power be diluted by promoting additional nodes tion and has a “self-centered” top tier T ⊂ V such that all to top tier status? Can nodes be “ﬁred” from the top tier? We top tier quorum slices are comprised of only top tier nodes make the case that, in general, a top tier T can neither grow (∀v ∈ V : Q(v) ⊆ T ). Then it is not possible, without nor shrink without either the active involvement of existing top tier nodes or a loss of safety guarantees. We base all Without loss of generality. Clearly, more robust top tier constructions subsequent projections on the status quo of an FBAS that are possible. 123 The sum of its parts: Analysis of federated... 413 compromising quorum intersection, to instantiate a new top of-band) coordination between members of V \T ,a i −1 i −1 tier T ⊆ V, T = T by changing only the quorum sets of (V , Q ) might be instantiated in which at least (V \T , Q ) i i i i −1 i non-top tier nodes v ∈ V \T. enjoys quorum intersection. It is conceivable that novel pro- tocols can be developed, possibly also leveraging the FBAS Proof Let T ⊆ V, T = T be the top tier of a new FBAS structure, that reduce the notorious difﬁculty of coordinating ˆ ˆ (V, Q ) that enjoys quorum intersection. Let U and U be the such bottom-up actions. sets of all minimal quorums of (V, Q) and (V, Q ), respec- ˆ ˆ tively. As per Def. 4.4, T = T implies that U = U . ˆ ˆ ˆ ˆ Assume there exists a U ∈ U \ U . Then U is a quorum 8 Conclusion w.r.t. Q and either (a) not a quorum w.r.t. Q or (b) not minimal w.r.t. Q . We demonstrate in this paper that, despite the complexity of However, we require that the quorum sets of top tier nodes the FBAS model, the properties of concrete FBAS instances don’t change: ∀v ∈ T : Q (v) = Q(v). Therefore U is can be described in a way that is both precise and intuitive, a quorum also w.r.t. Q , contradicting (a). Hence, (b) must and allows comparisons with more classical Byzantine agree- ˆ ˆ ˆ ˆ hold and there must be a U ∈ U such that U ⊂ U (cf. ment systems. We propose the notions of minimal blocking ˆ ˆ ˆ Def. 4.1). As U ⊆ U ⊆ T , U being a quorum w.r.t. Q sets, minimal splitting sets and top tiers to describe which implies it also being a quorum w.r.t. Q. But then U is not groups of nodes can compromise liveness and safety. In ˆ ˆ minimal w.r.t. Q, implying U ∈ / U and thus again leading to essence, minimal blocking sets and minimal splitting sets ˆ ˆ a contradiction. This proves that U ⊆ U . describe minimal viable threat scenarios, thereby enabling a ˆ ˆ ˆ ˆ ˆ Assume now there exists a U ∈ U \ U and let U ∈ U. comprehensive risk assessment in FBAS-based systems like ˆ ˆ ˆ As (V, Q ) enjoys quorum intersection, U ∩ U =∅ and U the Stellar network. While some analyses imply computa- contains members of the “old” top tier T . U is a quorum w.r.t. tional problems of exponential complexity, we developed and ˆ ˆ Q ,but U ∩ T cannot be a quorum w.r.t. Q as otherwise U implemented algorithms that enable the exact analysis of a would not be a minimal quorum. There must therefore exist wide range of interesting FBASs. a node v ∈ U ∩ T with a quorum slice q ∈ Q (v) such that Our implemented analysis framework also enables us to ˆ ˆ (U ∩ T ) ⊂ q ⊆ U (cf. Def. 3.4), i.e., q \ T =∅.As v ∈ T , investigate how individual conﬁgurations result in global we require that Q (v) = Q(v) and Q(v) ⊆ T , which leads properties. We ﬁnd that overly strategic conﬁguration poli- to a contradiction since q ∈ Q(v) and q \ T =∅.Itmust cies result in FBASs that are indistinguishable from per- ˆ ˆ ˆ ˆ therefore hold that U \ U =∅, U = U and T = T . missioned systems. Individualistic approaches, on the other hand, cannot guarantee safe results while quickly resulting 7.3 Consequences in systems that are infeasible to analyze. Adding some strate- gic decision-making at organically emerging top tier nodes Who determines which FBAS nodes get to form the top tier? offers a potential middle way towards robust FBASs instan- Our results imply that, if maintaining safety is seen as an tiated from the sum of individual preferences. untouchable requirement, the top tier T of an FBAS (V , Q ) Independently of the way in which a given FBAS came to i i i at “iteration” i is legitimated by decisions of, exclusively, be, however, the composition of a once established top tier members of T ∪ T (if none of them cooperates, we lose cannot be inﬂuenced without the cooperation of existing top i −1 i safety, if all of them cooperate, we don’t). Because of the tier nodes, without at the same time threatening safety. This top tier’s importance to the liveness, safety and performance seems to place the FBAS paradigm closer to the “permis- achievable within a given FBAS, open membership in V is sioned consensus” camp than hoped. More investigation is of little beneﬁt without open membership in T . needed to determine the exact impact of bottom-up top tier How closed is the membership in T ? It might be sufﬁ- changes (as in number of nodes affected by a loss of safety or cient that only some nodes in T support a transition to T . i −1 i liveness, for example) and to formulate possible coordination If reactive QSC policies are used (e.g., for enforcing top tier strategies to keep such impacts low. symmetry as discussed in Sec. 6.4), one cooperative top tier Acknowledgements We thank Ben Schumacher, Jakob Hoffmann and node v ∈ T might already be enough for growing the top i −1 pieterjan84 for helpful discussions. We thank Ingolf Pernice, tier in a way that is robust and doesn’t only dilute the rela- Rainer Böhme and Patrik Keller for providing valuable feedback at tive inﬂuence of v. How partially supported top tier changes various stages of this work. We thank the anonymous reviewers of this would play out must be investigated based on more speciﬁc work for their insightful comments and suggestions. scenarios. We expect the safe “ﬁring” of top tier nodes to be Funding Open Access funding enabled and organized by Projekt especially challenging. DEAL. This work was funded by the German Federal Ministry of Edu- Which begs the question—can the safety requirement cation and Research (BMBF) through its funding for the Weizenbaum be weakened? For example, given sufﬁciently good (out- Institute for the Networked Society. 123 414 M. Florian et al. ˆ ˆ Open Access This article is licensed under a Creative Commons (V, Q), U ⊆ U be the set of all minimal quorums, and B ⊆ Attribution 4.0 International License, which permits use, sharing, adap- 2 be the set of all minimal blocking sets. Then each minimal tation, distribution and reproduction in any medium or format, as blocking set B ∈ B of the FBAS is minimally blocking w.r.t. long as you give appropriate credit to the original author(s) and the ˆ ˆ ˆ ˆ U, i.e., B intersects every minimal quorum U ∈ U and no source, provide a link to the Creative Commons licence, and indi- ˆ ˆ ˆ cate if changes were made. The images or other third party material B ⊂ B intersects every minimal quorum U ∈ U. in this article are included in the article’s Creative Commons licence, Proof Let B ⊆ 2 be the set of all blocking sets w.r.t. U. unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your Based on Cor. A.2 and Cor. A.3, B is exactly the set of all intended use is not permitted by statutory regulation or exceeds the blocking sets for U. Hence the set of all minimal sets w.r.t. permitted use, you will need to obtain permission directly from the copy- B is exactly the set of all minimal blocking sets w.r.t. U and right holder. To view a copy of this licence, visit http://creativecomm therefore the set of all minimal blocking sets for (V, Q),or ons.org/licenses/by/4.0/. ˆ ˆ B ⊆ B. Likewise, as B is the set of all blocking sets w.r.t. U, ˆ ˆ B is the set of all minimal blocking sets w.r.t. U. A Additional corollaries, theorems and proofs A.3 Splitting sets A.1 Minimal quorums Deﬁnition A.1 (quorum expanders) For an FBAS (V, Q),a quorum expander is any node v ∈ V that is part of a quorum Corollary A.1 (minimal quorum intersection ⇐⇒ quo- slice q ∈ Q(v ) of another node v ∈ V that is a not a quorum rum intersection) Let U ⊆ 2 be the set of all quorums of ˆ slice for v, i.e., any node v ∈ V for which ∃v ∈ V, q ∈ the FBAS (V, Q), U ⊆ U be the set of all minimal quorums. Q(v ) : v ∈ q ∧ (∀q ∈ Q(v) : q q ). ˆ ˆ ˆ All pairs of U , U ∈ U intersect iff all pairs of U , U ∈ U 1 2 1 2 intersect. Theorem A.1 (minimal splitting sets formed exclusively of quorum expanders and top tier nodes) Let S ⊆ 2 be the Proof Since U ⊆ U, ∀U , U ∈ U : U ∩ U =∅ trivially 1 2 1 2 set of all minimal splitting sets of the FBAS (V, Q),X ⊆ V ˆ ˆ ˆ ˆ ˆ implies that ∀U , U ∈ U : U ∩U =∅. The other direction 1 2 1 2 the set of all quorum expanders of the FBAS (Def. A.1) and ˆ ˆ ˆ ˆ follows because ∀U , U ∈ U ∃U , U ∈ U : U ⊆ U ∧ 1 2 1 2 1 1 T ⊆ V the top tier of the FBAS (the union of all minimal ˆ ˆ U ⊆ U (U being the set of all minimal sets w.r.t. U;s.a. 2 2 quorums, Def. 4.4). Then it holds that S ⊆ T ∪ X. Def. 4.1). If all pairs in U intersect, so must therefore all pairs ˆ ˆ ˆ Proof Let S ∈ S and s ∈ S be an arbitrary node in that in U. splitting set. We show that s ∈ T or s ∈ X must hold. ˆ ˆ This was previously also shown in [12]. S is a minimal splitting set, therefore S \{s} is not a S\{s} splitting set for any s. Consequently, (V, Q) enjoys A.2 Blocking sets S ˆ ˆ quorum intersection while (V, Q) doesn’t. Let U , U ⊂ 1 2 ˆ ˆ V, U ∩ U =∅ be two non-intersecting minimal quo- 1 2 Corollary A.2 (blocking for all ⇒ blocking for all min- ˆ rums in (V, Q) such that U does not contain a quorum imal) Let U ⊆ 2 be the set of all quorums of the FBAS S\{s} ˆ ˆ in (V, Q) . (If both U and U contained quorums in 1 2 (V, Q), and U ⊆ U be the set of all minimal quorums. If B S\{s} (V, Q) , the FBAS would lack quorum intersection.) is a blocking set for U, then it is also a blocking set for U. S\{s} ˆ ˆ If U ∪{s} contains a quorum in (V, Q) , then U ∪{s} 1 1 ˆ ˆ Proof B is a blocking set for U ⇐⇒ ∀U ∈ U : B ∩ U =∅ contains a minimal quorum U ⊆ U ∪{s} that contains s. ˆ ˆ ˆ ˆ (Def. 4.2). U ⊆ U ⇒ ∀U ∈ U : B ∩ U =∅, so that B is S\{s} Consequently, s is part of the top tier T of (V, Q) , i.e., also a blocking set for U. s ∈ T . As the only effect of the delete operation (Def. 3.7)on Q is to remove nodes from quorum slices and both (V, Q) and Corollary A.3 (blocking for all minimal ⇒ blocking for S\{s} V (V, Q) enjoy quorum intersection, it holds that T ⊆ T all) Let U ⊆ 2 be the set of all quorums of the FBAS (V, Q), (the proof is analogous to the proof of Thm. 7.2). Conse- and U ⊆ U be the set of all minimal quorums. If B is blocking quently, s ∈ T . set for U, then it is also a blocking set for U. S\{s} If U ∪{s} does not contain a quorum in (V, Q) , ˆ ˆ ˆ Proof B is a blocking set for U ⇒ ∀U ∈ U : B ∩ U =∅ then, because U is a quorum in (V, Q) , the forming of a ˆ ˆ (Def. 4.2). U ⊆ U and all U ∈ U are minimal w.r.t. U S\{s} quorum fails because of s.For (V , Q ) := (V, Q) ,it ˆ ˆ ˆ ⇒ ∀U ∈ U ∃U ∈ U : U ⊆ U (cf. Def. 4.1) ⇒ ˆ ˆ must hold that ∃v ∈ U , ∃q ∈ Q (v) : q ⊆ U ∪{s} while 1 1 U ∩ B =∅ ⇒ B is blocking for all U ∈ U. ∀q ∈ Q (s) : q U ∪{s}. The node s is therefore one S\{s} of the quorum expanders X of (V, Q) , i.e., s ∈ X .It Corollary A.4 (minimal blocking sets result from minimal trivially holds that X ⊆ X and, therefore, s ∈ X. quorums) Let U ⊆ 2 be the set of all quorums of the FBAS 123 The sum of its parts: Analysis of federated... 415 A.4 Top tier Let S ∈ S be an arbitrary minimal splitting set for (V, Q). ˆ ˆ ˆ If 2t − m ≤ 0, there exist two minimal quorums U , U ∈ U 1 2 Corollary A.5 (minimal blocking sets formed exclusively (with cardinality t) that do not intersect. There is then only of top tier nodes) Let T be the top tier of an FBAS (V, Q), one S =∅ and the cardinality of all minimal splitting sets and B ⊆ 2 be the set of all minimal blocking sets of (V, Q). is trivially 0. In the following, we assume that 2t − m > ˆ ˆ ˆ Then ∀B ∈ B : B ⊆ T. 0 and (V, Q) therefore enjoys quorum intersection. Since (V, Q) consists entirely of a symmetric top tier, no v ∈ V ˆ ˆ Proof From Cor. A.4 it follows that all B ∈ B are formed of is a quorum expander. Splitting sets must therefore contain ˆ ˆ nodes contained in at least one minimal quorum U ∈ U.As an intersection of at least one pair of minimal quorums (for ˆ ˆ ˆ ˆ T = U (Def. 4.4), ∀B ∈ B : B ⊆ T . illustration, cf. the proof of Thm. A.1). There are therefore ˆ ˆ ˆ ˆ at least two minimal quorums U , U ∈ U such that S = 1 2 Theorem A.2 (each top tier node in at least one minimal ˆ ˆ ˆ ˆ U ∩ U .Let U = U ∪ U . N = T \ U must be empty, 1 2 1 2 blocking set) Let T be the top tier of an FBAS (V, Q), and otherwise we could, with an arbitrary N ⊆ S, |N |=|N | B ⊆ 2 be the set of all minimal blocking sets of (V, Q). Then ˆ ˆ ﬁnd a minimal quorum U = (U \ N ) ∪ N such that 3 2 for each top tier node v ∈ T there is at least one minimal ˆ ˆ ˆ ˆ U ∩ U ⊂ S (i.e., S is not minimal). It therefore holds that 1 3 ˆ ˆ ˆ blocking set B ∈ B such that v ∈ B. ˆ ˆ ˆ U = T and, since, |U |=|U |= t, |S|= 2t − m. 1 2 ˆ ˆ Proof Let v ∈ T be an arbitrary top tier node and U ∈ U an arbitrary minimal quorum such that v ∈ U (recall that ˆ ˆ ˆ ˆ ˆ B Example analysis: Toy network with cascad- T = U;Def. 4.4). T \ U intersects every U ∈ U \{U }, ˆ ˆ ˆ ˆ ing failures as otherwise there would be a U ∈ U such that U ⊂ U ˆ ˆ (i.e., U would not be a minimal quorum). Therefore, T \ U ˆ ˆ ˆ Consider the FBAS (V, Q) with V ={0, 1, 2, 3, 4, 5, 6} and is a blocking set w.r.t. U \{U } and B ={v}∪ T \ U is a ˆ ˆ Q such that: blocking set w.r.t. U. B \{v} is not a blocking set w.r.t. U ˆ ˆ ˆ because it doesn’t intersect U. Hence, all B ∈ B such that ˆ Q(0) = qset(0,({0, 1, 2}, ∅, 3)) B ⊆ B (and there must be at least one—B —because B is a blocking set w.r.t. U) must contain v. Hence the FBAS has Q(1) = qset(1,({0, 1, 2, 3}, ∅, 3)) ˆ ˆ at least one B ∈ B that contains v. Q(2) = qset(2,({0, 1, 2, 3, 4, 5, 6}, ∅, 5)) Q(3) = qset(3,({0, 1, 2, 3, 4, 5, 6}, ∅, 5)) Theorem A.3 (Bocking sets in non-nested symmetric top tier) For an FBAS (V, Q) with a symmetric top tier T ⊆ V, Q(4) = qset(4,({0, 1, 2, 3, 4, 5, 6}, ∅, 5)) m := |T | such that ∀v ∈ T : Q(v) = qset(v, (T , ∅, t )) it Q(5) = qset(5,({0, 1, 2, 3, 4, 5, 6}, ∅, 5)) ˆ ˆ holds that: All minimal blocking sets B ∈ B have cardinality Q(6) = qset(6,({0, 1, 2, 3, 4, 5, 6}, ∅, 5)) max(m − t + 1, 0). Proof We observe that for any v ∈ T , Q(v) ={q ⊆ V : v ∈ q} ∧|q ∩ T|≥ t (Def. 3.2 and 3.3). A U ⊂ T is therefore a This Q can be the result of a scenario in which all v ∈ V quorum in (V, Q) iff |U|≥ t (Def. 3.4). As all U ⊂ T with apply the QSC policy All Neighbors QSC (Sec. 6.2) based |U|≥ t are quorums in (V, Q), the minimal quorums in on following graph G (unidirectional edges highlighted as ˆ ˆ ˆ (V, Q) are exactly U ={U ⊆ T , |U|= t }. Then: dashed lines): For all B ⊆ T with |B|= m − t + 1 it holds that ∀U ⊆ T \ B :|U |= t − 1 < t. Hence, no U ⊆ T \ B is a quorum, there are no quorums that are disjoint with B and B is a blocking set (Def. 4.2). B is furthermore a minimal blocking set, as for any B ⊂ B it holds that U = T \ B is a quorum (as |U|≥ t), and so B is not a blocking set. Theorem A.4 (Splitting sets in non-nested symmetric top tier) For an FBAS (V, Q) that consists entirely of a symmetric top tier T = V,m := |T | such that ∀v ∈ V : Q(v) = We ﬁnd the minimal blocking sets B ⊂ 2 of (V, Q) using ˆ ˆ qset(v, (V, ∅, t )) it holds that all minimal splitting sets S ∈ S our analysis tool (cf. Sec. 5): have cardinality max(2t − m, 0). B ={{2}, {1, 3}, {1, 4}, {1, 5}, {1, 6}, {0, 3}, {3, 4, 5}, Proof Like in Thm. A.3, we observe that the minimal quo- ˆ ˆ ˆ rums in (V, Q) are exactly U ={U ⊆ T , |U|= t }. Then: {3, 4, 6}, {3, 5, 6}, {0, 4, 5}, {0, 4, 6}, {0, 5, 6}, 123 416 M. Florian et al. {4, 5, 6}} Despite the fact that most nodes in V have very “robust” quorum sets— being able to tolerate up to f = 2 failures, which corresponds to a minimal blocking set of cardinality 3— the smallest blocking set of (V, Q), {2}, actually has cardinality 1. Consider a failure of node 2. Node 0’s quo- rum set (Q(0)) is not satisﬁable anymore, so that 0 de-facto fails as well. With both 0 and 2 failed, node 1, being able to tolerate only f = 1 failures, becomes unsatisﬁable as well. With three nodes having de-facto failed, none of the remaining nodes’ quorum sets can be satisﬁed anymore, so that (V, Q) loses quorum availability. Enabled through the “weak” quorum sets of nodes 0 and 1, the failure of 2 trig- gers what we would call a cascading failure. The liveness Fig. 5 Analysis results for daily snapshots of the Stellar network. For “buffer” of (V, Q), as represented by its smallest blocking each presented FBAS snapshot, the plot charts the size of its top tier as well as the mean cardinalities of minimal blocking and minimal splitting sets, is determined by the most easily dissatisﬁed nodes in its sets, with area boundaries marking the cardinalities of the smallest and top tier. largest respective set We see a similar, although weaker effect with regards to minimal splitting sets. In the present example, there are fewer minimal splitting sets S ⊂ 2 than in an “ideal” FBAS of Stellar software, to the organizations they belong to. We use thesamesize(cf. Ideal Open QSC in Sec. 6.1) but all but one this data to merge nodes belonging to the same organization, of them have the “ideal” cardinality 3 or a larger cardinality: so that nodes in the subsequent discussion represent distinct organizations as opposed to individual physical machines . S ={{1, 2}, {0, 1, 3}, {0, 1, 4}, {0, 2, 3}, {0, 2, 4}, {0, 3, 4}, For maintaining the correctness of our results, we merge nodes in this way after completing the analyses. Prior to {1, 3, 4, 5}, {2, 3, 4, 5}} analysis, we ﬁlter out all nodes that are marked as inactive or induce one-node quorums (i.e., nodes v with a conﬁgu- Note that unlike blocking sets that can compromise live- ration such as Q(v) ={v}; we assume that this represents ness for all nodes in an FBAS, splitting sets are usually more an accidental misconﬁguration). We furthermore restrict our relevant to some nodes than they are to others. For example, minimal splitting sets analyses to a core subset of nodes for the smallest splitting set of (V, Q), {1, 2}, can potentially each FBAS snapshot, namely to the top tier and all nodes cause node 0 to diverge from the remainder of the network— transitively referenced by top tier nodes’ quorum sets. Doing this is likely a bigger problem for node 0 than for nodes so gives us more informative aggregate results as forming a {3, 4, 5} which would remain “in sync”. splitting set that affects only a few edge nodes is both signif- icantly easier and less impactful than forming a splitting set that can cause top tier nodes to diverge. All analyses were C Example analysis: Stellar network performed using the algorithms and implementation intro- duced in Sec. 5. The results of our study are presented in As an example for the results obtainable using the proposed Fig. 5. methodology and tooling, we will now present a short study 14 The top tier of the Stellar network is growing mono- into the Stellar FBAS [13] . Our analysis methodology has tonically through time in the studied interval, reaching furthermore been integrated into Stellarbeat , a popular 7 organizations in February 2020. The top tiers of most monitoring website for the Stellar network. analyzed snapshots are symmetric and resemble (on the For the presented study, we obtain daily snapshots of the 16 organizations level) a classical (non-nested) threshold-based Stellar FBAS from Stellarbeat , for the interval July 2019 – quorum system. In Fig. 5, symmetric top tiers of such a type January 2022. From the same source, we also obtain data for manifest themselves as data points in which the cardinalities allocating nodes, here individual network hosts running the Nodes can also be merged based on other criteria, such as their We maintain an interactive version of this study at: https://trudi. country or ISP, revealing different threat scenarios. For example, for a weizenbaum-institut.de/stellar_analysis/ Footnote 17 continued https://stellarbeat.io/ snapshot of the Stellar FBAS from November 2020, we determine that Data from Stellarbeat was also used in previous academic studies a certain large cloud hosting provider forms a blocking set—i.e., has such as [11]. the power to unilaterally compromise liveness. 123 The sum of its parts: Analysis of federated... 417 of all minimal blocking sets are identical, as are the cardi- 11. Kim, M., Kwon, Y., Kim, Y.: Is Stellar as secure as you think? In: 2019 IEEE European Symposium on Security and Privacy nalities of all minimal splitting sets. During February 2020, Workshops (EuroS&PW), pp. 377–385. IEEE, Stockholm, Swe- the top tier grew by one organization, disturbing the sym- den (2019) metry for a few days. However, eventually all top tier nodes 12. Lachowski, Ł.: Complexity of the quorum intersection prop- included the new organization into their quorum sets. This erty of the federated Byzantine agreement system (2019). arxiv:1902.06493 adaptation suggests that top tier nodes might be reacting to 13. Lokhava, M., Losa, G., Mazières, D., Hoare, G., Barry, N., Gafni, each others’ decisions and actively strive towards a symmet- E., Jove, J., Malinowsky, R., McCaleb, J.: Fast and secure global ric conﬁguration, as proposed in Sec. 6.4. Furthermore, the payments with Stellar. In: Proceedings of the 27th ACM Sym- thresholds of top tier quorum sets appear to be chosen based posium on Operating Systems Principles (SOSP ’19), pp. 80–96. ACM, New York, NY, USA (2019) on a 67% logic (balancing liveness and safety risks), as do 14. Losa, G., Gafni, E., Mazières, D.: Stellar consensus by instantia- most example policies we discuss in Sec. 6. tion. In: 33rd International Symposium on Distributed Computing (DISC 2019), pp. 27:1–27:15. Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany (2019) 15. Malkhi, D., Reiter, M.: Byzantine quorum systems. Distributed References comput. 11(4), 203–213 (1998) 16. Mazières, D.: The Stellar consensus protocol: A federated model 1. Bracciali, A., Grossi, D., de Haan, R.: Decentralization in open for internet-level consensus (2015). https://stellar.org/papers/ quorum systems: Limitative results for Ripple and Stellar. In: 2nd stellar-consensus-protocol.pdf International Conference on Blockchain Economics, Security and 17. Nakamoto, S.: Bitcoin: A peer-to-peer electronic cash system Protocols (Tokenomics 2020), pp. 5:1–5:20. Schloss Dagstuhl– (2008). http://nakamotoinstitute.org/bitcoin/ Leibniz-Zentrum für Informatik, Dagstuhl, Germany (2021) 18. Ndolo, C., Henningsen, S., Florian, M.: Crawling the MobileCoin 2. Cachin, C., Tackmann, B.: Asymmetric distributed trust. In: 23rd quorum system (2021). arxiv:2111.12364 International Conference on Principles of Distributed Systems 19. Page, L., Brin, S., Motwani, R., Winograd, T.: The PageRank cita- (OPODIS 2019), pp. 7:1–7:16. Schloss Dagstuhl–Leibniz-Zentrum tion ranking: Bringing order to the web. Tech. rep, Stanford InfoLab fuer Informatik, Dagstuhl, Germany (2020) (1999) 3. Cachin, C., Zanolini, L.: Asymmetric asynchronous byzantine 20. Stathakopoulou, C., David, T., Vukolic, ´ M.: Mir-BFT: High- consensus. In: Data Privacy Management, Cryptocurrencies and throughput BFT for blockchains (2019). arxiv:1906.05552 Blockchain Technology, pp. 192–207. Springer (2021) 21. Tarjan, R.: Depth-ﬁrst search and linear graph algorithms. SIAM 4. Castro, M., Liskov, B., et al.: Practical Byzantine fault tolerance. j. on comput. 1(2), 146–160 (1972) In: Proceedings of the Third Symposium on Operating Systems 22. Yin, M., Malkhi, D., Reiter, M.K., Gueta, G.G., Abraham, I.: Design and Implementation (OSDI), pp. 173–186. USENIX, New HotStuff: BFT consensus with linearity and responsiveness. In: Orleans, Louisiana, USA (1999) Proceedings of the 2019 ACM Symposium on Principles of Dis- 5. Douceur, J.R.: The Sybil attack. In: Peer-to-peer Systems, pp. 251– tributed Computing (PODC ’19), pp. 347–356. ACM, New York, 260. Springer, Berlin, Heidelberg (2002) NY, USA (2019) 6. Ford, B., Böhme, R.: Rationality is self-defeating in permissionless systems (2019). arxiv:1910.08820 7. Gallo, G., Longo, G., Pallottino, S., Nguyen, S.: Directed hyper- graphs and applications. Discrete appl. math. 42(2–3), 177–201 Publisher’s Note Springer Nature remains neutral with regard to juris- (1993) dictional claims in published maps and institutional afﬁliations. 8. García-Pérez, Á., Gotsman, A.: Federated Byzantine quorum systems. In: 22nd International Conference on Principles of Distributed Systems (OPODIS 2018), pp. 17:1–17:16. Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany (2018) 9. García-Pérez, Á., Schett, M.A.: Deconstructing Stellar consensus. In: 23rd International Conference on Principles of Distributed Sys- tems (OPODIS 2019), pp. 5:1–5:16. Schloss Dagstuhl–Leibniz- Zentrum fuer Informatik, Dagstuhl, Germany (2020) 10. Gaul, A., Khofﬁ, I., Liesen, J., Stüber, T.: Mathematical analysis and algorithms for federated Byzantine agreement systems (2019). arxiv:1912.01365
Distributed Computing – Springer Journals
Published: Oct 1, 2022
Keywords: Byzantine quorum systems; Asymmetric trust; Byzantine faults; Consensus; Stellar; Blockchain
You can share this free article with as many people as you like with the url below! We hope you enjoy this feature!
Read and print from thousands of top scholarly journals.
Already have an account? Log in
Bookmark this article. You can see your Bookmarks on your DeepDyve Library.
To save an article, log in first, or sign up for a DeepDyve account if you don’t already have one.
Copy and paste the desired citation format or use the link below to download a file formatted for EndNote
Access the full text.
Sign up today, get DeepDyve free for 14 days.
All DeepDyve websites use cookies to improve your online experience. They were placed on your computer when you launched this website. You can change your cookie settings through your browser.