RIFT Applicability

5.1. South Reflection

South reflection is a mechanism that South Node TIEs are "reflected" back up north to allow nodes in same level without east-west links to "see" each other.¶

For example, in Figure 4, Spine111\Spine112\Spine121\Spine122 reflects Node S-TIEs from ToF21 to ToF22 separately. Respectively, Spine111\Spine112\Spine121\Spine122 reflects Node S-TIEs from ToF22 to ToF21 separately. So ToF22 and ToF21 see each other's node information as level 2 nodes.¶

In an equivalent fashion, as the result of the south reflection between Spine121-Leaf121-Spine122 and Spine121-Leaf122-Spine122, Spine121 and Spine 122 knows each other at level 1.¶

5.2. Suboptimal Routing on Link Failures

              +--------+          +--------+
              | ToF21  |          |  ToF22 |                LEVEL 2
              ++--+-+-++          ++-+--+-++
               |  | | |            | |  | +
               |  | | |            | |  | linkTS8
  +------------+  | +-+linkTS3+-+  | |  | +-------------+
  |               |   |         |  | |  +               |
  |    +---------------------------+ |  linkTS7         |
  |    |          |   |         +    +  +               |
  |    |          |   +-------+linkTS4+------------+    |
  |    |          |             +    +  |          |    |
  |    |          |    +-------------+--+          |    |
  |    |          |    |        |  linkTS6         |    |
+-+----+-+      +-+----+-+     ++--------+       +-+----+-+
|Spine111|      |Spine112|     |Spine121 |       |Spine122| LEVEL 1
+-+---+--+      +-+----+-+     +-+---+---+       +-+----+-+
  |   |           |    |         |   |             |    |
  |   +-------------+  |         +   ++XX+linkSL6+---+  +
  |               | |  |      linkSL5              | |  linkSL8
  |   +-----------+ |  |         +   +---+linkSL7+-+ |  +
  |   |             |  |         |   |               |  |
+-+---+-+        +--+--+-+     +-+---+-+          +--+--+-+
|Leaf111|        |Leaf112|     |Leaf121|          |Leaf122| LEVEL 0
+-+-----+        +-+-----+     +-----+-+          +-+-----+
  +                +                 +              +
Prefix111        Prefix112     Prefix121          Prefix122

As shown in Figure 4, as the result of the south reflection between Spine121-Leaf121-Spine122 and Spine121-Leaf122-Spine122, Spine121 and Spine 122 knows each other at level 1.¶

Without disaggregation mechanism, when linkSL6 fails, the packet from leaf121 to prefix122 will probably go up through linkSL5 to linkTS3 then go down through linkTS4 to linkSL8 to Leaf122 or go up through linkSL5 to linkTS6 then go down through linkTS8 and linkSL8 to Leaf122 based on pure default route. It's the case of suboptimal routing or bow-tieing.¶

With disaggregation mechanism, when linkSL6 fails, Spine122 will detect the failure according to the reflected node S-TIE from Spine121. Based on the disaggregation algorithm provided by RIFT, Spine122 will explicitly advertise prefix122 in Disaggregated Prefix S-TIE PrefixesElement(prefix122, cost 1). The packet from leaf121 to prefix122 will only be sent to linkSL7 following a longest-prefix match to prefix 122 directly then go down through linkSL8 to Leaf122 .¶

5.3. Black-Holing on Link Failures

                +--------+          +--------+
                | ToF 21 |          | ToF 22 |                LEVEL 2
                ++-+--+-++          ++-+--+-++
                 | |  | |            | |  | +
                 | |  | |            | |  | linkTS8
  +--------------+ |  +-+linkTS3+X+  | |  | +--------------+
  linkTS1          |    |         |  | |  +                |
  +    +-----------------------------+ |  linkTS7          |
  |    |           +    |         +    +  +                |
  |    |      linkTS2   +-------+linkTS4+X+----------+     |
  |    +           +              +    +  |          |     |
  |   linkTS5      +-+    +------------+--+          |     |
  |    +             |    |       |  linkTS6         |     |
+-+----+-+         +-+----+-+    ++-------+        +-+-----++
|Spine111|         |Spine112|    |Spine121|        |Spine122| LEVEL 1
+-+---+--+         ++----+--+    +-+---+--+        +-+----+-+
  |   |             |    |         |   |             |    |
  +   +---------------+  |         +   +---+linkSL6+---+  +
  linkSL1           | |  |      linkSL5              | |  linkSL8
  +   +--+linkSL3+--+ |  |         +   +---+linkSL7+-+ |  +
  |   |               |  |         |   |               |  |
+-+---+-+          +--+--+-+     +-+---+-+          +--+--+-+
|Leaf111|          |Leaf112|     |Leaf121|          |Leaf122| LEVEL 0
+-+-----+          +-+-----+     +-----+-+          +-----+-+
  +                  +                 +                  +
Prefix111          Prefix112     Prefix121          Prefix122

This scenario illustrates a case when double link failure occurs and with that black-holing can happen.¶

Without disaggregation mechanism, when linkTS3 and linkTS4 both fail, the packet from leaf111 to prefix122 would suffer 50% black-holing based on pure default route. The packet supposed to go up through linkSL1 to linkTS1 then go down through linkTS3 or linkTS4 will be dropped. The packet supposed to go up through linkSL3 to linkTS2 then go down through linkTS3 or linkTS4 will be dropped as well. It's the case of black-holing.¶

With disaggregation mechanism, when linkTS3 and linkTS4 both fail, ToF22 will detect the failure according to the reflected node S-TIE of ToF21 from Spine111\Spine112. Based on the disaggregation algorithm provided by RIFT, ToF22 will explicitly originate an S-TIE with prefix 121 and prefix 122, that is flooded to spines 111, 112, 121 and 122.¶

The packet from leaf111 to prefix122 will not be routed to linkTS1 or linkTS2. The packet from leaf111 to prefix122 will only be routed to linkTS5 or linkTS7 following a longest-prefix match to prefix122.¶

5.4. Zero Touch Provisioning (ZTP)

RIFT is designed to require a very minimal configuration to simplify its operation and avoid human errors; based on that minimal information, Zero Touch Provisioning (ZTP) autoconfigures the key operational parameters of all the RIFT nodes, including the SystemID of the node that must be unique in the RIFT network and the level of the node in the Fat Tree, which determines which peers are northwards "parents" and which are southwards "children".¶

ZTP is always on, but its decisions can be overridden when a network administrator prefers to impose its own configuration. In that case, it is the responsibility of the administrator to ensure that the configured parameters are correct, in other words that the SystemID of each node is unique, and that the administratively set levels truly reflect the relative position of the nodes in the fabric. It is recommended to let ZTP configure the network, and when not, it is recommended to configure the level of all the nodes to avoid an undesirable interaction between ZTP and the manual configuration.¶

ZTP requires that the administrator points out the Top-of-Fabric (ToF) nodes to set the baseline from which the fabric topology is derived. The Top-of-Fabric nodes are configured with TOP_OF_FABRIC flag which are initial 'seeds' needed for other ZTP nodes to derive their level in the topology. ZTP computes the level of each node based on the Highest Available Level (HAL) of the potential parent(s) nearest that baseline, which represents the superspine. In a fashion, RIFT can be seen as a distance-vector protocol that computes a set of feasible successors towards the superspine and auto-configures the rest of the topology.¶

The autoconfiguration mechanism computes a global maximum of levels by diffusion. The derivation of the level of each node happens then based on Link Information Elements (LIEs) received from its neighbors whereas each node (with possibly exceptions of configured leaves) tries to attach at the highest possible point in the fabric. This guarantees that even if the diffusion front reaches a node from "below" faster than from "above", it will greedily abandon already negotiated level derived from nodes topologically below it and properly peer with nodes above.¶

The achieved equilibrium can be disturbed massively by all nodes with highest level either leaving or entering the domain (with some finer distinctions not explained further). It is therefore recommended that each node is multi-homed towards nodes with respective HAL offerings. Fortunately, this is the natural state of things for the topology variants considered in RIFT.¶

A RIFT node may also be configured to confine it to the leaf role with the LEAF_ONLY flag. A leaf node can also be configured to support leaf-2-leaf procedures with the LEAF_2_LEAF flag. In either case the node cannot be TOP_OF_FABRIC and its level cannot be configured. RIFT will fully determine the node's level after it is attached to the topology and ensure that the node is at the "bottom of the hierarchy" (southernmost).¶

5.5. Mis-cabling Examples

  +----------------+              +-----------------+
  |     ToF21      |       +------+      ToF22      |   LEVEL 2
  +-------+----+---+       |      +----+---+--------+
  |       |    |   |       |      |    |   |        |
  |       |    |   +----------------------------+   |
  |   +---------------------------+    |   |    |   |
  |   |   |    |           |           |   |    |   |
  |   |   |    |   +-----------------------+    |   |
  |   |   +------------------------+   |        |   |
  |   |        |   |       |       |   |        |   |
+-+---+--+   +-+---+--+    |    +--+---+-+  +--+---+-+
|Spine111|   |Spine112|    |    |Spine121|  |Spine122| LEVEL 1
+-+---+--+   ++----+--+    |    +--+---+-+  +-+----+-+
  |   |       |    |       |       |   |       |    |
  |   +---------+  |     link-M    |   +---------+  |
  |           | |  |       |       |           | |  |
  |   +-------+ |  |       |       |   +-------+ |  |
  |   |         |  |       |       |   |         |  |
+-+---+-+    +--+--+-+     |     +-+---+-+    +--+--+-+
|Leaf111|    |Leaf112+-----+     |Leaf121|    |Leaf122| LEVEL 0
+-------+    +-------+           +-------+    +-------+

Figure 6 shows a single plane mis-cabling example. It's a perfect Fat Tree fabric except link-M connecting Leaf112 to ToF22.¶

The RIFT control protocol can discover the physical links automatically and be able to detect cabling that violates Fat Tree topology constraints. It reacts accordingly to such mis-cabling attempts, at a minimum preventing adjacencies between nodes from being formed and traffic from being forwarded on those mis-cabled links. Leaf112 will in such scenario use link-M to derive its level (unless it is leaf) and can report links to Spine111 and Spine112 as mis-cabled unless the implementations allows horizontal links.¶

Figure 7 shows a multiple plane mis-cabling example. Since Leaf112 and Spine121 belong to two different PoDs, the adjacency between Leaf112 and Spine121 can not be formed. link-W would be detected and prevented.¶

 +-------+    +-------+           +-------+    +-------+
 |ToF  A1|    |ToF  A2|           |ToF  B1|    |ToF  B2| LEVEL 2
 +-------+    +-------+           +-------+    +-------+
 |       |    |       |           |       |    |       |
 |       |    |       +-----------------+ |    |       |
 |       +--------------------------+   | |    |       |
 |     +------+                   | |   | +------+     |
 |     |        +-----------------+ |   |      | |     |
 |     |        |   +--------------------------+ |     |
 |  A  |        | B |               | A |        |  B  |
 +-----+--+   +-+---+--+         +--+---+-+   +--+-----+
 |Spine111|   |Spine112|     +---+Spine121|   |Spine122| LEVEL 1
 +-+---+--+   ++----+--+     |   +--+---+-+   +-+----+-+
   |   |       |    |        |      |   |       |    |
   |   +---------+  |        |      |   +---------+  |
   |           | |  |      link-W   |           | |  |
   |   +-------+ |  |        |      |   +-------+ |  |
   |   |         |  |        |      |   |         |  |
 +-+---+-+    +--+--+-+      |    +-+---+-+    +--+--+-+
 |Leaf111|    |Leaf112+------+    |Leaf121|    |Leaf122| LEVEL 0
 +-------+    +-------+           +-------+    +-------+
+--------PoD#1----------+       +---------PoD#2---------+

RIFT provides an optional level determination procedure in its Zero Touch Provisioning mode. Nodes in the fabric without their level configured determine it automatically. This can have possibly counter-intuitive consequences however. One extreme failure scenario is depicted in Figure 8 and it shows that if all northbound links of spine11 fail at the same time, spine11 negotiates a lower level than Leaf11 and Leaf12.¶

To prevent such scenario where leafs are expected to act as switches, LEAF_ONLY flag can be set for Leaf111 and Leaf112. Since level -1 is invalid, Spine11 would not derive a valid level from the topology in Figure 8. It will be isolated from the whole fabric and it would be up to the leafs to declare the links towards such spine as mis-cabled.¶

+-------+    +-------+        +-------+    +-------+
|ToF  A1|    |ToF  A2|        |ToF  A1|    |ToF  A2|
+-------+    +-------+        +-------+    +-------+
|       |    |       |                |            |
|    +-------+       |                |            |
+    +  |            |  ====>         |            |
X    X  +------+     |                +------+     |
+    +         |     |                       |     |
+----+--+    +-+-----+                     +-+-----+
|Spine11|    |Spine12|                     |Spine12|
+-+---+-+    ++----+-+                     ++----+-+
  |   |       |    |                        |    |
  |   +---------+  |                        |    |
  |   +-------+ |  |                +-------+    |
  |   |         |  |                |            |
+-+---+-+    +--+--+-+        +-----+-+    +-----+-+
|Leaf111|    |Leaf112|        |Leaf111|    |Leaf112|
+-------+    +-------+        +-+-----+    +-+-----+
                                |            |
                                |   +--------+
                                |   |
                              +-+---+-+
                              |Spine11|
                              +-------+

5.6. Positive vs. Negative Disaggregation

Disaggregation is the procedure whereby [RIFT] advertises a more specific route southwards as an exception to the aggregated fabric-default north. Disaggregation is useful when a prefix within the aggregation is reachable via some of the parents but not the others at the same level of the fabric. It is mandatory when the level is the ToF since a ToF node that cannot reach a prefix becomes a black hole for that prefix. The hard problem is to know which prefixes are reachable by whom.¶

In the general case, [RIFT] solves that problem by interconnecting the ToF nodes. So the ToF nodes can exchange the full list of prefixes that exist in the fabric and figure out when a ToF node lacks reachability to some prefixes. This requires additional ports at the ToF, typically 2 ports per ToF node to form a ToF-spanning ring. [RIFT] also defines the southbound reflection procedure that enables a parent to explore the direct connectivity of its peers, meaning their own parents and children; based on the advertisements received from the shared parents and children, it may enable the parent to infer the prefixes its peers can reach.¶

When a parent lacks reachability to a prefix, it may disaggregate the prefix negatively, i.e., advertise that this parent can be used to reach any prefix in the aggregation except that one. The Negative Disaggregation signaling is simple and functions transitively from ToF to top-of-pod (ToP) and then from ToP to Leaf. But it is hard for a parent to figure which prefix it needs to disaggregate, because it does not know what it does not know; it results that the use of a spanning ring at the ToF is required to operate the Negative Disaggregation. Also, though it is only an implementation problem, the programmation of the FIB is complex compared to normal routes, and may incur recursions.¶

The more classical alternative is, for the parents that can reach a prefix that peers at the same level cannot, to advertise a more specific route to that prefix. This leverages the normal longest prefix match in the FIB, and does not require a special implementation. But as opposed to the Negative Disaggregation, the Positive Disaggregation is difficult and inefficient to operate transitively.¶

Transitivity is not needed to a grandchild if all its parents received the Positive Disaggregation, meaning that they shall all avoid the black hole; when that is the case, they collectively build a ceiling that protects the grandchild. But until then, a parent that received a Positive Disaggregation may believe that some peers are lacking the reachability and readvertise too early, or defer and maintain a black hole situation longer than necessary.¶

In a non-partitioned fabric, all the ToF nodes see one another through the reflection and can figure if one is missing a child. In that case it is possible to compute the prefixes that the peer cannot reach and disaggregate positively without a ToF-spanning ring. The ToF nodes can also ascertain that the ToP nodes are connected each to at least a ToF node that can still reach the prefix, meaning that the transitive operation is not required.¶

The bottom line is that in a fabric that is partitioned (e.g., using multiple planes) and/or where the ToP nodes are not guaranteed to always form a ceiling for their children, it is mandatory to use the Negative Disaggregation. On the other hand, in a highly symmetrical and fully connected fabric, (e.g., a canonical Clos Network), the Positive Disaggregation methods allows to save the complexity and cost associated to the ToF-spanning ring.¶

Note that in the case of Positive Disaggregation, the first ToF node(s) that announces a more-specific route attracts all the traffic for that route and may suffer from a transient incast. A ToP node that defers injecting the longer prefix in the FIB, in order to receive more advertisements and spread the packets better, also keeps on sending a portion of the traffic to the black hole in the meantime. In the case of Negative Disaggregation, the last ToF node(s) that injects the route may also incur an incast issue; this problem would occur if a prefix that becomes totally unreachable is disaggregated.¶

5.7. Mobile Edge and Anycast

When a physical or a virtual node changes its point of attachement in the fabric from a previous-leaf to a next-leaf, new routes must be installed that supersede the old ones. Since the flooding flows northwards, the nodes (if any) between the previous-leaf and the common parent are not immediately aware that the path via previous-leaf is obsolete, and a stale route may exist for a while. The common parent needs to select the freshest route advertisement in order to install the correct route via the next-leaf. This requires that the fabric determines the sequence of the movements of the mobile node.¶

On the one hand, a classical sequence counter provides a total order for a while but it will eventually wrap. On the other hand, a timestamp provides a permanent order but it may miss a movement that happens too quickly vs. the granularity of the timing information. It is not envisioned that an average fabric supports Precision Time Protocol [IEEEstd1588] in the short term, nor that the precision available with the Network Time Protocol [RFC5905] (in the order of 100 to 200ms) may not be necessarily enough to cover, e.g., the fast mobility of a Virtual Machine.¶

Section 4.3.3. "Mobility" of [RIFT] specifies an hybrid method that combines a sequence counter from the mobile node and a timestamp from the network taken at the leaf when the route is injected. If the timestamps of the concurrent advertisements are comparable (i.e., more distant than the precision of the timing protocol), then the timestamp alone is used to determine the relative freshness of the routes. Otherwise, the sequence counter from the mobile node, if available, is used. One caveat is that the sequence counter must not wrap within the precision of the timing protocol. Another is that the mobile node may not even provide a sequence counter, in which case the mobility itself must be slower than the precision of the timing.¶

Mobility must not be confused with anycast. In both cases, a same address is injected in RIFT at different leaves. In the case of mobility, only the freshest route must be conserved, since mobile node changed its point of attachment for a leaf to the next. In the case of anycast, the node may be either multihomed (attached to multiple leaves in parallel) or reachable beyond the fabric via multiple routes that are redistributed to different leaves; either way, in the case of anycast, the multiple routes are equally valid and should be conserved. Without further information from the redistributed routing protocol, it is impossible to sort out a movement from a redistribution that happens asynchronously on different leaves. [RIFT] expects that anycast addresses are advertised within the timing precision, which is typically the case with a low-precision timing and a multihomed node. Beyond that time interval, RIFT interprets the lag as a mobility and only the freshest route is retained.¶

When using IPv6 [RFC8200], RIFT suggests to leverage "Registration Extensions for IPv6 over Low-Power Wireless Personal Area Network (6LoWPAN) Neighbor Discovery (ND)" [RFC8505] as the IPv6 ND interaction between the mobile node and the leaf. This provides not only a sequence counter but also a lifetime and a security token that may be used to protect the ownership of an address [RFC8928]. When using [RFC8505], the parallel registration of an anycast address to multiple leaves is done with the same sequence counter, whereas the sequence counter is incremented when the point of attachement changes. This way, it is possible to differentiate a mobile node from a multihomed node, even when the mobility happens within the timing precision. It is also possible for a mobile node to be multihomed as well, e.g., to change only one of its points of attachement.¶

5.9. In-Band Reachability of Nodes

RIFT doesn't precondition that nodes of the fabric have reachable addresses. But the operational reasons to reach the internal nodes may exist. Figure 10 shows an example that the network management station (NMS) attaches to leaf1.¶

+-------+      +-------+
| ToF1  |      | ToF2  |
++---- ++      ++-----++
 |     |        |     |
 |     +----------+   |
 |     +--------+ |   |
 |     |          |   |
++-----++      +--+---++
|Spine1 |      |Spine2 |
++-----++      ++-----++
 |     |        |     |
 |     +----------+   |
 |     +--------+ |   |
 |     |          |   |
++-----++      +--+---++
| Leaf1 |      | Leaf2 |
+---+---+      +-------+
    |
    |NMS

If NMS wants to access Leaf2, it simply works. Because loopback address of Leaf2 is flooded in its Prefix North TIE.¶

If NMS wants to access Spine2, it simply works too. Because spine node always advertises its loopback address in the Prefix North TIE. NMS may reach Spine2 from Leaf1-Spine2 or Leaf1-Spine1-ToF1/ToF2-Spine2.¶

If NMS wants to access ToF2, ToF2's loopback address needs to be injected into its Prefix South TIE. This TIE must be seen by all nodes at the level below - the spine nodes in Figure 10 – that must form a ceiling for all the traffic coming from below (south). Otherwise, the traffic from NMS may follow the default route to the wrong ToF Node, e.g., ToF1.¶

In case of failure between ToF2 and spine nodes, ToF2's loopback address must be disaggregated recursively all the way to the leaves. In a partitioned ToF, even with recursive disaggregation a ToF node is only reachable within its plane.¶

A possible alternative to recursive disaggregation is to use a ring that interconnects the ToF nodes to transmit packets between them for their loopback addresses only. The idea is that this is mostly control traffic and should not alter the load balancing properties of the fabric.¶

5.10. Dual Homing Servers

Each RIFT node may operate in Zero Touch Provisioning (ZTP) mode. It has no configuration (unless it is a Top-of-Fabric at the top of the topology or the must operate in the topology as leaf and/or support leaf-2-leaf procedures) and it will fully configure itself after being attached to the topology.¶

    +---+         +---+         +---+
    |ToF|         |ToF|         |ToF|      ToF
    +---+         +---+         +---+
    |   |         |   |         |   |
    |   +----------------+      |   |
    |          +----------------+   |
    |          |  |   |  |          |
    +----------+--+   +--+----------+
    |     ToR1    |   |     ToR2    |      Spine
    +--+------+---+   +--+-------+--+
+---+  |      |   |   |  |       |  +---+
|   +-----------------+  |       |      |
|   |  |   +-------------+       |      |
|   |  |   |  |   +-----------------+   |
|   |  |   |  +--------------+   |  |   |
|   |  |   |                 |   |  |   |
+---+  +---+                 +---+  +---+
|   |  |   |                 |   |  |   |
+---+  +---+  .............  +---+  +---+
SV(1) SV(2)                 SV(n-1) SV(n)  Leaf

Sometimes, people may prefer to disaggregate from ToR to servers from start on, i.e. the servers have couple tens of routes in FIB from start on beside default routes to avoid breakages at rack level. Full disaggregation of the fabric could be achieved by configuration supported by RIFT.¶

5.11. Fabric With A Controller

There are many different ways to deploy the controller. One possibility is attaching a controller to the RIFT domain from ToF and another possibility is attaching a controller from the leaf.¶

                 +------------+
                 | Controller |
                 ++----------++
                  |          |
                  |          |
             +----++        ++----+
 -------     | ToF |        | ToF |
    |        +--+--+        +-----+
    |        |  |           |     |
    |        |  +-------------+   |
    |        |     +--------+ |   |
    |        |     |          |   |
             +-----+        +-+---+
RIFT domain  |Spine|        |Spine|
             +--+--+        +-----+
    |        |  |           |     |
    |        |  +-------------+   |
    |        |     +--------+ |   |
    |        |     |          |   |
    |        +-----+        +-+---+
 -------     |Leaf |        | Leaf|
             +-----+        +-----+

5.12. Internet Connectivity Within Underlay

If global addressing is running without overlay, an external default route needs to be advertised through RIFT fabric to achieve internet connectivity. For the purpose of forwarding of the entire RIFT fabric, an internal fabric prefix needs to be advertised in the South Prefix TIE by ToF and spine nodes.¶

5.13. Subnet Mismatch and Address Families

+--------+                     +--------+
|        |  LIE          LIE   |        |
|   A    | +---->       <----+ |   B    |
|        +---------------------+        |
+--------+                     +--------+
   X/24                           Y/24

LIEs are exchanged over all links running RIFT to perform Link (Neighbor) Discovery. A node must NOT originate LIEs on an address family if it does not process received LIEs on that family. LIEs on same link are considered part of the same negotiation independent on the address family they arrive on. An implementation must be ready to accept TIEs on all addresses it used as source of LIE frames.¶

As shown in the above figure, without further checks adjacency of node A and B may form, but the forwarding between node A and node B may fail because subnet X mismatches with subnet Y.¶

To prevent this a RIFT implementation should check for subnet mismatch just like e.g. ISIS does. This can lead to scenarios where an adjacency, despite exchange of LIEs in both address families may end up having an adjacency in a single AF only. This is a consideration especially in Section 5.8 scenarios.¶

5.14. Anycast Considerations

                        + traffic
                        |
                        v
                 +------+------+
                 |     ToF     |
                 +---+-----+---+
                 |   |     |   |
    +------------+   |     |   +------------+
    |                |     |                |
+---+---+    +-------+     +-------+    +---+---+
|       |    |       |     |       |    |       |
|Spine11|    |Spine12|     |Spine21|    |Spine22| LEVEL 1
+-+---+-+    ++----+-+     +-+---+-+    ++----+-+
  |   |       |    |         |   |       |    |
  |   +---------+  |         |   +---------+  |
  |   +-------+ |  |         |   +-------+ |  |
  |   |         |  |         |   |         |  |
+-+---+-+    +--+--+-+     +-+---+-+    +--+--+-+
|       |    |       |     |       |    |       |
|Leaf111|    |Leaf112|     |Leaf121|    |Leaf122| LEVEL 0
+-+-----+    ++------+     +-----+-+    +-----+-+
  +           +                  +      ^     +
PrefixA      PrefixB         PrefixA    | PrefixC
                                        |
                                        + traffic

If the traffic comes from ToF to Leaf111 or Leaf121 which has anycast prefix PrefixA, RIFT can deal with this case well. But if the traffic comes from Leaf122, it arrives Spine21 or Spine22 at level 1. But Spine21 or Spine22 doesn't know another PrefixA attaching Leaf111. So it will always get to Leaf121 and never get to Leaf111. If the intension is that the traffic should been offloaded to Leaf111, then use policy guided prefixes defined in RIFT [RIFT].¶

5.15. IoT Applicability

The design of RIFT inherits from RPL [RFC6550] the anisotropic design of a default route upwards (northwards); it also inherits the capability to inject external host routes at the Leaf level using Wireless ND (WiND) [RFC8505][RFC8928] between a RIFT-agnostic host and a RIFT router. Both the RPL and the RIFT protocols are meant for large scale, and WiND enables device mobility at the edge the same way in both cases.¶

The main difference between RIFT and RPL is that with RPL, there’s a single Root, whereas RIFT has many ToF nodes. This adds huge capabilities for leaf-2-leaf ECMP paths, but additional complexity with the need to disaggregate. Also RIFT uses Link State flooding northwards, and is not designed for low-power operation.¶

Still nothing prevents that the IP devices connected at the Leaf are IoT (Internet of Things) devices, which typically expose their address using WiND – which is an upgrade from 6LoWPAN ND [RFC6775].¶

A network that serves high speed/ high power IoT devices should typically provide deterministic capabilities for applications such as high speed control loops or movement detection. The Fat Tree is highly reliable, and in normal condition provides an equivalent multipath operation; but the ECMP doesn’t provide hard guarantees for either delivery or latency. As long as the fabric is non-blocking the result is the same; but there can be load unbalances resulting in incast and possibly congestion loss that will prevent the delivery within bounded latency.¶

This could be alleviated with Packet Replication, Elimination and Reordering (PREOF) [RFC8655] leaf-2-leaf but PREOF is hard to provide at the scale of all flows, and the replication may increase the probability of the overload that it attempts to solve.¶

Note that the load balancing is not RIFT’s problem, but it is key to serve IoT adequately.¶