| Internet-Draft | EVPN VPWS Gateways | July 2023 |
| Rabadan, et al. | Expires 11 January 2024 | [Page] |
Ethernet Virtual Private Network Virtual Private Wire Services (EVPN VPWS) need to be deployed in high scale multi-domain networks, where each domain can use a different transport technology, such as MPLS, VXLAN or Segment Routing with MPLS or IPv6 Segment Identifiers (SIDs). While transport interworking solutions on border routers spare the border routers from having to process service routes, they do not always meet the multi-homing, redundancy, and operational requirements, or provide the isolation that each domain requires. This document analyzes the scenarios in which an interconnect solution for EVPN VPWS using EVPN Domain Gateways is needed, and adds the required extensions to support it.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 11 January 2024.¶
Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
Ethernet VPN Virtual Private Wire Services (EVPN VPWS) [RFC8214] need to be deployed in high scale multi-domain networks, where each domain can use a different transport technology, such as MPLS, VXLAN or Segment Routing with MPLS or IPv6 Segment Identifiers (SIDs). While the so-call transport interworking solutions on border routers spare the border routers from having to process service routes, they do not always meet the multi-homing, redundancy, and operational requirements, or provide the isolation that each domain requires. This document analyzes the scenarios in which an interconnect solution for EVPN VPWS using EVPN Domain Gateways is needed, and adds the required extensions to support it.¶
This section summarizes the terminology that is used throughout the rest of the document.¶
This section describes the EVPN [I-D.ietf-bess-rfc7432bis] high level interconnect options and discusses their applicability to EVPN VPWS.¶
Service Interworking solution:¶
A-D per EVI A-D per EVI A-D per EVI
RD2 tag2 L22 RD3 tag3 SID33 RD4 tag4 vni44
<------------+ <--------------+ <-------------+
+------------+ +--------------+ +-------------+
| | | | | |
I-PE BR-1 BR-2 E-PE
+-------+ +-------+ +-------+ +-------+
|+-----+| |+-----+| |+-----+| |+-----+|
CE1--||VPWS1|| ||VPWS1|| ||VPWS1|| ||VPWS1||-->CE2
|+-----+| |+-----+| |+-----+| |+-----+|
+-------+ +-------+ +-------+ +-------+
| SR-MPLS | | SRv6 | | VXLAN |
+------------+ +--------------+ +--------------+
<------------> <--------------> <-------------->
Domain-1 Domain-2 Domain-3
[RFC9014] section 4 describes an end-to-end EVPN interconnect solution using EVPN Domain Gateways, or simply Gateways. The Gateways provide connectivity across EVPN Domains, where those Domains can use MPLS tunnels, NVO3 tunnels (e.g., VXLAN) or Segment Routing tunnels. Procedures are extrapolated to SRv6 domains too. The Gateways provide independence in terms of the Route Targets and Route Distinguishers used in each Domain, or the type of multicast tree used for BUM traffic in each domain, while keeping the key EVPN properties end-to-end, such as MAC mobility, MAC protection or ARP suppression. The Gateways also provide all-active and single-active multi-homing redundancy by extending the concept of the multi-homing Ethernet Segment for interconnect domains (I-ES). In this document, we refer to this solution as the Service Interworking option, and the Border Routers play the role of EVPN Domain Gateways. Since [RFC9014] section 4 only describes the solution for EVPN multi-point services, this document extends the procedures to support EVPN VPWS services with the required extensions. Figure 1 illustrates the Service Interworking solution across domains of different transport encapsulations when applied to EVPN VPWS services.¶
Inter-domain Option-B solution:¶
NHSelf NHSelf A-D per EVI
L22<-L33 L33<-L44 RD4 tag4 L44
<------------+ <--------------+ <-------------+
+------------+ +--------------+ +-------------+
| | | | | |
I-PE BR-1 BR-2 E-PE
+-------+ +-------+ +-------+ +-------+
|+-----+| | | | | |+-----+|
CE1--||VPWS1|| | | | | ||VPWS1||-->CE2
|+-----+| | | | | |+-----+|
+-------+ +-------+ +-------+ +-------+
| SR-MPLS | | SR-MPLS | | SR-MPLS |
+------------+ +--------------+ +--------------+
<------------> <--------------> <-------------->
Domain-1 Domain-2 Domain-3
[RFC8365] section 10 provides an alternative interconnect solution for EVPN services by using Border Routers that re-write the EVPN BGP next hops and program a swap operation of the VNIs or MPLS labels (depending on whether the encapsulation is NVO3-based or MPLS-based). This solution does not require the instantiation of Services on the Border Routers that perform a lookup on the inner destination MAC (as in the case of [RFC9014]), however the solution is limited to the interconnect of domains of the same encapsulation. In addition, the solution does not support per-ES mass withdraw of the EVPN MAC/IP Advertisement routes, as described in [RFC8365]. In this document we refer to this solution as Inter-domain Option-B. Figure 2 illustrates this model when applied to EVPN VPWS, where the three domains are all now of the same encapsulation, and there is no service instantiation on the Border Routers.¶
Transport Interworking solution:¶
A-D per EVI
RD4 tag4 L44
<---------------------------------------------+
+------------+ +--------------+ +-------------+
| | | | | |
I-PE BR-1 BR-2 E-PE
+-------+ +-------+ +-------+ +-------+
|+-----+| |Transp | |Transp | |+-----+|
CE1--||VPWS1|| |IW | |IW | ||VPWS1||-->CE2
|+-----+| | | | | |+-----+|
+-------+ +-------+ +-------+ +-------+
| SR-MPLS | | SRv6 | | SR-MPLS |
+------------+ +--------------+ +--------------+
<------------> <--------------> <-------------->
Domain-1 Domain-2 Domain-3
Other proposals are currently being investigated, in the context of SRv6 to MPLS interworking, e.g., [I-D.agrawal-spring-srv6-mpls-interworking]. In these solutions, the Border Routers do not change the EVPN BGP next hops, or process EVPN routes for that matter. The Border Routers provide stitching between MPLS and SRv6 tunnels. In this case, the solution allows the interconnect of domains of different encapsulation, as long as the ingress and egress PEs support the same encapsulation. A variation of this solution is the Inter-domain Option-C solution, where a BGP LU (Label Unicast) tunnel provides the stitching across the domains, as long as all the domains use the same encapsulation. In this document, we refer to this solution as Transport Interworking option. Figure 3 illustrates this model when applied to EVPN VPWS, where I-PE and E-PE are attached to domains of the same encapsulation. Intermediate domains, e.g., Domain-2, can be of encapsulations different from the ones used at the ingress and egress Domains. The EVPN route is not processed or changed by the Border Routers.¶
The three interconnect solutions described in Section 1.2 are valid, however, this section describes the requirements that make the Service Interworking solution needed. Those requirements are:¶
Per-domain EVPN Multi-Homing¶
The Service Interworking solution allows the use of different Ethernet Segment Identifiers (ESI) per domain, as well as the implementation of the aliasing and backup procedures on a per-domain basis. The use of different ESIs per domain may help guarantee the uniqueness of the ESI when different domains independently managed and operated are interconnected. The implementation of independent aliasing and backup procedures per domain, spares the need for propagation of the EVPN A-D per ES routes by the Border Routers (which are EVPN Domain Gateways in the Service Interworking solution). These A-D per ES routes are consumed within the domain, which results in a significant reduction of the number of routes that the ingress PEs need to process. Another consequence of the processing of A-D per ES routes per domain, is a faster convergence in case of ES PE or link failure, since A-D per ES routes are no longer propagated by all the Border Routers along the domains, but processed by the Border Routers of the originating domain. Per-domain EVPN Multi-Homing procedures are not possible in the Inter-domain Option-B or Transport Interworking solutions.¶
Per-ES Mass Withdrawal¶
In order to benefit from the per-ES mass withdrawal property of EVPN Multi-Homing, the received BGP next hops of the selected EVPN A-D per EVI and A-D per ES routes need to match on a PE. This cannot be guaranteed in an Inter-domain Option-B solution, as described in [RFC8365] section 10.2.2., however it is always the case in the Service Interworking or Transport Interworking solution.¶
Per-domain Route Distinguishers (RDs) and Route Targets (RTs)¶
In case of merge of domains coming from different administrative entities, the uniqueness of RDs and RTs across domains for the same service is not guaranteed. Hence the re-write of RD/RTs at the Border Routers may be required. If that is the case, the Service Interworking solution provides the support for re-writing RD/RTs. The Inter-domain Option-B may allow re-writing RD/RTs, however, it is not considered a common practice. The Transport Interworking solution does not support the translation of RD/RTs.¶
Ethernet Tag IDs per domain¶
Similar to per-domain RDs and RTs, re-writing of Ethernet Tag IDs used in the A-D per EVI routes may be needed in case of interconnect of domains that belong to different administrative entities. This can be only supported by a Service Interworking solution.¶
Control Word, Flow Label and MTU (Maximum Transfer Unit) signaling per domain¶
As described in [I-D.ietf-bess-rfc7432bis], the use of Control Word and Flow Label, as well as the MTU are signaled in the EVPN Layer 2 Attributes extended community along with the A-D per EVI routes. The signaling and use of Control Word is recommended in those domains where P routers can get confused when hashing based on the tunneled EVPN packet payload, but the Control Word may not be needed in some domains. Similarly, the Flow Label introduces an additional level of entropy in EVPN encapsulated packets, that may be needed in some domains but adding unnecessary extra overhead in other domains. Different MTUs may be supported in different domains, due to the domains running on different physical media. A Service Interworking model allows the signaling and use of Control Word, Flow Label, and Layer-2 MTU on a per domain basis. This is not the case in the other two models analyzed in this document.¶
Heterogeneous Encapsulations¶
Interconnecting domains that use different encapsulations (e.g., VXLAN, SRv6, MPLS, SR-MPLS, etc.) is a common requirement. This becomes important in case the domains have different platform features, or migrations to new encapsulations or transport types are needed. In the Service Interworking model the EVPN routes are generated and consumed at every Border Router (which is an EVPN Domain Gateway), hence the encapsulation indicated along with the route can be advertised independently at each Border Router. That is not the case in the models 2 and 3 in Section 1.2. The Inter-domain Option-B model requires the same encapsulation in each of the domains the Border Router connects, whereas the Transport Interworking model requires that at least the ingress and egress domains have the same encapsulation.¶
Per-domain EVPN Service OAM¶
[RFC9062] defines the Service OAM requirements for EVPN services. When applied to the Interconnect solutions, the three solutions in Section 1.2 allow for the use of MEPs and MIPs on the ingress and egress PEs, but only the Service Interworking solution supports MEPs and MIPs on the Border Routers. In other words, per-domain EVPN Service OAM is only supported in the Service Interworking option.¶
The above requirements and their support across the Interconnect solutions are summarized in Table 1.¶
| Requirement | Service Interworking | Inter-domain Option-B | Transport Interworking |
|---|---|---|---|
| Per-domain EVPN Multi-Homing | Yes | No | No |
| Per-ES Mass Withdrawal | Yes | No | Yes |
| Per-domain RD/RTs | Yes | Yes* | No |
| Ethernet Tag IDs per domain | Yes | No | No |
| Control Word, Flow Label and MTU signaling per domain | Yes | No | No |
| Heterogeneous encapsulations | Yes | No | Yes** |
| Per-domain EVPN Service OAM | Yes | No | No |
* Although possible, it is unusual to re-write RD/RTs in the Inter-domain Option-B solution¶
** Supported only when the ingress and egress domains are of the same encapsulation¶
The rest of the document specifies the extensions required for the EVPN Domain Gateways to implement the Service Interworking solution to deploy end-to-end EVPN VPWS services. In a nutshell, the AD per EVI routes advertised by the E-PE are redistributed across domains and delivered to the I-PE, while ES and A-D per ES routes advertised by E-PEs are not redistributed by the EVPN Domain Gateways. In addition, this document defines how Gateway redundancy works using either an Anycast Gateway solution, or by extending the I-ES concept already defined for multi-point EVPN services in [RFC9014].¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶
This section describes the EVPN VPWS extensions on the EVPN Domain Gateways (or simply Gateways) to support the Service Interworking model. An EVPN Domain Gateway in this context is a Border Router that connects EVPN Domains and implements the Service Interworking model of Section 1.2. Section 3.1 specifies the Gateway rules to redistribute EVPN routes. When redundant Gateways attached to two or more EVPN Domains are deployed, there are two redundancy mechanisms that can be used. Section 3.2 describes a redundancy method that we refer to as "Anycast" and is based on the redundant Gateways behaving as a single system for the remote PEs. Section 3.3 describes the redundancy based on I-ES, as an extension of the I-ES procedures specified in [RFC9014], only for EVPN VPWS services. The Anycast redundancy does not require the use of I-ES and supports single-active multi-homing connectivity, but it will not support all-active, aliasing, backup, or mass withdraw features that are supported along with the use of I-ES and EVPN Multi-Homing.¶
The EVPN Domain Gateways MUST establish separate BGP sessions for sending/receiving EVPN routes to/from each different Domain to which they are attached. We refer to redistribution of an EVPN route to all the procedures in the Gateway that involve the reception and process of the source domain EVPN route, the programming of the forwarding path for the route, and the readvertisement of the route to a different domain (the next destination domain).¶
The reception and processing of EVPN routes for an EVPN VPWS service follows [RFC8214]. If the D-PATH attribute is contained in the EVPN A-D per EVI route, loop detection and best path selection follows [I-D.sr-bess-evpn-dpath]. The Gateway imports the valid best EVPN A-D per EVI route required for an Ethernet Tag ID based on the matching import Route Target and the best path selection described in [I-D.ietf-bess-rfc7432bis], section 7.13.2. If a non-zero ESI is included in the route, the [RFC8214] procedures for aliasing, backup, and mass withdraw are followed on the Gateway. Note that the best path selection for A-D per EVI routes (with non-zero ESI) in [I-D.ietf-bess-rfc7432bis] section 7.13.2 also influences how the Gateway adds primary or back-up next hops to the created Ethernet Segment destinations. As an example, suppose a Gateway receives "m" A-D per EVI routes for ESI "x" and Ethernet Tag ID "y" (all of them with different Route Distinguishers and the flag P set) but supports only "n" paths in the Aliasing list for ESI "x" (with m>n). In this case, the Gateway orders the "m" routes following the best path selection in [I-D.ietf-bess-rfc7432bis] section 7.13.2, and selects the "n" top routes of the ordered list.¶
If an A-D per EVI route for a service is successfully imported and processed, forwarding state is programmed in the data path using the MPLS label, VNI or SRv6 SID that was received in the EVPN A-D per EVI route. In addition, depending on the encapsulation of the route's next destination domain, the router allocates a new MPLS label, VNI or SRv6 SID and programs a data path switching operation between the identifiers of the source and next destination domains. Immediately after, the Gateway re-advertises the route to the BGP speaker in the next domain. We refer to the source domain as the domain from which the Gateway receives the route, and the next domain as the EVPN Domain in which the Gateway redistributes the route. The following considerations apply to the redistributed EVPN A-D per EVI routes:¶
The redistributed route SHOULD carry the Communities, Extended Communities, Large Communities and Wide Communities of the source route.¶
EVPN VPWS services also make use of multi-homing routes, that is, EVPN A-D per ES routes and Ethernet Segment routes. These multi-homing routes are processed in the Gateway as in [RFC8214]. The A-D per ES and Ethernet Segment routes are only processed in the context of the domain they are received, and they MUST NOT be redistributed to any other domain. A-D per ES and Ethernet Segment routes may be originated at the Gateway though, if the Gateway is attached to an I-ES, as described in Section 3.3.¶
The procedures on the EVPN Domain Gateways described in this document are compatible with PEs that implement either the default Flexible Crossconnect (FXC) mode or the VLAN-Signaled Flexible Crossconnect mode described in [I-D.ietf-bess-evpn-vpws-fxc].¶
The Anycast Service Gateway redundancy is specified as follows:¶
All the Anycast Gateways attached to the same two domains MUST redistribute the EVPN A-D per EVI routes between domains as per Section 3.1 with the following considerations:¶
As an illustration of this redundancy method, suppose all four Service Gateways in Figure 4 are configured as Anycast Service Gateways, and local and remote Ethernet Tag IDs are configured as 1, 2 and 3 on all routers in the domains 1, 2 and 3 respectively.¶
A-D per EVI A-D per EVI
RD11 tag1 L111 RD21 tag2 SID21
<------------+ <--------------+
+------------+ +--------------+ +-------------+
| Domain-1 | | Domain-2 | | Domain-3 |
| BR-11 BR-21 A-D per EVI
| +-------+ +-------+ RD4 tag3 vni33
| |+-----+| |+-----+| <---------+
I-PE +--> ||VPWS1||-----> ||VPWS1||--+ E-PE
+-------+ | |+-----+| |+-----+| | +-------+
|+-----+|--+ +-------+ +-------+ | |+-----+|
CE1--||VPWS1|| | | | | +--> ||VPWS1||-->CE2
|+-----+| BR-12 BR-22 |+-----+|
+-------+ +-------+ +-------+ +-------+
| |+-----+| |+-----+| |
| ||VPWS1|| ||VPWS1|| |
| |+-----+| |+-----+| |
| +-------+ +-------+ |
| SR-MPLS | | SRv6 | | VXLAN |
+------------+ +--------------+ +-------------+
A-D per EVI A-D per EVI
RD12 tag1 L121 RD22 tag2 SID22
<------------+ <--------------+
In the example in Figure 4 E-PE advertises an EVPN A-D per EVI route for Ethernet Tag ID 3. Both BR-21 and BR-22 import the route and redistribute it with Ethernet Tag ID 2 and new RD and encapsulation into domain-2. When redistributing, both BR-21 and BR-22 update (if it existed before) or insert a D-PATH attribute with the domain-id of domain-3. That prevents BR-21 and BR-22 from redistributing back into domain-3 each other's route [I-D.sr-bess-evpn-dpath]. BR-11 and BR-12 import the routes after best path selection and perform the same process and redistribution into domain-1. I-PE will receive two routes for Ethernet Tag ID 1, from BR-11 and BR-12, and will perform best path selection for Ethernet Tag ID 1. Based on the best path selection carried out by I-PE and the BRs along the way, all flows from CE1 to CE2 will follow, e.g., I-PE, BR-11, BR-21 and E-PE. In case of failure on any of the BRs in the data path, the routers will select the alternate route for the Ethernet Tag ID. The same control plane exchange and traffic flow happen in the reverse direction, where I-PE becomes the egress PE and E-PE the ingress PE.¶
As illustrated in Figure 4, this model does not support per-flow load balancing (all-active multi-homing) to all the BR nodes along the way from CE to CE.¶
EVPN Multi-Homing procedures can be used on the EVPN Domain Gateways. For that, an I-ES and its assigned I-ESI will be configured on the Gateways for multihoming. The I-ES concept is introduced in [RFC9014], and it is used in this document for EVPN VPWS services. This I-ES represents a domain to the next domain, in both directions. Therefore two or more Gateways attached to the same two domains will use the same I-ESI when advertising routes to the two domains.¶
The Gateways attached to the same I-ES:¶
Figure 5 illustrates the use of I-ES or EVPN Multi-Homing procedures in EVPN Domain Gateways. In the example, BR-11 and BR-12 are attached to I-ES-1 (with ESI-1 as identifier), whereas BR-21 and BR-22 are attached to I-ES-2 (using ESI-2).¶
A-D per EVI A-D per EVI
RD11 tag1 ESI-1 L111 RD21 tag2 ESI-2 SID21
<------------+ <--------------+
+------------+ +--------------+ +-------------+
| Domain-1 | | Domain-2 | | Domain-3 |
| BR-11 BR-21 A-D per EVI
| +-------+ +-------+ RD4 tag3 vni33
| |+-----+| |+-----+| <---------+
I-PE +--> ||VPWS1||-+---> ||VPWS1||--+ E-PE
+-------+ | |+-----+| | +-> |+-----+| | +-------+
|+-----+|--+ +-------+ | | +-------+ +--> |+-----+|
CE1--||VPWS1|| I-ES1 | | | | | | I-ES2 ||VPWS1||-->CE2
|+-----+|--+ BR-12 | | BR-22 +--> |+-----+|
+-------+ | +-------+ +-|-> +-------+ | +-------+
| | |+-----+| | |+-----+| | |
| +--> ||VPWS1||---+-> ||VPWS1||--+ |
| |+-----+| |+-----+| |
| +-------+ +-------+ |
| SR-MPLS | | SRv6 | | VXLAN |
+------------+ +--------------+ +-------------+
A-D per EVI A-D per EVI
RD12 tag1 ESI-1 L121 RD22 tag2 ESI-2 SID22
<------------+ <--------------+
E-PE advertises an A-D per EVI route for tag3, that gets redistributed by BR-21/BR-22 first, and BR-11/BR-12 later, translating the Ethernet Tag ID and encapsulation in each redistribution. The BR nodes implement the EVPN Multi-Homing procedures for their own Ethernet Segment as in [RFC8214], and set the P and B flags accordingly when redistributing the A-D per EVI routes, to indicate the forwarding mode to the receiving nodes. If I-ES-1 and I-ES-2 are defined as all-active multi-homing Ethernet Segments, per-flow load balancing will be performed not only by the I-PE to the Gateways in domain-1, but also by the Gateways at each domain of the EVPN VPWS service, as depicted in Figure 5. The same control plane exchange and traffic flow happen in the reverse direction, where I-PE becomes the egress PE and E-PE the ingress PE.¶
I-ES-1 and I-ES-2 are independent of each other, e.g., I-ES-1 can work in single-active mode, whereas I-ES-2 uses all-active mode. If that is the case, BR-11 and BR-12 run Designated Forwarded (DF) Election and BR-11 signals P=1 and B=0 (in the EVPN Layer 2 Attributes extended community) if it is elected as DF, whereas BR-12 signals P=0 and B=1 if elected as Backup DF router. I-PE then sends all traffic to BR-11, and BR-21/BR-22 send all traffic to BR-11 in the reverse direction. Since BR-21/BR-22 work in all-active mode, they both signal P=1/B=0 to both, E-PE and BR-11/BR-12. Therefore traffic from BR-11/BR-12 is sprayed to both BR-21/BR-22, and so is traffic from E-PE.¶
If EVPN Multi-Homing is used in the redundant Gateways, Fast Reroute procedures as in [I-D.burdet-bess-evpn-fast-reroute] MAY be applied to speed up convergence in case one of the Gateways loses its connectivity to the adjacent domain.¶
The Anycast Gateway and the EVPN Multi-Homing redundancy solutions can coexist. The Gateways of the same redundancy group MUST implement the same redundancy method, but different redundancy Gateway groups MAY implement different methods. In the example, BR-11/BR-12 constitutes a redundancy group and BR-21/BR-22 constitutes a different redundancy group.¶
To be added in a future version.¶
None.¶