| Internet-Draft | Abbreviated Title | May 2023 |
| Geib | Expires 9 November 2023 | [Page] |
Within a Segment Routing domain, segment routed measurement packets can be sent along pre-determined paths. This enables new kinds of measurements. Connectivity monitoring allows to supervise the state and performance of a connection or a (sub)path from one or a few central monitoring systems. This document specifies a suitable type-P connectivity monitoring metric.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 9 November 2023.¶
Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
Within a Segment Routing domain, measurement packets can be sent along pre-determined segment routed paths [RFC8402]. A segment routed path may consist of pre-determined sub paths, specific router-interfaces or a combination of both. A measurement path may also consist of sub paths spanning multiple routers, given that all segments to address a desired path are available and known at the SR domain edge interface.¶
A Path Monitoring System (PMS, see [RFC8403]) is a dedicated central Segment Routing (SR) domain monitoring device (as compared to a distributed monitoring approach based on router-data and -functions only). Monitoring individual sub-paths or point-to-point connections is executed for different purposes. IGP exchanges hello messages between neighbors to keep alive routing and swiftly adapt routing to topology changes. Network Operators may in addition to that be interested in monitoring connectivity and congestion of interfaces or sub-paths at a timescale of seconds, minutes or hours. In all cases, the periodicity of probing samples and statistics based on these samples is often significantly smaller than commodity interface monitoring based on router counters, which may be collected on a minute timescale to keep the processing- or monitoring data-load low.¶
The IPPM architecture was a first step to that direction [RFC2330]. Commodity IPPM solutions require dedicated measurement systems, a large number of measurement agents and synchronised clocks. Monitoring a domain from edge to edge by commodity IPPM solutions increases scalability of the monitoring system. But localising the site of a detected network behaviour change may then require suitable network tomography methods.¶
The IPPM Metrics for measuring connectivity offer generic connectivity metrics [RFC2678]. These metrics allow to measure connectivity between end nodes without making any assumption on the paths between them. The metric and the type-p packet specified by this document follow a different approach: they are designed to monitor connectivity and performance of a specific single link or a path segment. The underlying definition of connectivity is partially the same: a packet not reaching a destination indicates a loss of connectivity. An IGP re-route may indicate a loss of a link between neighbors, while it doesn't necessarily cause a loss of connectivity between end systems. The metric specified here detects a loss of connectivity between neighbors, defined by a complete absence of a path between two nodes in both directions of communication (whereas a re-routing will briefly disturb a path, but connectivity is restored by the network after a short disturbance).¶
A Segment Routing PMS is part of an SR domain. The PMS is IGP topology aware, covering the IP and (if present) the MPLS layer topology [RFC8402]. This allows to steer PMS measurement packets along arbitrary pre-determined concatenated sub-paths, identified by suitable Segment IDs. Basically, the SR connectivity metric as specified by this document requires set up of a number of constrained, overlaid measurement loops (or measurement paths). The delay of the packets sent along each of these measurement loops is measured. A single congested interface along a monitored sub-path adds latency along a unique subset of several measurement loops. If a monitored sub-path no longer provides IP/MPLS connectivity between two nodes, another unique subset of measurement loops will drop all traffic while connectivity is lost. The number of measurement loops required in total may be limited to one per sub-path (or connection) to be monitored, if a hub-and-spoke like sub-path topology as described below is monitored. In addition to information revealed by a commodity ICMP ping measurement, the metrics and methods specified here identify the location of a congested interface (or ingress of a congested sub-path, respectively). To do so, tomography assumptions and methods are combined to first plan the overlaid SR measurement loop set up and later on to evaluate the captured performance metrics.¶
There's another difference as compared with commodity ping: the measurement loop packets remain in the data plane of passed routers. These need to forward the measurement packets without any additional processing apart from that.¶
It is recommended to consider automated measurement loop set-up. The methods proposed here are error-prone, if the topology and measurement loop design isn't followed properly. While details of an automated set-up are not within scope of this document, some formal defintions of constraints to be respected are given.¶
This document specifies type-p metrics determining properties of an SR path which allows to monitor connectivity and congestion of interfaces. The specified methods further allow to locate the path or interface which caused a change in the reported type-p metrics. This document is limited to the Segment Routing MPLS layer, but the methodology may be applied within SR domains or MPLS domains in general.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 [RFC2119].¶
The Segment Routing IGP topology information consists of the IP and (if present) the MPLS layer topology. The minimum SR topology information consists of Node-Segment-Identifiers (Node-SID), identifying an SR router. The IGP exchange of Adjacency-SIDs (Adj-SID) [RFC8667], which identify local interfaces to adjacent nodes, is optional. It is RECOMMENDED to distribute Adj-SIDs in a domain operating a PMS to monitor connectivity as specified below. If Adj-SIDs aren't availbale, [RFC8287] provides methods how to steer packets along desired paths by the proper choice of an MPLS Echo-request IP-destination address. A detailed description of [RFC8287] methods as a replacement of Adj-SIDs is out of scope of this document. Monitoring interfaces connecting nodes requires Adj-SIDs, if re-converged IP/MPLS layer connectivity would result in re-routing packets (and re-establishment of IP/MPLS layer connectivity) by using Node-SIDs.¶
An active round trip measurement between two adjacent nodes is a simple method to monitor connectivity of a connecting link. If multiple links are operational between two adjacent nodes and only a single one looses connectivity, a single plain round trip measurement may fail to notice that or fail to identify which link has lost connectivity. A round trip measurement further fails to identify which particular interface is congested, even if only a single link connects two adjacent nodes.¶
Segment Routing enables the set-up of extended measurement loops. Several different measurement loops can be set up to form a partial overlay. If done properly, any network change impacts more than a single measurement loop's round trip delay or causes drops of packets of more than one loop. Randomly chosen measurement loop paths including the interfaces or paths to be monitored may fail to produce the desired unique result patterns, hence commodity network tomography methods aren't applicable [CommodityTomography]. The approach pursued here uses a pre-specified measurement loop overlay design to produce the desired results with a minimum effort.¶
A centralised monitoring approach doesn't require report collection and result correlation from two (or more) receivers. The metrics captured along different measurement loops however still need to be correlated.¶
An additional property of the measurement loop set-up specified below is that it allows to estimate the packet round trip delay of a monitored link or sub-path.¶
An example hub and spoke network, operated as SR domain, is shown below. The included PMS shown is supposed to monitor the connectivity of all the 6 links (a link is a simple and generic kind of sub-path) attaching the spoke-nodes L050, L060 and L070 to the hub-nodes L100 and L200. L300 only serves to connect the PMS to nodes L100 and L200.¶
+---+ +----+ +----+
|PMS| |L100|-----|L050|
+---+ +----+\ /+----+
| / \ \_/_____
| / \ / \+----+
+----+/ \/_ +----|L060|
|L300| / |/ +----+
+----+\ / /\_
\ / / \
\+----+ / +----+
|L200|-----|L070|
+----+ +----+
Example hub and spoke network allowing link connectivity verification with a PMS¶
The SID values are picked for convenient reading only. Node-SID: 100 identifies L100, Node-SID: 300 identifies L300 and so on. Adj-SID 10050: Adjacency L100 to L050, Adj-SID 10060: Adjacency L100 to L060, Adj-SID 60200: Adjacency L60 to L200 and so on (note that the Adj-SID are locally assigned per node interface, meaning two per link).¶
Monitoring the 6 links between hub nodes Ln00 (where n=1,2) and spoke nodes L0m0 (where m=5,6,7) requires 6 measurement loops, which have the following properties:¶
Note that any 6 links connecting two to five nodes can be monitored that way too. Further note that the measurement loop overlay chosen is optimised for 6 links and a hub and spoke topology of two to five nodes. The 'one measurement loop per measured sub-path' paradigm only works under these conditions.¶
The above overlay scheme results in 6 measurement loops for the given example. The start and end of each measurement loop is PMS to L300 to L100 or L200 and a similar sub-path on the return leg. These parts of the measurement loops are omitted here for brevity (some discussion may befound below). The following delays are measured along the SR paths of each measurement loop:¶
For brevity, in the following delay M1 also identifies the corresponding measurement loop number 1 and so on.¶
An example for a stack of Adj-SID segments the loop resulting in M1 is (top to bottom): 100 | 10050 | 50100 | 10060 | 60200 | PMS. As can be seen, the Node-SIDs 100 and PMS are present at top and bottom of the segment stack. Their purpose is to transport the packet from the PMS to the start of the measurement loop at L100 and return it to the PMS from its end. When connectivity is lost, a path determined by Adj-SIDs behaves deterministic: packets forwarded to an Adj-SID without connectivity to the neighboring node are dropped.¶
An example for a stack of a loop consisting of Node-SID segments allowing to capture M1 is (top to bottom): 100 | 050 | 100 | 060 | 200 | PMS.¶
The evaluation of the measurement loop round trip delays M1 - M6 allows to detect the follwing state-changes of the monitored sub-paths:¶
As an example, the formula to calculate the (sub-path) Round Trip Delay (RTD) for link L100-L050 is given here¶
4 * RTD_L100-L050-L100 = 3 * M1 + M3 + M6 - M2 - M4 - M5.¶
This formula is reproducible for all other links: sum up 3*RTD measured along the loop passing the monitored link of interest in round trip fashion, and add the RTDs of the two measurement loops passing the evaluated monitored link only in a single direction. From this sum subtract the RTD captured for the measurement loops not passing the monitored link evaluated to get four times the RTD of the monitored link evaluated.¶
A closer look reveals that any single event of interest for the proposed metric, which are a single loss of connectivity or a single case of congestion, only impacts a unique set of measurement loops which can be determined a-priori. If, e.g., connectivity is lost between L200 and L050, measurement loops M3, M4 and M6 indicate packet loss (or a change of the measured delay, if a Node-SID based approach is preferred).¶
As a second example: if the interface L070 to L100 is congested, measurement loops M3 and M5 indicate a change in the measured delay. Without listing all events, it can be shown that all cases of single losses of connectivity or single events of congestion influence only delay measurements of a unique set of measurement loops.¶
The measurement loops are best set up while there's no congestion. In that case, the congestion free RTDs of all monitored links can be calculated as shown above which later allows to estimate the queue-depth under congestion. A single congestion event adds queuing delay to the RTD measured of two specific measurement loops. The two measurement loops impacted indicate the congested interface and enable estimation of the queue-depth (in terms of seconds based on comparing actual and prior delay measurements). The per link RTD can be calculated while the network is operating without congestion, say at interval T0. Then as an example, assume a queue of an average depth of 20 ms to build up at interface L200 to L070 at interval T1. The measurement loops M5 and M6 are the only ones passing the interface in that direction. Both indicate an added delay along M5 and M6 of + 20 ms during a measurement interval T1 with congestion on this interface, while M1-4 indicate unchanged delays. The location of the congested interface is determined by the combination of the two (and only two) measurement loops M5 and M6 showing a significant delay increase. The average queue depth [s] = ( M5[T1] - M5[T0] + M6[T1] - M6[T0] )/2.¶
As mentioned there's a constant delay added for each measurement loop, which is the delay of the path passed from PMS -> L100 + L200 -> PMS. Please note, that this added delay is appearing twice in the formula resulting in the monitored link delay estimate of the example network. Then it is the RTD PMS -> L100 + RTD L200 -> PMS. Both RTDs can be directly measured by two additional measurements Cor1 = RTD ( PMS -> L100 -> PMS) and Cor2 = RTD (PMS -> L200 -> PMS). The monitored link RTD formula was linkRTDuncor = 3*Mx + My + Mz - Ms - Mt - Mu. The correct 4*linkRTDx = 4*linkRTDxuncor - Cor1 - Cor2.¶
If the interface between PMS and L100/L200 is congested, all measurement loops M1-M6 as well as Cor1 and Cor2 will see a change. A congested interface of a monitored link doesn't impact the RTDs captured by Cor1 and Cor2.¶
The measurement loops may also be set up between hub nodes L100 and L200, if that's preferred and supported by the nodes. In that case, the above formulas apply without correction.¶
The metric and methods specified below can be applied to monitor networks or sub-paths forming a hub and spoke topology. A single sub-path status change of type loss of connectivity or congestion can be detected. The nodes don't have to act as hubs or spokes, this terminology is only chosen to describe a topology requirement. In detail, the topology to be monitored MUST meet the following constraints:¶
Shared resources, like a Shared Risk Link Group (e.g., a single fiber bundle) or a shared queue passed by several logical links need to be considered during set up. Shared resources may either be desired or to be avoided. As an example, if a set of logical links share one parental scheduler queue, it is sufficient to monitor a single logical connection to monitor the state of that parental scheduler.¶
The methodologies sepcified by this document REQUIRE a measurement loop path overlay of all path delay measurement streams Fi, i in [1, 2...6] as defined in this section. In the follwing, a path delay measurement stream Fi is called measurement (loop) Fi for brevity.¶
The measurement loop set up to monitor all Sub-paths SPi is completed, if:¶
This document specifies sub-path monitoring within a closed domain by a controlled and pre-designed measurement loop set-up. The path traversed by the packet SHOULD be reported, as detecting data plane forwarding in line with the desired measurement loop set-up is essential for the metric to enable and verify accurate evaluation. See [RFC8287] for SR MPLS OAM and [ID.draft-ietf-6man-spring-srv6-oam] for SRv6 OAM.¶
Packets per measurement loop Fi are sent periodically by a temporal distance of IncT. For convenience, packets of the 6 measurement loops are assumed to be equally spaced at the sender too. Let's define the temporal distance IncF between two consecutive packets sent along to different measurement loops Fi and Fj at a single sender to be¶
IncF = IncT / 6¶
Further it seems useful to suggest IncF to be bigger than the largest measurement loop delay max (mi) under stable network operation (i.e., including some tolerance). Further assume the standard deviation of the measurement values mi to be much smaller than the delay mi, which is likely for a sub path being a regional or national link in many countries. Note that this definition isn't a strict requirement. Interpretation of results is however simplified by it. For the rest of the document assume¶
IncF > 2 * max (mi), i in [1...6], which results in¶
IncT > 12 * max (mi)¶
Discussion and reasoning for a reasonable smallest interval IncF in relation to max(mi) follows below.¶
To reduce the redundant information presented in the detailed metrics sections that follow, this section presents the specifications that are common to two or more metrics. The section is organized using the same subsections as the individual metrics, to simplify comparisons.¶
All metrics use the Type-P convention as described in [RFC2330]. The rest of the name is unique to each metric.¶
Refer to section 3.2. Metric Parameters: Type-P-* of [RFC6673]. The following parameters are added, enhanced or removed:¶
Type-P-SR-Path-Periodic-Delay¶
See section Section 4.2.¶
A sequence of consecutive time values. The value of a Type-P-SR-Path-Periodic-Delay is either a real number or an undefined (informally, infinite) number of seconds per singleton of each stream Fi.¶
Section 3.4 of [RFC7679] applies per singleton of each stream Fi. The additional information related to singletons of section 4.2.4 of [RFC3432] applies too.¶
See section 3.5 of [RFC7679]. One generalisation seems appropriate: a global satellite navigation system affords one way to achieve synchronization within usec.¶
Section 3.6 of [RFC7679] applies per stream Fi with one exception: at the Src host, select Src and Dst IP addresses, if IP-routing is applied, or select the proper functional IP-destination address if an [RFC8287] SR MPLS OAM packet format is applied. Further add the appropriate stack of Segment IDs MLAi determining the monitoring loop Fi and form a test packet of Type-P with these addresses and the segment stack.¶
Editors note: To be added based on existing loss metrics. A delay based approach indicating loss of a physical interface by detecting delay changes caused by re-routing can't be assumed to reliably cause unique delay change patterns under all circumstances (consider a shortest path routed multi-hop MPLS sub-path to be monitored rather than a link or a scenario where a bundle of 6 equivalent links is monitored connecting a single hub and spoke).¶
Type-P-SR-Path-Packet-Loss¶
See section Section 4.2.¶
The value of a Type-P-SR-Path-Packet-Loss is either a zero (signifying successful transmission of the packet) or a one (signifying loss) per singleton of each stream Fi.¶
Section 2.4 of [RFC7680] applies per singleton of each stream Fi.¶
Section 2.6 of [RFC7680] applies per stream Fi with one exception: at the Src host, select Src and Dst IP addresses, if IP-routing is applied, or select the proper functional IP-destination address if an [RFC8287] SR MPLS OAM packet format is applied. Further add the appropriate stack of Segment IDs MLAi determining the monitoring loop Fi and form a test packet of Type-P with these addresses and the segment stack.¶
This sections defines metric samples and metrics derived from samples.¶
To reduce the redundant information presented in the detailed metrics sections that follow, this section presents the specifications that are common to two or more metrics. The section is organized using the same subsections as the individual metrics, to simplify comparisons.¶
Type-P-SR-Path-Periodic-Delay-*¶
See section Section 5.3.¶
Given Ti0 and Tif and nominal inter-packet interval incT, those time values greater than or equal to Ti0 and less than or equal to Tif are then selected. At each of the selected times in this process, we obtain one value of Type-P-SR-Path-Periodic-Delay. The value of the sample is the sequence made up of the resulting [time, delay] pairs. If there are no such pairs, the sequence is of length zero and the sample is said to be empty.¶
The only definition required for this metric is a unique metric name.¶
Type-P-SR-Path-Periodic-Delay-Stream¶
The smallest sample Type-P-SR-Path-Periodic-Delay-Stream is one of two consecutively received values. These may be used to calculate a Segment Routed Path Delay-Variation (SRDV) singleton, defined below.¶
Type-P-SR-Path-Periodic-Delay-Variation¶
SRDV[i,j], for each sample of packets j and j-1 of stream Fi, j > 1, the delay variation between successive packets is calculated as:¶
SRDV[i,j] = Delay[i,j] - Delay [i,j-1],¶
j in [2,3...N] and N the total number of packets received at Dst. If one or more of the M packets sent by Src are lost, they are ignored for the metric, as no reasonable metric value is defined here. If N > 1, the metric is calculated for every valid packet received and the preceding one.¶
Evaluation statistics of differential SRDV metric samples may help to identify issues.¶
The only definition required for this metric is a unique metric name.¶
Type-P-SR-Path-Periodic-Delay-Variation-Stream¶
Given Ti0 and Tif, those time values greater than or equal to Ti0 and less than or equal to Tif are then selected. At each of the selected times in this process, we obtain one value of Type-P-SR-Path-Periodic-Delay. The value of the sample is the sequence made up of the resulting [time, delay-variation] pairs with time being set to the Dst timestamp of the Delay-Variation singleton, for which a valid singleton is calculated. If there are no such pairs, the sequence is of length zero and the sample is said to be empty. If N Delay singletons are captured and sampled N-1 Delay-Variation singletons are sampled during the same interval¶
Change point detection requires statistical defintions. These are provided below. The names of the statistics contain an "*" placeholder, which may be replaced by "Delay" or "Delay-Variation".¶
For a type-p metric, the mean is specified by:¶
SR-*Mean = (1/N) * Sum(from a=1 to N, value[a])¶
For a type-p metric, the Standard-Deviation Std is specified by:¶
SR-*Std = [1/(N-1)] * Sum(from a=1 to N, [SR-*Mean - value[a]]^2 )¶
The definition as given above requires a two-pass calculation per sample. Algorithms estimating the standard-deviation by one-pass calculation have been published and might be preferable, if metric singletons and samples aren't buffered or calculations need to be fast.¶
The packet loss ratio is a useful metric to characterise congestion.¶
To produce meaningful sub-path monitoring values, the measurement loop metrics are captured during a phase with stable networking conditions. In a backbone network domain, the absence of congestion often is a sufficient condition (frequent traffic shifts due to changes in routing and traffic engineering aren't expected). This may be different in a network based on a shared medium. It may be outright difficult in networks with frequently changing traffic management- and routing-policies.¶
In the following, the index CS indicates a statistic captured during a mesurement interval with stable routing and no congestion.¶
Capture a sample of delay values Type-P-SR-Path-Periodic-Delay-Stream of sample size N for each measurment loop Fi. As a rule of thumb choose N in [30, 100].¶
For each measurement loop Fi, calculate the following metrics characterising the monitored Sub-Paths during stable and congestion free network conditions:¶
A stable and uncongested network should produce rather constant delays, resulting in low standard-deviation values and almost zero mean delay variation. [Editors note: Add text to select the median of a small set of stream mean captures, like 5 samples captured consecutively.]¶
Example data was captured in a lightly loaded Gigabit network. 11 routers are passed per measurement loop. The sample size is 30 packets, more than 200 samples were captured per measurement loop. The loops are set up for a different purpose than specified here, they are picked due to a high number of passed routers. Note that SR-DV-Mean here refers to an abs(SR-DV-Mean) sample, thus small, positive, non-zero means result. The time unit is microseconds.¶
Metric|Quantile|SR-D-Mean|SR-D-Std|SR-DV-Mean|SR-DV-Std
------+--------+---------+--------+----------+---------
Loop1 | 95% | 34507 | 62 | 41 | 84
------+--------+---------+--------+----------+---------
Loop2 | 95% | 35104 | 45 | 34 | 49
------+--------+---------+--------+----------+---------
Loop1 | 50% | 34496 | 19 | 19 | 17
------+--------+---------+--------+----------+---------
Loop2 | 50% | 35088 | 15 | 14 | 12
------+--------+---------+--------+----------+---------
Loop1 | 5% | 34491 | 14 | 20 | 12
------+--------+---------+--------+----------+---------
Loop2 | 5% | 35080 | 13 | 12 | 9
------+--------+---------+--------+----------+---------
Example baseline metrics for an 11 hop measurement loop (quantiles refer to SR-D-Mean)¶
Delay outliers may occur at any time in any communication network, and the measurement system packet processing itself may also produce some. It is fair to expect only single outliers in a stable, not congested network. It may be worth to capture several consecutive SR-Path-Periodic-*-Stream samples and compare their statistics, before picking reasonable baseline metric values. Samples showing higher standard deviations (compare the 95% quantile values in the above figure to the 50% quantile values) may benefit from removing the maximum singleton value from the sample. This will smooth the mean and standard-deviation, and if the result then is closer to those of the majority of the samples, foster confidence in determining the baseline metrics. Depending on the preferred method of data-processing and storing, this may require capturing the sample maximum as a separate metric.¶
Within a single evaluation interval of identical Time T0 and Tf, SR-Path-Delay-MeanCSi(from now on DMeanCSi)is the mean delay of the measurement loop passing the monitored Sub-Path SPi by a round trip. Let's keep the indexig applied above, then Fj and Fk with captured mean delays DMeanCSj and DMeanCSk pass SPi uniderictional. Further, 3 measurement loops Fx, Fy and Fz don't pass Sub-Path SPi at all. The corresponding mean delays are DMeanCSs, DMeanCSt and DMeanCSu.¶
The the SR-Path-Sub-Path-RTD-Estimate of the Round Trip Delay along the monitored Sub-Path Fi, RTD_Fi, is¶
RTD_Fi=(3*DMeanCSi+DMeanCSj+DMeanCSk-DMeanCSx-DMeanCSy-DMeanCSz)/4¶
The asterisk stands for "Interface" as well as "Connectivity". If connectivity is lost and no path is available between two nodes, any packets to be transmitted will are dropped. A change in sub-path routes with a change in measurement loop delay indicitates a re-routimg event (a temporal loss in connectivity), not a long lasting loss of connectivity. Hence a change in measurement loop delays caused by a re-routed monitored sub isn't useful to derive a metric indicating connectivity loss on a monitored sub path (a sub-path-route-change metric might be of interest, but isn't within scope of this document).¶
Network changes like congestion or re-routing are often characterised by a change in the mean delay of a monitoring measurement. CUSUM (cumulative sum ) charts have been shown to be efficient in detecting shifts in the mean of a process [NIST]. The upper bound CUSUM is defined as:¶
Sup(t)-Fi-Delay = max(0,Sup(t-1) + xt - SR-Path-*-MeanCSi - ki)¶
with Sup(0) = 0, ki = Delta * SR-Path-*-StdCSi (Delta is a dimensionless integer number), xt = Type-P-SR-Path-Periodic-* singleton for measurement loop Fi at time t.¶
The actual SR-Path-Delay-Mean of Measurement Loop Fi is decided to be significantly above SR-Path-*-MeanCSi, if:¶
Sup(t)-Fi-Delay > h_SP, with h_SP = d*ki (d is a dimensionless integer number).¶
An analogus CUSUM controls changes to a lower mean delay (which may be caused by a re-routing event):¶
Slo(t)-Fi-Delay = max(0,Slo(t-1) + SR-Path-*-MeanCSi - xj - k)¶
The actual SR-Path-Delay-Mean of Fi is decided to be significantly below SR-Path-*-MeanCSi, if:¶
Slo(t)-Fi-Delay > h_SP¶
CUSUM chart based changepoint detection is sensible even to small changes in the mean. CUSUM charts offer a limited protection against single, isolated outliers. A cumulated sum only grows, if the controled process consistenly changes its mean (or standard deviation, respectively). Assuming constant physical minimum delays to characterise wireline communication networks, a change in standard deviation not affecting the mean delay doesn't seem to be caused by a change in networking conditions.¶
The measured delays will change once a Sub-Path route has changed, or once persistent congestion starts to fill a queue. Both indicate changes in the network. As the Sub-Pathes SPi form an overlay with designed properties, every network change affecting a sub-path creates correlated SR-Path-* metric changes. As the correspondance of network changes to Sub-Path metrics is known a-priory, detecting correlated SR-Path-* metric changes allows to locate the change.¶
In the absence of packet re-routing, packet loss is characterising a loss of connectivity. Packet loss requires a time threshold when to decide that an active measurement packet was lost, and consecutive loss requires receiver awareness, that packets have been sent (this argues for the sender to be the receiver, unless both comminicate fast and reliable out of band).¶
The preferred CUSUM parametrisation will depend on the kind of events to detected and on the outlier characteristics.¶
ki = Delta * SR-Path-*-StdCSi may be set to a value relevant high enough to exclude single outliers to trigger an alert, but low enough to indicate persistent changes in delay. The same holds for the to be picked for d.¶
A broader discussion on CUSUM parametrisation may be found in literature. Networking skills are required to parametrise CUSUM, as well as to interprete the results (notably to differ re-routing from congestion).¶
An interface along a single monitored Sub-Path SPi whose queue is persistently filled adds latency to measurement loop Fi and one of the two unidirectional measurement loops Fj and Fk passing Sub-Path SPi. Fj has been defined to pass SPi from Hub to Spoke and Fk pass SPI in opposite direction. Then SR-Path-Sub-Path-Congestion-Location metric for the traffic directed from "Hub to Spoke" along Sub-Path SPi is:¶
SPi_ConLoc_ij = Sup(t)_SPi_Periodic-Delay + Sup(t)_SPj_Periodic-Delay¶
And for the opposite traffic direction, from "Spoke to Hub":¶
SPi_ConLoc_ik = Sup(t)_SPi_Periodic-Delay + Sup(t)_SPk_Periodic-Delay¶
Note that another 10 SR-Path-Sub-Path-Congestion-Location metrics are calculated, one per monitored Sub Path and traffic direction. The evaluation can be simplified as follows:¶
Then Sub-Path SPi faces congestion in direction "Hub to Spoke".¶
Then Sub-Path SPi faces congestion in direction "Spoke to Hub".¶
Here, h_SP is a universal threshold in unit time to indicate a filling queue or a significant change in delay due to a Sub-Path reroute or another persistent change in topology (like e.g. automated Layer 1 / Layer 2 topology changes). Packets following SPx, SPy and SPz don't pass the congested interface of Sub-Path SPi.¶
The idea of this document is to monitor a set of sub-paths for a single case of congestion or a single loss of connectivity. If a single sub-path SPi looses connectivity, i.e., all packets are dropped in both sub-path forwarding directions, then three measurement loops mi, mj and mk fail to receive any traffic. A single interface congestion will add latency to mi and one of mj or mk, respectively. Still, if it is congestion of a single sub-path SPi interface causing additional latency, either mj or mk face no congestion and the one measured delay mj or mk should be within the expected range of values. Rather than basing a loss of connectivity metric on a "reliable" indication SR-Path-Packet-Loss on each measurement loop mi, mj and mk by waiting for Tmax to receive any of the missed packets, this allows for a reaction independant of a conservative packet loss threshold like Tmax. The idea is to judge on disconnectivity if no packet is received on all three measurement loops mi, mj and mk after the time interval the last single packet was expected to be received, if there was no prior indication of congestion.¶
If the spacing of packets along consecutive measurement loops Fi is IncF as defined within section Section 3.4, then under stable network conditions every measurement packet sent along measurement loop Fi is received, before the next measurement packet is sent along measurement loop Fj. If a measurement interval starts at T1 and none of the three measurement loops Fi, Fj and Fk received a packet within T1 + incT = T1 + 6 * incF, monitored Sub-Path i is disconnected. It doesn't matter, along which of the three measurement loops the first not received packet was sent (there's no order here).¶
incF > max (SR-Path-Delay-MeanCSi+ d * Delta * SR-Path-Delay-StdCSi ), i in [1...6]¶
With d and Delta being integer numbers as specified in section Section 10.4. If Fi and Fi+1 are measurement loops along which measurement packets are sent in consecutive order, this definition of incF ensures that the measurement packet sent along measurement loop Fi is received prior to sending the next measurement packet along measurement loop Fi+1 (under stable network conditions). The product d * Delta * SR-Path-Delay-StdCSi allows to set the preferred tolerance for outliers. It impacts the tradeoff between speed of detection and false positive ratio. With this parameterisation, the metric indicationg a loss of bidirectional connectivity along Sub-Path i is defined as¶
either zero or one (or some logical equivalent), where LofCi=1 indicates loss of continuity along monitored Sub-Path Fi and LofCi=0 indicates successful arrival of at least one packet sent along measurement-loop Fi, Fj or Fk within incT.¶
Under conditions of section Section 3.4, if at any sliding interval incT no singleton was received along measurement-loops Fi, Fj and Fk, no more packets are forwarded in any direction of monitored sub-path SPi.¶
Faster detection of disconnectivity is likely possible by a different metric definition, which likely will depend on the measurement-loop delay Mi, Mj and Mk. The metric chosen above allows for a simple parametrisation. Metrics allowing for a faster determination of disconnection are not within scope of this document.¶
The sub-path SPi is judged to be disconnected from the earliest time, when a packet was sent but not received on any of the three sub-paths Fi, Fj or Fk. The sub-path SPi is judged to be connected, whenever a measurement packet sent along one or more of the measurement-loops Fi, Fj and Fk is received again.¶
Fi = send time of a packet along measurement-loop Fi
i in [1...6]
Mi = receive time of a packet sent along Fi
incT interval between two packets sent along Fi
incF > max (Mi)
IncF IncT = 6 * IncF
__/\__ ___________________/\__________________
/ \ / \
+------+------+------+------+------+------+------+------+
t=0 1 | 2 3 4 | 5 6 | 7 | 8
F1 | F2 F3 F4 | F5 F6 | F1 | F2
M1 M4 M6 M1 |
|
At time 8, next packet should be sent along F2. |
No packets were received along F2, F3 and F5 yet. |
Indicates discontinuity along SP3 at time 8. <------+
Illustration of the sub-path disconnectivity metric; sub-path SP3 is link L100 <-> L070 of the example network Figure 1.¶
Note, if F2 sent at time 2 was received at time 2 + M2, but no more packet passing SP3 afterwards, discontinuity of SP3 is indicated at time 9, when F3 is to send the next packet. Also note that discontinuity of SP3 could be indicated as early as time 6 in the example. That requires a different metric. Basing the metric definition on incT however covers all potential intervals between relevant Fi, Fj and Fk.¶
A loss of connectivity is detected after a temporal distance of IncT, the time period between two packets beeing sent along the same measurement-loop Fi. IncT is specified as 6*IncF, where IncF is 2 times the largest measurement-loop delay in the absence of congestion. Hence a loss of connectivity is indicated after 12 * the largest measurement-loop delay.¶
Reliable indications of lost connectivity may be possible also at smaller timescales. The specification chosen seems to be simple as well as reliable and thus defines a starting point for advanced designs offering faster reaction.¶
If standardised, the metric will require an entry in the IPPM metric registry.¶
This draft specifies how to use methods specified or described within [RFC8402] and [RFC8403]. It does not introduce new or additional SR features. The security considerations of both references apply here too.¶