Enabling Silent Telemetry Data Transmission with InvisiFlow
Enabling Silent Telemetry Data Transmission with InvisiFlow Yinda Zhang, Liangcheng Yu, Gianni Antichi, Ran Ben Basat, and Vincent Liu NSDI'25
Watch out for symptoms of techno-Eeyore syndrome (TES):
You frequently feel like all of the low hanging fruit has been picked
Someone has recently asked you: “if this is such a good idea, why aren’t other companies doing it?”
You believe that the only remaining path for innovation requires spending a significant fraction of global GDP on new datacenters
If you are experiencing TES, then this paper will snap you out of it. Herein lies a wonderfully elegant solution to a real-world problem.
Schrödinger's Network Telemetry
The task at hand is collecting comprehensive network telemetry without disrupting the network itself. Services, operating systems, NICs, and switches are all great sources of telemetry. A global view of network telemetry could provide valuable insight into the behavior of a network.
Collecting information about network activity requires sending telemetry data over the network in question (unless one wants to build a separate network just for telemetry, which would be expensive). The problem this paper addresses is: how to collect such telemetry without altering the behavior of the network itself? The paper cites prior work which claims that enabling network telemetry can degrade application-level network throughput by 20%.
Telemetry Flows Like Water
Here is the refreshingly elegant solution. Designate one or more servers in the network as telemetry collector sinks. These sinks are the ultimate destination for any packet containing telemetry data. Any device which produces telemetry data is called a source. Sources produce network packets which contain telemetry information, and those packets make their way through the network until they reach a sink which consumes them.
The magic of this system is that when a source produces a telemetry packet, the address of the sink is not known. The packet meanders through the network (on uncongested links) until it arrives at a sink. The paper makes a great analogy: telemetry packets flowing through the network are like drops of water flowing down a mountain toward the ocean.
Each component in the network does its part via the following steps:
Maintaining a dedicated buffer (i.e., a telemetry buffer) for telemetry packets which are waiting to continue their journey through the network
Periodically sending pull requests to neighbors (networking equipment physically connected to the component in question). This is somewhat like PFC, where a NIC can send flow-control data to the switch it is connected to (and vice versa). This kind of pull request has nothing to do with a Git pull request. A pull request informs neighboring network components how full the telemetry buffer is. In the hydrodynamic analogy: the fullness of a telemetry buffer represents how high a networking component is above sea level.
Responding to pull requests from neighbors. When a network component receives a pull request, it checks to see if its own telemetry buffer is more or less full than the neighbor’s buffer. If the neighbor’s buffer is less full, then telemetry packets are moved to the neighbor. Telemetry packets are only sent in windows of time where there are no application-level packets to send on a link (i.e., telemetry packets fill in the empty space on each link).
Telemetry sinks send pull requests when they are ready to ingest more telemetry data. When a telemetry packet arrives at a sink, the sink processes the packet, and the packet disappears from the network.
The author’s call this “gradient-based transmission”; the differences in telemetry buffer utilization create a gradient, and packets descend that gradient. If there are many paths between a particular switch and a sink, this gradient information will cause the switch to choose the least-congested path.
The only hiccup here occurs when most telemetry buffers are empty most of the time. In this case, the gradient is near zero everywhere, and telemetry packets can oscillate rather than make forward progress to a sink. The fix for this is to determine the distance (in network hops) between each network component and the nearest sink. This distance is used to bias the gradient toward sinks.
Results
The authors were able to use P4 to implement this algorithm on Wedge100BF-32X switches.
Fig. 7 contains some great results:
Dangling Pointers
It seems like this could be generalized to more than just telemetry. If there is ever a situation where a network packet could be sent to one of a set of destinations, it may be beneficial for the sender to not pick a destination and instead organize the behavior of each network component such that the network has an emergent property that causes packets to take an uncongested path toward one of the destinations.