How the two main network performance data sources compare in troubleshooting, latency, forensic analysis, and more.
Network engineers face the constant challenge of maintaining application performance, preventing security breaches, and solving complex issues. To do this effectively, they need access to data visualizations that help them make informed decisions. Two of the most powerful data sources available today are NetFlow (IPFIX) and Packet Data, which have helped gain network visibility for years.
By analyzing the strengths and weaknesses of both methods, we see specific scenarios where each one excels. Sometimes engineers work with one or the other. But, what are we missing if we only have packet data? What visibility gaps does a NetFlow solution have? What is best for security versus performance analysis? One thing is certain: network professionals will find many uses for these two visibility methods.
NetFlow
NetFlow is a technology that was originally developed and implemented by Cisco Systems in its Layer 3 switches and routers. Over time, other vendors began to implement similar network monitoring methods, creating the need for a standard.
This standard was published in 2008 by the IETF and defines the export of Internet Protocol Flow Information (IPFIX). There are other versions in use, such as jFlow, sFlow, and AWS VPC Flow logs, however, for simplicity, we refer to all flow-based protocols in this document as NetFlow.
Traffic Intelligence
NetFlow is a method for reporting on traffic passing through infrastructure devices. Rather than deploying probes throughout the environment to capture, analyze, and report on traffic behaviors, NetFlow-enabled devices can send statistics to a central collector that converts the protocol into useful and readable tables and graphs. Core routers and switches are an excellent place to collect data about network traffic, as they are inline and handle packets as they are transferred along the route.When a packet enters an interface with NetFlow, it is determined whether it is part of an already-seen data flow, or if it is the first packet of a new flow. If it has been seen before, only the number of packets and bytes observed will be incremented and the packet will be sent to the next process. If the packet is the first in a new flow, a NetFlow record will be created. Any subsequent packet from the same flow will be counted in that record until it is exported to the collector.
Packet attributes that define the flow
There are seven main packet attributes, common to all NetFlow versions, that define a flow:- Source IP address
- Destination IP address
- Source Port Number (TCP or UDP)
- Destination Port Number (TCP or UDP)
- IP Protocol ID
- Differentiated Services value
- Ingress interface
Generating a flow record
When a packet enters the router, its attributes are examined to determine if it belongs to a new flow. If the attributes exactly match any previous packet, it is classified as part of a previous flow and its byte length will be added to the flow record.
Traffic passes through NetFlow-enabled devices. Flows are created from packet attributes and then condensed into a database called the NetFlow cache.
Any change in the attributes will result in the creation of a new flow record. The record is fairly small and contains basic statistics about the flow: the seven attributes, the number of packets, and the amount of data that was seen.
For example, if a client opens two TCP connections to a server, the client-side port number will be different. Although all other values in the two packets could be identical (source IP, destination IP, etc.), the difference in source port numbers will force the creation of two different NetFlow records.
After one minute, devices will export records from the NetFlow cache memory, as well as completed TCP connections (FIN or RST), active flows with a one-minute timeout, or inactive flows (which will be marked as completed).
Routers typically export many flows in a single NetFlow packet, so little bandwidth is required to report traffic statistics from a link.
Benefits of NetFlow
NetFlow allows us to see who, what, and when in network traffic without the need to capture all traffic at the packet level. Although we could analyze the same statistics with packet data, NetFlow offers a much lighter method to monitor network traffic, enabling long-term storage. Collectors can store from one month to several years of information in a reasonably sized database.
NetFlow is one of the key data sources for ObserverGigaFlow to illustrate relationships in the infrastructure and analyze network routes.
Ideal scenarios for using NetFlow over Packet data are:
- Security assessments
- Basic traffic analysis
- Bandwidth monitoring
- Office and branch visibility
- Capacity planning and link sizing
- Congestion troubleshooting
Packet Data
Packet data has been the gold standard for network visibility since the first analyzer was invented. The common expression of network analysts is “packets never lie”. They show what is actually happening, detailing all protocols, conversations, and timing involved. Packet collection and obtaining true history enable troubleshooting that would otherwise remain indefinitely unresolved.
However, capturing packets, storing them, and interpreting them can be an exhaustive task. In today’s networks, 10/40/100 Gb links are now standard and create digital highways that can be difficult to manage without proper tools. For example, a 10 Gb link running at 50% utilization will transmit over 180 GB of data in five minutes. For a packet analyzer to store data at this speed, it needs a lot of disk space and processing power.
The right analysis tools help get more value from your network. Packets provide an unprecedented level of visibility, filling gaps that other technologies, like NetFlow, leave completely open.
The unique visibility of packet data
Packets tell us the exact microsecond when a packet passes a capture point. This allows us to precisely measure application response time, validate network latency, determine the cause of delay in slow applications, and much more.
By analyzing captured packets, ObserverApex breaks down response time and delay in visualized network conversation along with end-user experience score and issue analysis
Having complete packet headers allows tracking issues that can be detected using IP and TCP fields. With NetFlow, most header data is not retained, making it difficult to troubleshoot performance issues beyond network congestion.
Critical forensic data from the packet payload will show application request and response codes, URLs, file names, database errors, and more. Objects, such as images and audio calls, can be extracted and replayed, allowing you to see exactly which component of the application is affected by poor performance. When investigating and mitigating an attack, the payload can be critical to analyzing which systems were compromised and how, and what data was recovered.
With a NetFlow analysis solution, these details are not retained for the analyst, which can lead to guessing rather than troubleshooting.
Scenarios where packets are key
Packets should be captured and analyzed in the following scenarios:
- Application performance analysis
- Network latency measurement
- Server delay measurement
- Low performance analysis
- Security forensics - Payload
- Application error investigation
- TCP retransmission monitoring, out of service
- Slow network troubleshooting
NetFlow and packet data allow you to visualize network traffic, enabling you to mitigate attacks, troubleshoot performance issues, and stay aware of issues that affect the business. Depending on the problem scenario, one or the other will be better, however, both together create a complete picture of what is really happening in your network.
Be sure to leverage these two data sets in your network to maximize the visibility you need to find and resolve issues quickly.
Adaptation and translation of the original document, NetFlow vs. Packet Data, Chris Ger.