Troubleshooting Throughput and Speed

If you notice any anomalies between the traffic being transmitted and the traffic you expect to receive, your network may be experiencing throughput or speed issues. We recommend that you run through the following troubleshooting steps to assist in isolating the cause.

Troubleshooting actions

Action Steps
Check interface or CRC errors and packet drops on the device Interface statistics and logs can help identify which end of the cross connect is causing the fault, and the potential solution. For example, an increasing number of incoming errors on a network interface generally rules out that specific SFP and indicates a potential issue with other components of the cross connect.
Verify the Tx and Rx optical levels on the device Check the transmitted (Tx) and received (Rx) light levels. This health check enables you to validate physical connectivity. Considerations:
  • If no Rx light is received, the service is down.
  • If you observe degradation of Tx and Rx light levels, the service may be interrupted. Megaport recommends that you check your physical connections.
  • If you are not transmitting (Tx) or receiving (Rx) light to/from Megaport, it may be caused by one of the following:
    • Fibre polarity issue – Verify by rolling the fibres at your end.
    • Connectivity issues within your environment or cross connect – Verify by performing physical loopback testing within your environment.
    • Connectivity issues within the Megaport environment – Verify by performing physical loopback testing from your environment towards Megaport.
Verify physical connections with the data center (reseat and replace SFP, clean and replace cables, loopback test) Open a ticket with the data center to:
  1. Check the cross connect for damages or cleaning, if needed.
  2. Ensure that the data center is transmitting adequate light outside of the demarcation point at its end of the connection. The data center should check the light at the demarcation point with a light reading meter.
Verify carrier circuit status (if any) Some cross connects are set through one or multiple carrier network devices before reaching the Megaport network. Verify that the device interfaces in the cross connect path are free of errors and optic light readings are operating correctly.
Validate equipment performance While troubleshooting, Megaport does not have visibility or access outside the Megaport network. To verify that the cause of an issue is within the Megaport network, Megaport Support requires customers to validate the performance of their equipment. This includes ensuring that the hardware specifications and limitations are compatible with Megaport’s Technical Specifications, and monitoring the network traffic and the workload on the hardware to avoid congestion or degraded performance. To ensure that hardware and your network is operating as expected, we recommend that you validate the performance of the following:
    Hardware
  • Optic (SFP type, speed, and wavelength) and fibre type
  • Port capacity
  • Switch, router, and firewall models
  • Firmware version
    Network
  • Traffic flow
  • Port utilization
  • CPU utilization
  • Configuration
  • Overall network design
If you identify any anomalies, capture the logs, graph details, or any relevant error messages.
Perform traceroute (or other test) to locate the symptom Traceroute testing can help to determine if a destination is reachable. Traceroute sends a sequence of User Datagram Protocol (UDP) packets between two points and shows you the route the packets take. Traceroute also measures the transit delays of packets across an IP network.

Perform end-to-end traceroute testing
  • From the host that is originating traffic (A-End), start the traceroute to the destination host (B-End). Then run the traceroute from the destination host to the origin host. Commands and flags may differ by device.
Analyze the results
  • Look for potential asymmetric routing. If the traceroute results are not taking the same path, a traceroute will help pinpoint asymmetric routing somewhere in the network.
  • Are there any places in the traceroute where the response time has significantly increased? If so, are those delays within your network?
    Are there any firewalls or access list rules prohibiting traffic from reaching the destination?
Perform throughput tests iPerf is a cross-platform tool used to create standardized performance measurements and tune your network. iPerf has both client and server functionality and can create data streams to measure the throughput between two ends, in one or both directions.

Perform the test
Megaport recommends performing a 15 minute test on each side (the A-End client and B-End server, then the B-End client and A-End server) for a total of 30 minutes of testing and approximately 10 to 15 minutes between each test. This test must be run using UDP. Here is an example of the command to run on the A-End or B-End:

iperf3 -c <IP address> -b1000m -t 900 -u

Note: UDP streams must be used to measure throughput between the two ends of the connection without the overhead of TCP negotiation, congestion avoidance, and windowing.

Analyze the results
  • Look for potential asymmetric routing. A traceroute will help to pinpoint whether the traceroute results are taking different paths, which may indicate asymmetric routing somewhere in the network.
  • Are there any places in the traceroute where the response time has increased significantly? If so, are those delays within your network?
  • After testing, provide your interface statistics and take a screenshot of:
    • Traffic graphs (if possible)
    • Ingress/egress points in your network nearest to Megaport
    • Traffic graphs for B-End ingress/egress (if possible)
  • Specify which device(s), port(s), and VLANs on the network diagram the graphs relate to.

Next steps

If the troubleshooting actions do not resolve your issue, contact support. Before requesting assistance, collect the following information.

  • Troubleshooting results – Provide all the troubleshooting steps you have taken in detail. For example, if loops were placed, note their location and which direction they faced.
  • Source IP address and destination IP address – The source IP address is the IP address of the host that sent the packet. The destination IP address is the IP address of the host that should receive the packet.
  • High-level network diagram – Understanding how your network design is implemented and the connection into the Megaport network helps identify additional focus areas within the troubleshooting process. Provide a network diagram that includes all devices in the path; note each device’s involved IP addresses and VLANs.
  • Ping test results – Provide the output of each ping test performed on the service. Provide all output tests if you have multiple services related to different products (for example, a Port or VXC).
  • Traceroute results – Provide traceroute results, indicating which side of the connection initiated the test and which side was the destination. We recommend that you use the A-End and B-End information from your VXC.
  • iPerf (throughput) test results – Provide all data based on the steps above and any relevant information related to the questions below:
    • Are you using traffic shaping on your network?
      If you are shaping, policing, or filtering traffic before it reaches Megaport, it can cause us to see only the shaped ingress traffic in the Megaport network. Customers and resellers must ensure that the equipment used outside of Megaport’s network can support the desired speeds.
    • Have you contacted the B-End of the connection to ensure that there aren’t any problems on that side of the path?
      Provide the case number, if applicable. Once traffic is sent out from Megaport’s network interface to the provider interface, we no longer control that flow.
    • Are there any other providers involved, such as telco carriers? If a carrier is involved in the network, has a case been opened with them to investigate potential routing issues?
      Provide the case number, if applicable. It is important to verify whether you are using a telco carrier to route traffic flow to/from your network to Megaport, as we can only troubleshoot flow through our devices. For example, we cannot account for any loss or other issue before it reaches our network.
    • If this is an Azure connection, are you using the Q-in-Q option on the Megaport Portal as described in Configuring Q-in-Q?
      Azure with Q-in-Q can be tricky, and it must be configured correctly to send the traffic properly to Megaport and then on to Azure.
  • Packet capture logs (optional) – Packet capture (or PCAP) logs help collect network traffic, monitor bandwidth, detect malware, and assist in incident response. If relevant to the issue, provide packet capture logs for a greater understanding of your network.

Last update: