Troubleshooting VXC Connectivity

If you are experiencing connectivity issues, we recommend that you start by troubleshooting your Virtual Cross Connect (VXC) to isolate the issue. While the VXC will often display different symptoms, the root cause may be attributed to another area of the network.

Tip

You can verify VXC status from the Megaport Portal. On the Services page in the Portal, find the VXC and mouse over its icon. A message displays the status. (The color of the icon also indicates the service status.)

Troubleshooting actions

Action Steps
Check for CRC errors, packet drops, and device logs Interface statistics and logs can help identify which end of the cross connect is causing the fault, and the potential solution. For example, an increasing number of incoming errors on a network interface generally rules out that specific SFP (small form pluggable) and indicates a potential issue with other components of the cross connect.
Verify the Tx and Rx optical levels on the device Check the transmitted (Tx) and received (Rx) light levels. This health check enables you to validate physical connectivity. Some considerations:
  • If no Rx light is received, the service is down.
  • If you observe degradation of Tx and Rx light levels, the service may be interrupted. Megaport recommends that you check your physical connections.
  • If you are not transmitting or receiving light to/from Megaport, it may be caused by one of the following:
    • Fibre polarity issue – Verify by rolling the fibres at your end.
    • Connectivity issues within your environment or cross connect – Verify by performing physical loopback testing within your environment.
    • Connectivity issues within the Megaport environment – Verify by performing physical loopback testing from your environment towards Megaport.
Verify physical connections with the data center Open a ticket with the data center to perform the following tests:
  1. Check the cross connect for damages or cleaning, if needed.
  2. Ensure that the data center is transmitting adequate light outside of the demarcation point at its end of the connection. The data center should check the light at the demarcation point with a light reading meter.
  3. SFP models should align with the Interface Speed and Interface details in the Megaport Technical Specifications.
Verify carrier circuit status (if any) Some cross connects are set through one or multiple carrier network devices before reaching the Megaport network. Verify that the device interfaces in the cross connect path are free of errors and optic light readings are operating correctly.
Validate equipment performance While troubleshooting, Megaport does not have visibility or access outside the Megaport network. To verify that the cause of an issue is within the Megaport network, Megaport Support requires customers to validate the performance of their equipment. This includes:
  • Ensuring that the hardware specifications and limitations are compatible with Megaport Technical Specifications.
  • Monitoring the network traffic and the workload on the hardware to avoid congestion or degraded performance.
In addition, it is recommended that you validate the below performance to ensure it is operating as expected. If any anomalies are identified, capture the logs, graph details or any relevant error messages.
    Hardware
  • Optic (SFP type, speed, and wavelength) and fibre type
  • Port capacity
  • Switch, router, and firewall models
  • Firmware version
    Network
  • Traffic flow
  • Port utilization
  • CPU utilization
  • Configuration
  • Overall network design
If you identify any anomalies, capture the logs, graph details, and any relevant error messages.
Compare latencies Play video    Watch a 1-minute video on how to compare your latency to Megaport’s published latency.
Perform traceroute or other test to locate the symptom Traceroute is a network diagnostic tool that tracks in real-time the pathway taken by a packet on an IP network from source to destination, reporting the IP addresses of all the routers along the path. Traceroute also records the time taken for each hop the packet makes during its route to the destination.

Perform end-to-end traceroute testing
  • From the host that is originating traffic (A-End), start the traceroute to the destination host (B-End). Then run the traceroute from the destination host to the origin host. Commands and flags may differ by device.
Analyze the results
  • Look for potential asymmetric routing. If the traceroute results are not taking the same path, a traceroute will help pinpoint asymmetric routing somewhere in the network.
  • Are there any places in the traceroute where the response time has significantly increased? If so, are those delays within your network?
    Are there any firewalls or access list rules prohibiting traffic from reaching the destination?
Perform throughput tests iPerf is a cross-platform tool used to create standardized performance measurements and tune your network. iPerf has both client and server functionality and can create data streams to measure the throughput between two ends, in one or both directions.

Perform testing
Megaport recommends performing a 15 minute test on each side (the A-End client and B-End server, then the B-End client and A-End server) for a total of 30 minutes of testing and approximately 10 to 15 minutes between each test. This test must be run using UDP. Here is an example of the command to run on the A-End or B-End:

iperf3 -c <ip address> -b1000m -t 900 -u

Note: UDP streams must be used to measure throughput between the two ends of the connection without the overhead of TCP negotiation, congestion avoidance, and windowing.

Analyze the results
  • Look for potential asymmetric routing. A traceroute will help to pinpoint whether the traceroute results are taking different paths, which may indicate asymmetric routing somewhere in the network.
  • Are there any places in the traceroute where the response time has increased significantly? If so, are those delays within your network?
  • After testing, provide your interface statistics and take a screenshot of:
    • Traffic graphs (if possible)
    • Ingress/egress points in your network nearest to Megaport
    • Traffic graphs for B-End ingress/egress (if possible)
  • Specify which device(s), port(s), and VLANs on the network diagram the graphs relate to.

Next steps

If the troubleshooting actions do not resolve your issue, contact support. Before requesting assistance, collect the following information:

  • Troubleshooting results – Provide all the troubleshooting steps you have taken in detail. For example, if loops were placed, note their location and which direction they faced.
  • Port status and Tx/Rx optical levels on the device – Use the Megaport Portal to review your Port status. See a 1-minute video on how to view service status.
  • Data center ticket reference (optional) – Once the cross connect is installed the data center will send a completion notice. The notice will include the data center cross connect order number, which Megaport requires to provide to the data center technician. Also provide any existing ticket reference numbers that you have opened directly with the data center.
  • Source IP address and destination IP address – The source IP address is the IP address of the host that sent the packet. The destination IP address is the IP address of the host that should receive the packet.
  • High-level network diagram – Understanding how your network design is implemented and the connection into the Megaport network helps identify additional focus areas within the troubleshooting process. Provide a network diagram that includes all devices in the path; note each device’s involved IP addresses and VLANs.
  • Ping test results – Provide the output of each ping test performed on the service. Provide all output tests if you have multiple services related to different products (for example, a Port, VXC, or MCR).
  • Traceroute results – Provide traceroute results, indicating which side of the connection initiated the test and which side was the destination. We recommend that you use the A-End and B-End information from your VXC.
  • iPerf (throughput) test results – Provide all data based on the steps above and any relevant information related to the questions below:
    • Are you using traffic shaping on your network?
      If you are shaping, policing, or filtering traffic before it reaches Megaport, it can cause us to see only the shaped ingress traffic in the Megaport network. Customers and resellers must ensure that the equipment used outside of Megaport’s network can support the desired speeds.
    • Have you contacted the B-End of the connection to ensure that there aren’t any problems on that side of the path?
      Provide the case number if applicable. Once traffic is sent out from Megaport’s network interface to the provider interface, we no longer control that flow.
    • Are there any other providers involved, such as telco carriers? If a carrier is involved in the network, has a case been opened with them to investigate potential routing issues?
      Provide the case number if applicable. It is important to verify whether you are using a telco carrier to route traffic flow to/from your network to Megaport, as we can only troubleshoot flow through our devices. For example, we cannot account for any loss or other issue before it reaches our network.
    • If this is an Azure connection, are you using the Q-in-Q option on the Megaport Portal as described in Configuring Q-in-Q?
      Azure with Q-in-Q can be tricky, and must be configured correctly to send the traffic properly to Megaport and then on to Azure.
  • Packet capture logs (optional) – Packet capture (or PCAP) logs help collect network traffic, monitor bandwidth, detect malware, and assist in incident response. If relevant to the issue, provide packet capture logs for a greater understanding of your network.
  • MVE/SD-WAN configurations (if applicable) – Confirm the following:
    • Login credentials are accurate
    • The correct license and template/workflow are attached
    • The instance size and software versions
    • The MVE interface status, such as configured IP addresses and VLAN
    • For gateway and interconnect connectivity, check BFD sessions, bandwidth, and BGP status
    • Verify the MVE IP address in the Megaport Portal
    • Verify the configured bandwidth, VLAN, IP address and subnet masks, and ASNs
    • Validate connectivity details such as the interface status and BGP neighbor status

For more details on our SD-WAN providers, see Introducing MVE.


Last update: