On Sunday morning, December 8th, 2019, Paul Custodio received an alert that 11 customers had lost internet service at the same time. Initially, he suspected a damaged splice or case.
However, the issue was much bigger than that.
A fire under the street had destroyed numerous cables for many different internet service providers.
Paul, with the help of Shannon Quigley, sprung to action. They opened their fiber management software, OSPInsight, and plugged OTDR traces into the Find Fault tool. They found a common point of failure for all of their customer’s buildings that had lost service.
Looking at the digital map of their network, they were able to pinpoint exactly where the outage had occurred. Paul and Shannon immediately dispatched a technician to the scene.
The Find Fault enables users to combine OTDR inputs with geographic representations of their networks.
Thanks to Paul and Shannon’s quick thinking, Pilot Fiber was the first internet service provider on-site. Their team arrived to find an open manhole, a massive web of burned cables, and the electric company, that would ultimately control the order at which internet service providers could address network issues.
“Being first on-site means that we were first to be able to go in.”
Because Pilot Fiber was the first to arrive, they would have been the first to go in for fixes. Unfortunately, the electric company was unable to allow access to anyone as a high voltage circuit was still active. It had to be shut down before any technicians could address downed cables.
The Pilot Fiber team decided to explore other potential solutions to get their network up and running more quickly in case access to the damaged manhole took longer than expected.
Fortunately, Paul and Shannon had powerful software at their fingertips that easily allowed them to try various rerouting scenarios. Using digital tools and mapping capabilities, they determined that the best path forward for their clients was to place new cables and circumvent the damaged site. They were able to see their network in its entirety and thoroughly evaluate different options for restoring services to their 11 clients.
Using OSPInsight, Paul and Shannon designed a reroute that involved two sections with 3,000 total feet of new cable. OSPInsight’s integration with mapping software helped the team visualize the most efficient way to install the cable for all 11 clients that had lost service.
Using their fiber management software, Paul and Shannon were able to run a report, called the Taper Report, that provided detailed intel on how all of their existing fibers were being used throughout the network. The Taper Report highlighted every end user, location address, fiber priority, end point, and current status so that it was clear who would be impacted by the rerouting.
Additionally, rather than assess buildings and routes individually, Paul and Shannon were able to see how everything fit together in a single application.
“The Taper Report was instrumental in helping us figure out which were the affected fibers and which were the affected customers that we had to reroute. It made it really easy, in a tabular format, to see which fibers we had to cut over from the feed to the new output”
The Taper Report provides a comprehensive look at how all fibers are being used within a network.
The Splice Key Editor tool also came in handy as Paul and Shannon determined how to integrate the reroute into the existing network without damaging the backbone architecture.
After, they generated the necessary work orders for the field techs and put the team to work. In just nine hours, Pilot Fiber was able to identify the source of a network issue, design a reroute, and implement a solution.
“I can’t even imagine how much more complicated the engineering would have been. OSPInsight made it really easy...to not have to worry about the tool you’re using was a real relief in this emergency situation.”
Thanks to the hard work and creativity of the entire Pilot Fiber team, Paul, Shannon, and the field technicians restored service to all clients in under 72 hours. They used powerful fiber management software technology to quickly diagnose the central issue, plan a reroute, and execute a complex plan in a few short days.
The Splice Key Editor presents splicing information in a table format so that operators can see all critical splices in one place.
08:15 am: Notified that 11 buildings were down with ~70 subscribers
12:00 pm: Pilot Fiber is the first internet service provider on-site
03:30 pm: Team decides to reroute given access delays
05:00 pm: Technicians arrive to begin new cable placement
10:00 pm: First cable section placement completed
03:00 am: Splice crew begins work on the new cable
02:30 pm: Second cable section placement is completed
08:00 am: Service is restored to 9 out of 11 buildings
11:30 am: Service is restored to the 10th building
04:00 am: Service is restored to the last Pilot Fiber building
After such a significant event, most internet service providers would expect backlash or criticism from clients. However, the Pilot Fiber team experienced no negative repercussions.
They acted quickly and communicated transparently throughout the process. They kept clients up to date on their progress and were able to restore services days faster than other providers.
Without OSPInsight, several steps along the way would have taken longer or been more challenging. Identifying the primary source of the issue, engineering the reroute, creating work orders, and splicing new cables would have been more complex with less sophisticated software.
Pilot Fiber teammates, Paul Custodio and Shannon Quigley, used OSPInsight to circumvent a major network malfunction in New York City and restore services to businesses as quickly as possible.
Find Fault Tool
The Find Fault tool was instrumental in determining the common source of the issue that caused 11 buildings to lose service.
The Taper Report helped the Pilot Fiber team understand exactly which clients were affected by the manhole fire, as well as which cables needed to be rerouted.
Splice Key Editor
The Splice Key Editor helped Paul and Shannon design the reroute efficiently and accurately.
Pilot Fiber is a New York-based fiber network owner and developer that supports businesses with high-speed, reliable internet services.