Beyond Redundancy: Why Last Month’s AT&T Outage Sucked So Bad

Ryan Burns / Yesterday @ 12:34 p.m. / Business , Technology

It’s been more than a month since vandals severed AT&T’s fiber optic line in a remote area south of Ukiah, sending untold tens of thousands of customers into an information blackout. (“Untold” because AT&T isn’t telling. A spokesman said hard numbers might encourage more vandalism.)

The outage affected wireless and wireline service in Mendocino, Humboldt and Del Norte counties, knocking out at least seven 911 call centers, disabling credit card machines and ATMs and bringing many government offices and local businesses to a standstill.

Many doctors’ offices, pharmacies and medical clinics were unable to access computerized patient records, make phone calls or contact specialists. For example, a man suffering a medical crisis couldn’t fax test results to his specialist in Portland, nor could he reach the doctor by phone.

In the days after the outage, locals recounted the impacts on Facebook. A local tech company employee was talking to a potential client when the phone cut out mid-call. “It’s hard to be competitive and close deals when phone lines are down,” he wrote.

A woman said her friends had trouble getting food since EBT and credit card systems weren’t functioning.

Librarian Ruth Moon noted, “Students taking online courses were unable to access course materials, delaying their completion of homework and assigned readings.”

“Imagine the millions of dollars of productivity we lost yesterday on the North Coast due to a lack of redundancy,” wrote local attorney Bill Barnum.

An emergency medical dispatcher for Southern Trinity Area Rescue was unreachable at 5 p.m., when she was scheduled to go on-call. Unbeknownst to her, there was a gap in on-call services while the clinic tracked down another volunteer dispatcher, who was on Verizon. “We have no way of knowing if any parties were unable to contact the clinic or STAR [Southern Trinity Area Rescue] number with an emergency,” the woman wrote.

The blackout was expensive, dangerous and more than a little frustrating. Most of us here in Humboldt County thought the days of such debilitating internet and phone outages were behind us. We suffered a series of them last decade, including severed fiber-optic lines in 2006 and 2007 that, like this recent outage, crippled not only the internet but also phone service, credit card machines and ATMs.

Back then, the vast majority of broadband internet in the county got routed through a single fiber optic “trunk line” running north-south, which meant that every time the line got severed — be it by an errant backhoe bucket or a massive mudslide — darkness descended.

But in late 2011, the community’s cries for broadband redundancy were answered when a company called IP Networks finished construction of a second “trunk” line running east-west, a project that cost $14.4 million, including almost $5.8 million from the taxpayer-funded California Advance Services Fund (CASF).

Or at least we thought our cries for redundancy had been answered. Turns out it’s more complicated than that.

Having two separate fiber-optic arteries does not automatically mean redundancy, which is why that word irks some of the area’s more techno-savvy residents. The second line merely created another pathway, increasing our region’s route diversity and allowing for more redundancy potential. But if your personal internet service provider isn’t leasing space on at least two routes, or if you’re not buying service from two different providers using two different routes, or if your service doesn’t have the infrastructure for automatic failover, then you don’t have redundancy. (Even then, if there’s an outage closer to your location — in the so-called “middle mile” or “last mile” of your connection — your service will go down.)

Suddenlink customers weren’t affected in last month’s outage, and a spokesman said the company leases space on both the north-south line and the east-west line. “We have taken steps that would allow us, in many instances, to avoid customer service interruptions,” he said. 

Local company 101Netlink offers redundancy because it leases space on the east-west line and uses microwave infrastructure to route traffic south to another east-west line that connects in Ukiah.

But here’s where things get murky. AT&T supposedly has built-in redundancy, too. The company told this reporter more than three years ago that it was leasing space on the east-west route, presumably as backup for connections through the north-south route that it owns. AT&T boasts of its “redundant fiber network” in advertising and informational materials, and indeed, as the Outpost reported during this latest outage, some AT&T customers weren’t affected at all.

Why? That’s a tough question to answer since all telecom companies treat the details of their infrastructure as proprietary information. Despite repeated attempts to get details from AT&T, a spokesman remained vague about many aspects of the situation.

“Our network is designed to provide back-up service wherever possible for all customers, taking into account factors such as geography, the route data must travel, and age of existing facilities,” the company’s official statement read. “In rare instances, customers with large or sensitive facilities may opt to purchase back-up fiber service where available for added redundancy.”

Is that the key? Were the only AT&T customers left standing the ones who purchased backup fiber service? Some businesses and agencies, including Redwood Capital Bank and Humboldt State University, have taken measures to ensure coverage during such outages, purchasing space on dedicated backup routes. Homes and businesses in the vicinity of such big-time customers often benefit because they’re served by the same circuits.

Follow-up emails to AT&T were answered by Steven Ramirez, a spokesman for a PR agency called Beyond the Arc, which handles AT&T’s public relations in northern California and Nevada. “For this specific cable that was cut,” Ramirez wrote, “we know some circuits served by the damaged fiber had a backup route where others did not.”

Customers wanting added protection, Ramirez said, can buy local access “or even dedicated end-to-end routing engineered to specific needs.” How much does that cost? “These are typically custom options and priced according to the design and the service provided.”

How can individual customers and businesses know whether or not they’ll be protected during the next outage? Ramirez declined to get specific. “[W]e do not have a customer-by-customer breakdown and do not share specific sensitive network infrastructure details publicly.”

Earlier this week, AT&T submitted an official report to Humboldt County supervisors, recounting certain aspects of the outage and restoration, which, judging by the report, was a harrowing experience indeed. 

“AT&T work crews overcame significant obstacles posed by the remote location of the damage, including cold weather and the presence of rattlesnakes,” the statement recounts. Despite these dangers, the workers bravely forged ahead.

“The nearest road to the area is a twenty to thirty minute walk away. Technicians teamed up with a local fire department’s railroad trolley operator, who used the adjacent railway to haul in power generators, fiber fusing machines, building cases for splicing, a backhoe, and all the new fiber that would be needed for repairs.

“With the necessary equipment in place the crew began splicing the 96 fibers of the severed cable and worked through the night using light and power from generators while fending off against rattlesnakes and without any space heaters. By sunrise the following morning, the crew had restored most services and a relief crew arrived to complete the remaining work.”

(Read the entire nail-biting account here.)

It took a little more than 24 hours to restore all wireline services. Wireless services weren’t fully restored until 2:30 Friday afternoon. Remote fiber line switches in Hopland and Eureka were isolated, leaving customers in those areas with dial tones but unable to make or receive phone calls outside their local areas, including calls to 911. 

“The Hopland switch was restored at on 9:55 p.m. Thursday and the Eureka switch was restored at 8:45 a.m. on Friday,” the report states.

AT&T defends its record of reliability in the report, noting that during 2014 the company received an average of just two trouble reports per 100 lines in Humboldt and Mendocino counties, “well within the CPUC’s guidelines.”

CPUC is the California Public Utilities Commission, which has also been rather short on information about the outage. Spokesman Christopher Chow told the Outpost in an email, “We are not an active participant in the criminal investigation that caused the specific outage in question.”

The agency monitored AT&T’s efforts to restore service, and it has received more detailed information from the company in the month since. But it’s not releasing that information publicly. “Details about the incident and its specific impacts were provided by AT&T to the CPUC as confidential information,” Chow said.

Why would that information be confidential? After all, it was given to a regulatory government agency charged with serving the public interest. “The information is confidential because the investigation is still going on,” Chow said. “You should ask AT&T.”

Where does that leave customers? It depends on their size and finances. Last year the county commissioned a report that spelled out the local broadband situation and, in retrospect, could have served as a warning for last month’s outage. Prepared by Tina Nerat of NeraTech, the report, which can be read online here, outlined a range of concerns raised by local tech support companies. Among them: “Businesses need to do more contingency planning for emergencies. Few companies have business continuity plans, hot sites, or alternative bandwidth.”

The report also noted, “Complacency has increased about the need to install backup bandwidth now that there are multiple middle mile routes out of the region.” In other words, many people just assumed they had backup bandwidth since the technology now exists here.

Larger businesses and government agencies like Redwood Capital Bank and HSU may have set up dedicated symmetrical services that offer an extra degree of protection, but, the report notes, “Many local businesses cannot afford to purchase backup bandwidth in case of disasters.” Plus, some of the county’s more remote areas simply don’t have the option of multiple providers.

Seth Johannessen, owner of 101Netlink, said that, to a large extent, local businesses get what they pay for. “If Internet connectivity is mission critical or really valuable, you need two connections, and you need hardware installed by a network management company,” he said.

That way, if one line goes down, you have the option of a backup line and the failover system to connect you to that second line automatically. Johannessen’s company offers that, and he suggested customers call their own provider to ask specific questions about their systems.

It’s also important to consider the reliability of each internet service provider and the response time during outages, Johannessen said. 

The Mendocino County Sheriff’s Department continues to investigate the vandalism that led to last month’s outage. AT&T officials will soon deliver a report to Humboldt County supervisors, according to 1st District Supervisor Rex Bohn. State Senator Mike McGuire called for “a long-term fix to ensure that there is redundant 911 emergency phone service throughout California.” And Connie Stewart, coordinator for the Redwood Coast Connect Consortium, is working with regional partners to advocate for greater broadband deployment, adoption and policy in the region.

For a look at that group’s checklist, click here.

For now it’s clear that our redundancy celebrations were premature. Humboldt County still has a ways to go before achieving reliable and truly redundant broadband access for the majority of residents.

Note: This post was updated from an earlier version to include an updated response from a Suddenlink spokesman.


(VIDEO) Here’s a Forest Ranger Performing an Original Rap About An Endangered Bird

Ryan Burns / Thursday, Oct. 8 @ 4:44 p.m. / Animals , Nature

The above video, featuring “Ranger Ryan,” was uploaded to the Redwood National Park Service’s YouTube page a couple weeks ago.

Ranger + gangsta = “Rangsta,” is what we’re guessing here. Way to represent.

(UPDATED) Sheriff’s Office SWAT Operation Underway at Pine Street and West Del Norte in Eureka

Hank Sims / Thursday, Oct. 8 @ 3:32 p.m. / Crime

UPDATE, FRIDAY MORNING: The two people briefly detained yesterday at the scene of this police action were questioned and released, according to Lt. George Cavinta of the Humboldt County Sheriff’s Office. 

Cavinta said that the search warrant served at West Del Norte and Pine was part of an “ongoing investigation.” He couldn’t release any information about that investigation, but the people put in handcuffs outside their home yesterday were not its targets, nor are they suspected of any crime.


UPDATE, 4:08 p.m.: The streets have been opened. Investigators conducting interviews.


UPDATE, 4:04 p.m.: Friend of the Outpost Kanan Best captured video of deputies taking one person into custody:


Two people have been detained so far, says the Outpost’s Andrew Goff.


Photo: Andrew Goff.

A heavily armed squad of police officers has guns trained on a house on Eureka’s West Side at the moment.

Humboldt County Sheriff’s Office spokesperson Selena Zorrilla-Mendoza told the Outpost moments ago that she has no information on what her department’s officers are doing on the scene, and had only been informed of it moments before the Outpost called.

Lt. George Cavinta, on the scene, tells the Outpost’s Andrew Goff that deputies are there to serve a search warrant.

Pine Street is closed between West Del Norte and Wabash.

More as we get it.

Photo from a reader.

(VIDEO) UFC Champion Ronda Rousey Reaches Out to Paralyzed Fighter Steve Watts

John Ross Ferrara / Thursday, Oct. 8 @ 3:02 p.m. / Sports

Number 1 of 2It has taken me a while to respond to Ronda Rousey’s kind message to me sent over a week ago. The original video is below, followed by my response.

Posted by Steve Watts on Wednesday, September 9, 2015

When Steve Watts awoke early one morning last month, he turned to see his phone glaring with a strange text message.

“It was five in the morning and I just happened to roll over and see the phone light up,” he told the Outpost. “Of course I couldn’t move, but I saw it read ‘Ronda Rousey.’”

Watts is most recently known for sustaining an injury that left him paralyzed from the neck down while fighting in the Cher-Ae Heights Casino “So You Wanna Fight” mixed martial arts tournament last year.

Steve Watts before his injury. Watts’ Facebook profile picture.

He said in a phone interview today that he was stunned to see UFC Bantamweight Champion Ronda Rousey reach out to him in a personal video. Unknown to Watts, a mutual friend who works in the entertainment business shared his story with the sports icon.

Soon after, Watts received the video from Rousey wishing him well on his recovery.

“Hi Steve, my name’s Ronda,” Rousey said in the video. “I stumbled across your story. A friend of mine sent it to me and I just wanted to wish you the best possible luck on your recovery. I can’t possibly understand everything that you’re going through, but if I’ve ever learned anything, it’s that being in pursuit of a near impossible goal is one of the most rewarding endeavors that you can ever embark on.”

Watts said it was a surreal experience because he had trained with mutual coaches and developed an affinity for “Rowdy” Ronda Rousey before she was a household name.

“Because she’s such a badass, people think she’s mean or cruel. We all have these many faces, but I noticed she was holding her hand to her heart, and for a martial artist of that stature, that struck me as unusual and clearly kind,” Watts said. “[The video] shows the level of tightness within the martial arts community.”

After pondering the video for a week, Watts responded with a video message of his own.

“It can be a really, really, really cold hard world on the best of all days,” Watts said in the video. “But for me, it was a wonderful day to start on that foot of the very kind and understanding video that you sent me.”

Watts is unsure if he will hear from Rousey again, but in her video the sports superstar said she hoped to meet him one day.

“I don’t know, it could certainly happen the way she phrased it. Particularly if it was the type of thing where everything was aligned and we were in the same place at the same time,” Watts said. “Beyond that I don’t think she’s going to call me at home and start conversing with me or anything, but you never know.”

Watts said he continues to rehabilitate his injury, but is unsure if he will ever recover.

Steve Watts working to rehabilitate his arms. Screenshot from Watts’ Facbook video.

“It’s a struggle, it has a lot of ups and downs. I’m in a wheelchair that I can only operate by using my mouth,” Watts said. “Sometimes you have good days and sometimes you have really bad weeks. Some days you measure your success in different manners. Some days just getting out of bed and doing some exercise is a success. It is not a fast process and healing is never guaranteed.”

The Give Forward page set up to help Watts pay for medical expenses has raised more than $26,000, and friends of the fighter continue to show support through various means.

Watts’ old high school wrestling teammates will be competing in a Tough Mudder race on Oct. 17 to raise money for his cause. Close friend Bryan Raskin also recently raised $2,500 in pledges for running the New York City Marathon.

Watts is also a musician, and is working with friends to host a day long concert in the spring.

“We’re putting a lot of energy into it. I really do need further help and it’s a steep and painful road,” Watts said. “I’m finding that I’m one of the more rare cases that is harder to treat. But that’s right at the heart of everything I’ve done for so long.”

Number 2 of 2My response to Ronda.

Posted by Steve Watts on Wednesday, September 9, 2015

Big Waves Coming! Hurricane Oho to Bash Our Shores With Monster Breakers

Hank Sims / Thursday, Oct. 8 @ 3:02 p.m. / Ocean

The National Weather Service is recommending that you not get pitted-so-pitted for stretches of the coming days.

Hurricane Oho is traveling a path from Hawaii to the Pacific Northwest and NWS Eureka wants to alert ocean lovers that we could experience 13 to 16 foot breakers starting Friday afternoon and into Saturday.

Stay off the jetties, you crazy kids!

Funny Money Alert: Grocery Outlet Hit With Counterfeit C-Notes

Ryan Burns / Thursday, Oct. 8 @ 2:27 p.m. / Crime

Grocery Outlet in Eureka has been handed three fake hundred-dollar bills in recent days, Manager Dennie Linde told the Outpost this afternoon. How did they know the money was counterfeit?

“You can tell by looking at it,” Linde said. It disintegrates in water. Plus, all three bills have the same serial number.

The discount grocer’s bank, Wells Fargo, told Linde they’ve been finding fake 10s and 20s as well.

So beware, local merchants! Check those watermarks.

The Humboldt Bay Power Plant Is Not Exploding

Hank Sims / Thursday, Oct. 8 @ 10:16 a.m. / Non-Emergencies

Not the Humboldt Bay Power Plant in King Salmon this morning.

This morning you’re going to be seeing fire trucks and hazmat vehicles of all sorts freaking out around the Humboldt Bay Power Plant in King Salmon — location of the first commercial nuclear power plant west of the Mississippi (now decommissioned).

This is only a test.

Says Humboldt Bay Fire:

Humboldt Bay Fire along with HMRT,PG&E and City Ambulance will be participating in a HAZ MAT spill drill at the Humboldt Bay Power Plant this morning from 10-12.