Infrastructure disaster caused by defective float switch. Image by: EMILY M. ENG AND THOMAS WILBURN / THE SEATTLE TIMES

Plant Managers: Infrastructure disaster caused by a damaged $200 float switch. Here’s 7 tips to avoid it.

Posted by

Don’t let a $200 part cause $57 million in infrastructure damages. It happened in Seattle and it can happen to you.  

“A disaster years in the making”  

“Seattle Plant Failure Dumps Millions of Gallons of Sewage”

“Float switches may have failed at flooded West Point Wastewater Plant”

And the worst one of all:  

“Human Error Blamed in Review of Seattle Wastewater Catastrophe”.

These are all headlines that screamed across the news following the February 9 plant infrastructure failure at the West Point Treatment Center in Seattle.

It could have been avoided

No plant manager wants this kind of notoriety. You don’t want the damage to your plant or the environment and you don’t want the bad publicity that goes along with it. The really bad news is that it was avoidable. This disaster could have been prevented had the staff followed recommended maintenance protocol and replaced a $200 broken float switch. But they didn’t.

Power failure hits and all h*ll breaks loose

In the early morning hours of February 9, 2017 a power failure hit the West Point Wastewater Treatment Center in Seattle, WA  in the middle of a heavy storm. The plant was running at full capacity that evening when the pumps were knocked out.

Within 30 minutes the plant was flooded with millions of gallons of wastewater. The emergency bypass gate, designed to protect the plant, failed. The basement, where much of the equipment is positioned, was completely flooded. In the end, 235 million gallons of untreated wastewater, including 30 million gallons of raw sewage and hundreds of tons of partially treated solids, were dumped into Puget Sound.

Results from engineering firm AECOM announced

An independent investigation done by engineering firm AECOM concluded that the bypass gate failed to open due to the failure of all eight float switches. AECOM determined that the float switches had been damaged due to incorrect cleaning and then were never mechanically inspected, allowing the damage to go undetected.

These float switches had been a problem at West Point for years prior. The switches had failed in both 2000 and 2008. A consultant recommended “mechanical testing” of the devices in 2004, but the recommendation was not adopted.

The damage estimates began at $25,000,000 and ballooned to $57,000,000. This damage originated with a failed $200 float switch that staff employees had already recommended changing.

How do you avoid these problems and protect yourself and the infrastructure of your plant?

The following items were recommended for West Point in the AECOM report but can be applied to your plant as well.

#1: Do a complete systematic evaluation of your plant.

This approach needs to commence with a thorough systematic evaluation of plant systems in terms of function, performance and safety. This approach will improve vertical and horizontal communications regarding plant risks, provide better documentation of decisions, and result in stricter levels of maintenance.

#2: Avoid the normalization of deviation (Vaughan 1997).  

This happens when personnel become accustomed to the poor performance of a piece of equipment. Personnel need training to understand that poor equipment performance should be reported and resolved before, and not after, failure has occurred. 

#3: Recognize the Incident Pyramid. 

Most serious incidents are preceded by a series of less-serious incidents. Recognizing minor incidents early and taking appropriate actions will help to prevent major system failures.

#4: Institute a Life Safety Management system (LSM).

Similar to a Process Safety Management Plan currently in use where regulatory requirements demand it. Under Life Safety Management, operations and maintenance procedures are more strictly applied, and operators, maintenance works, engineers, safety managers, asset managers, administrators and manager and policy makers work together to achieve the best possible outcomes.

Keeping the PSM and the LSM systems separate will simplify federal audits, leverage the best parts of PSM across the plant and greatly raise the bar regarding how the utility conducts the business of treating wastewater at WPTP. It will also reduce the serious risk to life safety, risk of release of untreated sewage into the sound and financial risks to the utility.

#5: Conduct Comprehensive Emergency Response Training, 

AECOM recommends that WTD formulate and adopt a formal emergency training and emergency exercise program compliant with the Homeland Security Exercise and Evaluation Program. Training can help employees maintain competency for completing their specific tasks during an emergency, as the roles they assume in an emergency could differ from their normal duties. Training is particularly critical for tasks that employees do not perform regularly, such as those related to emergency bypass procedures.

#6: Conduct an integrated evaluation to address plant constraints and improve redundancy.

It is important to address the lack of passive systems which would allow the plant to handle flows through the facility in the event of loss of automation, lack of power or delayed decision-making. One of the most critical points of failure at WPTP is the emergency bypass gate. If the gate fails to open, the facility can be flooded, posing a serious life-safety risk. It is recommended that passive overflows be evaluated.

#7: Optimize a capital improvement plan to maximize redundancy–protect your infrastructure.

The longer term planning processes should consider capital improvements that have immediate and significant impacts on plant capacity. Some improvements that can have immediate impacts on operations during a peak flow involve configuring the Ovation system to prioritize alarms and integrating pump stations and emergency bypass gates with plant controls.

Read the full report from AECOM for the detailed explanation of points above.

The nature of an emergency is somewhat unpredictable, but paying close attention to minor incidents in advance can give you a head start on avoiding a catastrophe in the future. You can protect your plants infrastructure, the environment, and the lives of your employees.

In short, just don’t get complacent. Be vigilant and demand the same from your staff.

By: Jennifer Croft

An excellent in-depth article from the Seattle Times on the West Point disaster

Article sighting $57 Million in damages

AECON’s report to King County

For information on sister-facility Brightwater Waste Treatment Center, Woodinville, WA

Float switch illustration by EMILY M. ENG AND THOMAS WILBURN / THE SEATTLE TIMES