CINCINNATI — Six weeks after a power outage sent more than 300 million gallons of raw sewage into the Mill Creek and Ohio River, the Metropolitan Sewer District of Greater Cincinnati says its treatment plant is again operating at full capacity.
But it could take weeks longer to determine what caused a new transformer to fail one month after installation, shutting down power to the entire plant.
The transformer’s manufacturer and the Illinois-based engineering firm that installed it have each retained third-party consultants to investigate the cause, while MSD is looking for its own consultant to check their work.
“MSD has committed to engaging a third-party consultant to complete a lessons-learned analysis on the Mill Creek Wastewater Treatment Plant (WWTP) Substation Upgrade Project, including a review of risk management and contingency planning related to loss of power during the project,” MSD spokeswoman Deb Leonard wrote in one of 20 responses to I-Team questions. “The results of the evaluation will be made public when completed.”
In a March 17 letter to Ohio EPA, MSD called the March 5 outage a “catastrophic failure” that was “outside of its control.”
In other public statements, it disclosed the transformer failed — then passed — a factory stress test. It operated fine for about a month until MSD disconnected its only backup so the 50-year-old unit could be replaced.
"Site space constraints required that there would be a period without a redundant transformer," Leonard said. "The second new transformer is on hold at the manufacturer, pending the results of the investigation into why the first transformer failed."
'Why no backup?'
Those answers only bring more questions to mind for Marilyn Wall, a Sierra Club volunteer whose 2002 lawsuit led to a $3.1 billion consent decree aimed at ending sewage pollution by MSD.
“It just raises questions about the whole project,” Wall said. “The age of equipment at MSD, was that the problem? Was it the transformer itself? You know, after all this time, after failing at the factory and then testing OK and then running for a while then failing catastrophically. And just the whole process of installation like, ‘Why no backup?’”
Wall is also concerned about contract documents that show the project doubled in price and was more than five months behind schedule when the transformer failed.
“It’s certainly very concerning because actually it was 2014 when MSD’s report said there were enough problems with the transformers they needed to replace them,” Wall said. “And so, it’s been a long delay to just getting it started.”
Reviewing the contracts
The I-Team obtained contract documents from MSD and submitted written questions to trace the project’s history. This is what we learned:
- MSD hired a Kansas City consulting firm to prepare bidding documents and help it manage the project because MSD lacked “sufficient staff of permanent employees to do so,” according to a 2016 contract with Black & Veatch Corp. That contract required the city to pay up to $565,716.
- A 2018 request for proposals led to the hiring of Illinois-based Patrick Engineering Inc., which signed a design-build contract worth up to $6.7 million in the spring of 2019. That price was intended to cover the design phase of the project, with a construction budget and “guaranteed maximum price” to be established later.
- The project’s total budget swelled to $18.5 million by December 2020, when Hamilton County Commissioners approved a funding request that itemized $8 million in increased costs since Patrick Engineering delivered its initial estimate in 2018. MSD said it has paid bills totaling $12.9 million on the project so far.
- The original completion date for the project was January 2020, according to the RFP. However, that date was modified twice in contract documents, including a December 2020 amendment that called for final completion by Sept. 14, 2022.
- MSD’s contract allows it to pursue liquidated damages of up to $3,000 for every day the project is past its scheduled completion date. But MSD has not pursued liquidated damages against Patrick Engineering. “MSD is evaluating the obligations of its contractors and will take appropriate action as needed,” Leonard said.
What do the contractors say?
Patrick Engineering has not responded to the I-Team’s request for information. RINA Consulting, an Italian company that bought the Chicago firm in January, referred all questions to MSD.
Patrick's 2019 contract identified 14 project leaders in an organizational chart. Three of the top six left to form their own company in October 2021. Eight of the 14 were gone by February 2022, according to their LinkedIn bios.
Did the departures contribute to the transformer’s failure? MSD has not answered the question. But Phil Thaman doesn’t think so.
“Everybody did what they were supposed to be doing,” said Thaman, president of Glenwood Electric, the project’s biggest subcontractor. “There was one project manager that left. He was replaced and I don’t see any issue with that. It’s not unusual.”
Thaman’s company installed the wiring, circuit breakers and switches that connected the new transformer to power lines carrying 138,000 volts of electricity and feeder lines that brought between 4,000 and 13,000 volts to the plant. He is confident those connections were sound.
“Everything’s tested,” Thaman said. “Everything’s commissioned and checked out beforehand.”
His best guess is that the new transformer — which also passed on-site tests — caused the outage.
“I’d say there was probably a failure internally,” Thaman said. “It happens with electrical equipment.”
The FAT test
The transformer was manufactured by Delta Star Inc., which modified the unit after it failed its Factory Assessment Testing in April 2022, according to a report prepared for Patrick by Stantec Consulting Services.
"It was agreed that the BIL rating for the high-voltage winding would be reduced from 550 kV BIL from the original 650 kV level," said the November 2022 report. "The high-voltage windings were completely replaced with new conductors."
'BIL rating' refers to the maximum voltage that electrical equipment can withstand without damage.
After those changes were made, the transformer passed a second factory test in August. But neither MSD nor Delta Star answered questions about the test. So, it's not clear whether its lower BIL rating was a factor in the March failure.
“At this time, we are collecting event data from MSD and the local electric utility and have engaged third-party consultants to assist in the evaluation of what caused the failure," Delta Star spokesman Asa Keimig wrote in an email response to the I-Team’s questions. "When Delta Star ascertain the root cause of the failure, a full report will be supplied to MSD and their consulting team.”
Power anomolies
Contract documents point to another potential cause.
In May 2020, Patrick Engineering sought a change order to install a different kind of air break switch than MSD specified. Air break switches are used to isolate problems in high-voltage power lines. Patrick Engineering received a $15,000 change order to install motor-operated air break switches after providing this explanation:
“After reviewing and discussing the 138kV Automatic Throw-Over (ATO) scheme proposed by Duke, it was determined Duke would not provide an immediate means to protect the proposed transformers in the event of a breaker failure event.”
MSD said it isn’t aware of any problems with Duke Energy’s power feed to the site. But Duke Energy and Delta Star both mentioned power anomalies in statements about the failure.
“Delta Star is aware of the power surge and subsequent power outage that occurred at the Mill Creek Wastewater Treatment Plant," its spokeswoman wrote.
Duke Energy played no role in the transformer’s failure and “did not decline to provide any protection” to MSD’s equipment, spokeswoman Sally Thelen said via email.
“As the sequence of events unfolded, at one point a “fault” (short-circuit) was initiated that was detected and cleared by Duke Energy equipment at the Duke Energy substations to the north and south,” Thelen added. “The Duke Energy equipment acted as designed and intended to isolate the faulted equipment. The Duke Energy system was not in any danger and no other customers on the Duke Energy system were interrupted.”
What's next?
Six weeks after the treatment plant’s meltdown, MSD is not sure when it will replace the 50-year-old transformer that has operated without a backup since mid-March.
Mill Creek is expected to recover from the deluge of bodily waste that oozed out of combined sewer overflows over 11 days in March.
“The creek has shown itself to be amazingly resilient,” said Dave Schmitt, executive director of Mill Creek Alliance. “It was once essentially lifeless through the lower two-thirds of the watershed … Now there are bald eagles and ospreys and fresh-water mussels and mink, all kinds of critters returning to the stream. So, I think it will bounce back, given a little time and a little help.”
Schmitt provided some of that help in early March by convincing the Army Corps of Engineers to release millions of gallons from the Winton Woods reservoir, flushing out the concentrated sewage that had settled on the creek floor. Volunteers, who found “sky high” levels of E. coli contamination in early March, documented a reduction on March 14 but they were still too high to allow recreational use.
“Undoubtedly, some material, some raw sewage, organic matter has settled to the bottom of the creek,” Schmitt said. “We won’t know until later this year, when we can get out and do some sampling for macroinvertebrates and fish, the critters that live on the bottom of the creek, at the base of the food chain. When we can do some sampling, we’ll know for sure” if the creek will suffer long-term damage.
But that assumes MSD’s only transformer will remain functional long enough for two new replacement transformers to arrive.
“My understanding is either of those big transformers can handle the whole load for the plant and it was working prior to the start of this replacement process,” Schmitt said. “So, you know, knock on wood. One functioning transformer should be OK. But, obviously, there is a risk if something should occur.”
