Fix the pumps

That's all folks

2007-07-03T14:12:00.000-07:00

As some of you may know, I'm trying to step away from all of this. It's taken a huge toll on me, as if Katrina weren't damaging enough.

So this is my last post. For a while, I'd considered this a closed loop process, where a defined end would be reached at some point.

However, I've concluded "Katrina" will never end, and at this point the only way to find my end is to declare it.

I had thought about continuing for a while, despite some major transitions in my life, but I just don't think it's worth it. I've got to move on. I've put out as much information as I can get, and if people don't understand the issues at hand by now, they'll never understand.

Thanks to all those who have supported me. I wish the best for New Orleans.

What MVN thinks of the internal

2007-07-03T13:11:00.001-07:00

There were ten technical recommendations in the Corps' internal investigation into the floodgate pumps. Since General Crear ignored almost all of them in his ~~press release~~ cover letter, it fell to the local media to find out their disposition.

Courtesy of WWL-TV, below you will find the New Orleans District's responses to the internal investigation's technical recommendations. I can't verify the truth of anything but part of the first one. The hydraulic reservoirs are being raised (see photos below).

The idea behind raising the hydraulic reservoirs is to pull hydraulic fluid from lower in the reservoirs, preventing entrained air from entering the delicate Denison hydraulic pumps and wrecking them. MWI's original scheme (the one which has been in existence up until last month) was to pull hydraulic fluid off the top of the reservoirs, working against gravity and guaranteeing air would get trapped in the inlet lines to the Denison pumps. This wrecked Denison pump after Denison pump. Considering that the scheme has been in operation up until last month, it is a strong possibility there are now wrecked Denison pumps on the skids, a conclusion reached in the internal investigation (see my commentary on internal recommendation #1 below).

The new scheme raises the reservoirs and reworks the piping and hoses between the reservoir and the Denison pumps, with the hydraulic fluid coming out of a point lower on the reservoir (hopefully the very bottom, but that's not clear).

They've started raising the reservoirs with Orleans Avenue for some unknown reason (I would have thought 17th St and London would be far more urgent, since those floodgates gates are intended to be dropped long before Orleans). All of the Orleans Avenue reservoirs appear to be raised.

17th Street does not have raised reservoirs on any of the original 12 pumps, as of June 20 (the six newer pumps came with raised reservoirs last year). Considering how much importance the Corps puts on 17th Street - every major press availability is held there, and it's the only site the Corps took the GAO when they visited, not that it mattered to the GAO - you'd think they would have started there.

So how does one tell if the reservoirs have been raised? It's fairly simple. First, let's get oriented to where the hydraulic reservoirs can be found on the drive skids. We'll use a picture from Dennis Strecker's April 18, 2006 report, which is included in the internal investigation (Adobe page 136):

The "day tank" is another name for the fuel tank. One can see the hydraulic reservoir is located inboard - or behind - the fuel tank. One can also see that the hydraulic reservoir is shorter than the fuel tank. This makes for a handy reference to determine whether the reservoirs have been raised. If the top of the reservoir is above the fuel tank, they've gone up. If the fuel tank is higher than the reservoir, they haven't been raised.

Here's a "before" picture of a drive skid at the east drive shed at Orleans Avenue:

As you can see, the top of the fuel tank was above the top of the reservoir. Thus, the reservoir was at its original, as-delivered height on March 25, 2007.

Now look at an "after" shot of the same skid:

The top of the reservoir is above the top of the fuel tank. The reservoir is now sitting up on a new set of approximately foot-high legs. This work was done in late May or early June.

All five drive skids on the east side of Orleans Avenue had gotten this treatment as of June 20th:

I don't have a picture of it, but I did see that the five skids on the west side of Orleans also had their reservoirs raised as of June 20, 2007.

Here's a picture of the actual legs being used to raise the reservoirs, along with hoses that have been removed from skids:

Moving over to London Avenue...

Here's the three southernmost skids in the west drive shed at London Avenue, none of which had their reservoirs raised at the time of this photo:

Here's the other three skids in that shed (sorry about cutting off the skid on the right):

How do we know the reservoirs are unraised in this shed? By examining the relative heights of the fuel tanks and the reservoirs. Here's a detail from the first London Avenue shot above, which shows the reservoir unraised:

All of the rest of the London Avenue - west drive skids were exactly the same as of June 20, 2007.

I don't have pictures of the London Avenue - east drive skids, but I did see they were also unraised as of June 20, 2007.

Finally, let's do the same treatment on 17th Street. First, the east drive platform:

Below is a detail shot of the drive skid on the left. While it's a little hard to make out (it's actually easier to see in the long shot above), you can see that the top of the fuel tank remains above the top of the reservoir:

Here's the best picture I could get of the 17th Street west drive shed:

And here's a detail, showing that the reservoirs hadn't been raised in that shed as of June 20, 2007 (I'm using the "daylight" argument):

My camera doesn't pick up as much detail as the naked eye, so you'll have to trust that I could clearly see the reservoirs in that shed were still at their original heights that day.

The internal recommendations follow, along with the New Orleans' District's responses. I've included a little commentary inside brackets. "MVN" is the Corps code for the New Orleans District.

"1) Flooded suction for all hydraulic oil pumps to prevent equipment failures and unsafe priming by operating personnel. Simultaneously with this action is the removal of the Denison Hydraulic pumps to inspect for any failed or in process of failing components. Any unsuitable components shall be replaced."

MVN responds: In process.

[It's important to note the second sentence of this recommendation. The internal team believes there could be "failed or in process of failing" components inside the Denison hydraulic pumps - even today. This is pretty huge, since the Denison pumps are the heart of the drive skids, and their failure leads to the failure of a water pump. There are 80 Denison pumps installed at the three canals, along with another six installed on spare drive skids. That's 86 units that are still in jeopardy of going kablooey.

Such Denison pump failures happened all the time during the MWI factory testing, and they continued during the initial field testing in the spring of 2006 (note: not complete field performance testing - which the Corps has no intention of conducting - just turning the pumps on to see if they worked).

We know the Denison hydraulic pump failures continued during the field testing because the internal investigation folks included data on field tests in Attachment #10 to the internal report. It's only two pages, but's it's sure disturbing. Here's the results for the tests of just two drive unit (DU)/pump assembly (PA) combos at Orleans Avenue in May and June of 2006:

DU 8845/PA 4590 (test date: 5/31/06)
Total run time: 25 minutes
Component failure: Filter side Denison hydraulic pump

DU 8842/PA 4581 (test date: 5/24/06)
Total run time: none - static pressure test only
Component failure: Filter side Denison hydraulic pump

DU 8842/PA 4581 (test date: 6/1/06)
Total run time: none - 20.2 minutes
Component failure: Control panel side Denison hydraulic pump

What's interesting about DU 8842 is that it was the skid that was used for the only 24 hour "endurance" test performed during the MWI factory testing in the third week of April, 2006. Except that test had the skid running at 1000 psi, less than a third of the pressure the drive was supposed to generate in the field. But after 24 hours of running on cruise control (and a brief repair to the gear oil circulation pump), 8842 was good enough to send out to Orleans Avenue, where we can see it promptly broke - twice - after less than an hour of runtime.

None of the information about the poor performance of the pumps in the field after they were installed was published in the media at the time it happened. Articles from May, June, July, and August of 2006 show no acknowledgement by the Corps of any problems on the MWI pumps, until the vibration problems emerged in mid-August. Those vibration problems are also documented in those Orleans Avenue tests three months previously (more withholding of information).

The same withholding by the Corps goes for public meetings attended by in-the-know Corps New Orleans officials such as Jim St. Germain, Colonel Bedey, Colonel Wagenaar, and Walter Baumy, some of which I have on tape. No one in the public - and I mean NO ONE - was clued in to the fact that drive skids were going kablooey and pump assemblies were vibrating practically on a daily basis throughout the summer of 2006. That's a horrible way to treat the people of New Orleans, but it's a great way to treat a manufacturer that doesn't want bad press.

Now, a year later the Corps is doing the exact same thing, because it's the exact same people - St. Germain, Bedey, Bradley. They run the pumps and drive units for 45 minutes, assumedly at 3000 psi (though we have no idea, and the freaking media won't ask), get worthless flow numbers from flowmeters that exemplify GIGO, and then they say, "Everything's fine!" Everything is not fine. Qualified, non- New Orleans individuals in the Corps believe there are still vital pieces of equipment primed for failure on these skids today. That's serious.

We're talking about protecting peoples' lives here. It's not a game. This is why comprehensive, extensive testing is required of this equipment, not the crappy crud with which MWI and the New Orleans District bamboozles the local media. The contract says so, reality says so, the public says so, and common decency says so. The only people that say it isn't are the New Orleans District and MWI.]

"2) Recommend that a certified hydraulic systems inspector, per ASME B31.1 Power Piping, inspect the piping system and certify that the hydraulic piping system is safe to operate for the intended use. The inspector may add operating requirements due to the reduced factor of safety. Any additional operating requirements must be included in the training of, and provided to any pump system operators."

MVN responds: They analyzed...said not true.

[This one cracks me up, and kind of makes me cry a bit. The Corps is, I believe, relying on a static test of the piping at 4500 psi to make this statement, as well as some really crummy calcs included in the internal report (found on the last two pages).

Such a static test is inadequate to determine if the pipes are suitable for dynamic conditions lasting hours on end. Considering how much the piping vibrates when the pumps are run, and how rusty the pipes are now, and how undersized they are according to code, it's clear the New Orleans District and its Metairie-based consultants at NY Associates, Linfield Hunter Junius, and URS likely have no idea what they're talking about when it comes to the pipes failing. They're playing with peoples' lives for God's sake. Are they really willing to leave clearly deteriorating equipment in the field for use during a hurricane?]

"3) Evaluate to determine if the automatic clutch system should be required to be installed to reduce shockwaves within the hydraulic piping system as required per contract documents to provide a smooth, soft start."

MVN responds: At first said don't think need to be done....then said in process.

[If they think the piping is okay, then why are they looking at the clutch to prevent damage to the pipes?]

"4) Information on welds that have been previously repaired by the Government and other welds which were not repaired should be reviewed. Further review is needed to determine whether additional welds should be repaired."

MVN responds: Done.

[We will likely never see information about this.]

"5) Provide mechanical computations for flange bolts to insure they are sized properly to handle the weight of the components."

MVN responds: Done.

[We will likely never see these calculations.]

"6) Remove inlet strainers per Denison recommendations."

MVN responds: Evaluating.

[How can they still be "evaluating" this one? All the information was laid out over a year ago on this issue, as shown in the internal report and the shop inspection records. Denison themselves says not to use strainers on the inlets to their pumps. What is there to evaluate?]

"7) Installation of the hydraulic cones in London and Orleans, along with vortex suppressors as determined by the Model Studies conducted at ERDC. The cones are currently being installed at London Avenue Canal."

MVN responds: Done.

"8) Each pump should be provided with a hydraulic system monitoring device (as required by the contract) to allow diagnosing hydraulic system behavior even while the pump is submerged. If a monitoring device is not provided, then a credit to the Government is due."

MVN responds: Evaluating.

[After all the hydraulic system problems, you'd think this one would be a no-brainer. You'd think...]

"9) Calibrate and locate the ultra-sonic equipment in the proper locations as required by the manufacturer with direction as required by Dr. Maynord at ERDC. This is to determine the correct amount of flow capacity for the pump at all three Canal sites."

MVN responds: In process.

[As I mentioned a few posts back, there's very little reason to believe the flowmeters will ever work. In ERDC's own April 2007 tests at London Avenue, they weren't used to find the capacity of the pumps. Instead, ERDC "backed into" the capacity by getting the static head in the pumps and used a derived pump curve (with its own significant scatter errors) to interpolate to the capacity. From an engineering standpoint, this is far less reliable than a direct measurement of the flow. It also subverts the entire floodgate pump control system, which is based on the flowmeters, not some jury-rigged piezometer-and-pump-curve smashup.

This would appear to be another source of a tremendous amount of wasted taxpayer funds - probably millions of dollars. Here we are over a year after the installation of these control systems, and at their most fundamental level, they don't work. What a mess.

This one lands smack at Dan Bradley's feet. He's the electrical engineer on this job, and was responsible for the controls system.]

"10) Have MWI provide manufacturer’s information and shop drawings (including Durst information) as discussed earlier in this report."

MVN responds: Done.

[I have a hard time believing this is "done," since the ERDC folks didn't have MWI's certified information on which to rely just over two months ago.]

Cover up, Part 3: GAO SUX

2007-07-02T13:32:00.000-07:00

Updated 10/8/07. See below

In Parts 1 and 2 of this report, we looked at the lies, extremes in poor judgement, and skirting of the law in which the Corps New Orleans District regularly trades. Today, we'll expand our focus to what passes for oversight over the Corps, and why that can't even be trusted.

When I heard this past March that, at Louisiana Senator Mary Landrieu's request, the Government Accountability Office (GAO) was investigating the floodgate pumps mess, I was hopeful but skeptical. GAO's previous work on Katrina issues has been spotty at best. It seems, based on comparisons to what the Corps internal investigation turned up, that my skepticism was justified.

What GAO gave to Senator Landrieu's office was a steaming turd. It doesn't even track what the Senator asked for in her letter to the GAO. And now that we have the Corps internal investigation in hand, we can see the blatant mistakes by GAO clearly.

The GAO report doesn't mention the numerous contract administration improprieties (including - but not confined to - troubling use of implied authorities by individuals like Jim St. Germain and Dan Bradley who did not possess such authorities), it only glancingly mentions the critical technical difficulties with the hydraulic systems (accepting the inadequate interim fix on the reservoirs as adequate, instead of bringing up the fact that the permanent fix was critical and had not been implemented for nearly a year, and still hasn't been implemented on most of the drive units), and I have to wonder if they even read Ms. Garzino's correspondence and the shop inspection reports. In fact, Ms. Garzino's name inexplicably does not appear anywhere in a report that would not exist without her memo.

In addition, there's zero mention of the continuing non-execution of the performance testing section of the contract, which is at the heart of the cover-up (see Part 1 and Part 2 of this report). In fact, there's only parrotting of the Corps' line: "Any performance problems with the pump systems could be resolved after they were installed on the canals."

Except (and this is only 9 pages later in the GAO report), "According to the Corps, it is not possible to test each pump system for an extended period of time or to test the entire system due to limitations in simulating the amount of water that would be present in the canals and the lake during an actual storm event."

So the performance problems could be resolved later, presumably through contractually-mandated full performance testing. Except that "later," there's no way to actually perform those tests? Huh?

How in the world did the GAO not catch and highlight this clear contradiction in the Corps' statements, especially since it appears the GAO's own report? Why didn't they catch it and perhaps think about whether the Corps ever intended to do that performance testing, or if the Corps has never intended to do it (see the earlier parts of this report)? And then why wouldn't they think about possible motivations for such inaction, and really dig in? Why couldn't they do that bit of thinking?

This is what I mean when I say the GAO investigation is junk. They appear to have taken whatever the Corps New Orleans District handed them and did no critical reasoning on it. Hey GAO, here's critical reasoning on the Corps statement about the impossibility of extended testing of the pumps on site: What about taking the pumps out of their platforms and running them elsewhere, where they could be lowered and raised - thus eliminating the "limitations in simulating the amount of water?" I'm not the first person to think of it. That's exactly what the Corps themselves did last August when the vibration problems emerged! They hooked pumps to a crane and dunked them in the canal at different depths. Surely everyone remembers that. Here's an August 6, 2006 picture of that widely publicized testing, taken from the Corps' own press release (dated December 1, 2006) (it also flashes by in the multi-image GIF on the Corps New Orleans Hurricane Protection Office front page, which is where I got the date):

The caption cracks me up. It makes it seem like this was routine testing, not everyone at the Corps and MWI grasping at straws.

You can see the drive unit for the pump on the deck at the lower left of the photo behind the mist cloud. Now, if this was possible last August, why couldn't a barge be rigged up with a crane (to raise and lower the pump), some kind of framework (to hold the pump), and a set of appropriately long discharge piping in which to measure flow? And why couldn't such a system be run for a legitimately long amount of time to measure the entire range of flows and heads, as well as a long enough time to simulate storm conditions?

The answer is: of course it could, but I'm sure MWI would refuse to do it, even though they are contractually obligated to do so. Because it costs too much. And the Corps New Orleans District refuses to do it because it would show how poor their contract administration has been, i.e. they've paid millions of taxpayer dollars for equipment which does not meet spec. (Side question: I wonder if MWI paid for all the crane rental time, labor, and materials for the testing shown in the photo above?)

And that's how you do critical reasoning.

There's even flat-out lies in the GAO report that should have been obvious. On Adobe page 35, there's a chart of Orleans Avenue canal capacity dating back to June of 2006:

It shows a constant capacity for Orleans at just under 2000 cfs from June, 2006 until mid-May of this year, when there's a small, unexplained bump up to just over 2000 cfs. I'd like to know how the Corps could have gotten 2000 cfs out of 10 pumps at Orleans when five of them were sitting out of the water up on the deck from October, 2006 through March, 2007, 2007 (read the linked blog entries for more details and pictures):

The year on the picture above should have read "2007." Oops.

Yes, I understand that the period the pumps were out at Orleans was outside the height of hurricane season. But the pumps weren't even in the damn water for over six months! When basic, easily checkable facts like that are screwed up, why should the rest of the GAO investigation carry any credibility?

The whole GAO thing seems less like an investigation and more like a mistake-ridden, propaganda-laced summary of current conditions. GAO themselves needs to be investigated by their inspector general to determine how Senator Landrieu could write the following as their marching orders...

"I request that the Government Accountability Office immediately undertake an investigation of the Army Corps of Engineers rationale for the the installation of faulty flood control pumps despite the internal memoranda indicating their insufficiency for the task. Furthermore, the investigation should examine the contract arrangements with the company in question and should determine whether or not any improprieties exist in the award or fulfillment of these contracts."

...and then the GAO could interpret/twist that into the following "questions to be answered:"

"- What were the specifications and requirements of the contract and what was the basis for the selection of the supplier for the pump systems?
- In light of the factory test failures known to the Corps, why did it install the pump systems?
- What pumping capacity existed on June 1, 2006, and what actions has the Corps taken to address the known problems with the pump systems?
- What is the current pumping capacity at the drainage canals and what are the Corps’ plans to increase capacity during the 2007 hurricane season?"

They're kinda close, but GAO's missing a large chunk of what the Senator asked for. Where are the GAO's questions about the contract arrangements and improprieties in the fulfillment of the contract? "Fulfillment" entails looking at the entire span of the contract (which means up to the day GAO issued their report), not the "basis for selection of the supplier." That is a tremendous narrowing of the scope of the Senator's request, and GAO needs to be taken to task for it.

Why is there zero mention of the substantial improprieties on the financial and contract administration side of the project? The Corps internal investigation uncovered suspiciously inadequate record-keeping, an uncooperative Contracting Officer, and millions of dollars in payments to MWI that couldn't (or wouldn't) be accounted for. Isn't that precisely what the General Accountability Office should have also found? After all, they've got the word "account" right there in their name.

Even taking the GAO's questions on their own terms, how can they ask their last question about current pumping capacity without mentioning the persistent, documented capacity shortfalls revealed by Corps-overseen testing in April 2006, November 2006, and once again in April 2007? The November 2006 and April 2007 testing was performed at the behest of the Corps internal investigative team. GAO themselves say they met with "members of the Mississippi Valley Division’s independent team reviewing issues related to the contract and pump performance." If they listened at all to that team, how could GAO not reach the same conclusions (on all matters, not just capacities) the internal team was reaching?

You can probably see that I think very little of the GAO investigation. I can say now that while they were investigating in March, April and May, they were reading this blog every day, sometimes for hours at a stretch. That gave me limited hopes, because one of the reasons I started blogging was to make documents easily available to the eventual investigations I knew would come. Even today, someone at the GAO checks in regularly. How they got from what I've reported to their stuff is a mystery. And when you consider the everlasting, gob-stopping treasure trove of the Corps internal investigation (which is light years beyond any of my little rhetorical bottle rockets), the GAO report is measurably inferior.

There's only three possibilities for the radical differences between the GAO's report and the Corps internal investigation:

1) GAO didn't get access to the same data the Corps internal team did. This is very, very unlikely, despite what they told Senator Landrieu's office. There's no way GAO could have written a report without looking at the contract, the modifications, and the correspondence. Also, did they not request the same documents the Corps internal team was using?

2) The Corps and GAO had access to the same data, but GAO actively ignored it for some unknown reason. This is the scariest possibility, because is speaks to a compromised investigatory process.

3) The Corps and GAO looked at the same data, but GAO didn't realize its significance. In other words, the GAO team was incompetent.

In any case, whatever the reason for the inadequacy of the GAO report, the consequence is clear. It has muddied the waters into what happened, extending the cover-up. Indeed, after its release, both MWI and Colonel Bedey crowed about how they were in the clear.

MWI (in the May 21st Palm Beach Post letters to the editor):

"...The Associated Press further reported on Thursday that Sen. Mary Landrieu, D-La., said an investigation and forthtcoming [sic] report by the Government Accountability Office found no evidence of fraud or improper influence in the pump project."

Colonel Bedey (in the May 17th AP article about the briefing GAO gave Senator Landrieu's office):

"'We always said there was no smoking gun,' Col. Jeffrey Bedey, who is overseeing reconstruction of the city's levee system for the corps, said in reaction to the report."

As we now know from the internal investigation, there's an armory of smoking guns. GAO either ignored them or was told to ignore them.

In some ways, I wish the GAO had just kept their own counsel. Fortunately, Senator Landrieu is the chair of the Senate Appropriations Subcommittee that controls GAO's budget, so GAO had better do a much more thorough job when they dig back into this fiasco. And, as I mentioned above, I hope the GAO Inspector General takes a close look at this one too. It's certainly destroyed my faith in yet another agency of the government, which I didn't think was possible after all that Katrina has wrought.

Update, 10/8/07:
After the internal came out, and it became readily apparent that the GAO investigation was inadequate, Senator Landrieu said she strongly encouraged GAO to revisit their findings:

"...the GAO must keep digging to ensure that Congress and the people of New Orleans have the truth"

That press release is here. That was on June 8th.

It took a while, but the GAO did restart their inquiry. Oddly, it was MWI themselves that first revealed this publicly. On September 12, 2007, they responded to an article posted on a website called CorpWatch around Katrina's 2nd anniversary. The original article summarized the pump story to date. MWI's response attempted to rebut it, and accidentally revealed:

"The GAO is now in the process of pursuing an additional inquiry and we welcome
the opportunity to meet with them again."

I seriously doubt they "welcomed" the opportunity to be investigated, but that's standard corporate-speak.

This was confirmed on October 8th, along with some other huge news:

Times-Picayune, Monday, October 8:
Third probe of pumps on canals requested

Associated Press, Monday, October 8:
Report: New Orleans pump contract likely handled improperly

Ms. Garzino went to the Office of Special Counsel, an independent federal investigatory agency. Her referral raised enough eyebrows at OSC that they ordered Secretary of Defense Robert Gates to further investigate the entire imbroglio. Going through OSC provides an extra degree of protection to ensure the integrity of the investigation.

The existence of the Defense Department investigation was revealed by an illegal leak of its existence from within the Pentagon to a whistleblower group called PEER. Their source has likely lost his or her job, because PEER did not bother to submit a FOIA request for the documents they leaked to the Times-Picayune. Had they done so, their source would have been protected, and the DoD investigation would have likely gone a lot smoother.

Be that as it may, the documents they gave to the Times-Picayune are attached to their October 8, 2007 press release here. The direct link to the documents is here.

Cover up, Part 2

2007-06-28T23:13:00.000-07:00

In Part 1 of this report, we detailed Colonel Bedey's lie on national television about full performance testing of the pumps and drives. Bedey claimed the Corps had "decided" in the spring of 2006 to move that testing from the Florida factory of MWI to the worksites in New Orleans.

We demonstrated the transparent falsity of that lie by looking at actual contract documents, emails, and analysis within the internal Corps floodgate pumps report. All of those sources show that Corps had no intention of conducting full performance testing at the worksites. They also show that there has not been any full performance testing at the sites for over a year, a clear violation of the contract. Colonel Bedey made up out of whole cloth the fiction that full performance testing would be conducted in New Orleans. The lack of such testing is a serious shortfall in the performance of the contract.

In this part, we'll look at the implications of the lack of full performance testing, as well as the Corps' actions in trying to avoid doing such testing.

What does the lack of full performance testing mean?

For one thing, it completely undercuts all of the Corps New Orleans District's reassurances about how great the pumps are working. The fact is they have no way to determine whether all of the pumps are working up to spec, because through their inaction they have illegally de facto eliminated that part of the contract.

Remember what the internal investigation says about full performance testing of all the pumps:

"This testing was not performed in accordance with the contract requirements and a significant credit is due to the Government for the nonperformance of it"

This is in the present tense, i.e. the full performance testing was not performed as of June, 2007. Perhaps that's why Bedey said earlier in the interview (see my June 13th post) that the internal report was a snapshot of September, 2006 conditions: he was clumsily trying to undercut the fact that for over a year, he and the people under his command and their supplier have likely colluded to avoid enforcement of a vital aspect of the pump supply contract. Enforcement of that provision - and its probable results that the pumps don't meet the spec (see below for backup on that) - would likely land the Corps and MWI in court, and due to the New Orleans District's cruddy recordkeeping, MWI might win.

But what about all the testing the Corps says they've run? They say the pumps are ready

All those dog and pony shows the Corps has run for the local media, which the media then turns into b-roll footage for the nightly news... none of them were actual performance tests over the entire range of the pumps' required flows and heads. Bedey admitted in the press briefing when the internal report was released that none of the latest round of "tests" went longer than 45 minutes. There's no way one could run a satisfactory flow test in such a brief time.

The Corps' own press release from May 31, 2007 (just a few weeks ago) refers to them as "demonstrations." That's also a term that appears in the very vague section on field testing in the original contract (a section that was taken straight from MWI's own specifications). It's probably a key semantic difference in Corps-speak which is conveniently left unexplained to the layman. Dollars to doughnuts, a "demonstration" does not qualify as a full performance test. But the Corps is perfectly happy muddying the waters for the unitiated.

In running all these "demonstrations," and in not performing the contract as it is written, the New Orleans District has decided not to determine if what the taxpayers have spent millions of dollars on actually meets the specification! It also appears that in portraying the demonstrations as legitimate performance tests (schizophrenic, or maybe duplicitous, quote from Corps May 31 press release: "they've been successfully tested"), there has been a coordinated effort to cover up the lack of full performance testing.

Why not run the full performace tests?

Why wouldn't the New Orleans District and MWI want to know for sure if the pumps meet the spec? Why not yank every one of the pumps out one at a time, along with their twinned drive unit, mount them on a barge in the lake, and run 'em like they're supposed to be run under Hydraulic Institute testing standards? That is, why not test the pumps through their entire range of performance as they were specified?

The internal investigation also provides that answer: the pumps don't meet the specification. According to previously unreported limited, but pretty rigorous, pump testing performed late last year (internal report Attachment #3) and earlier this year (Attachment #13) by Corps hydraulic scientists, all the pumps fall below the requirements for flow. In factory testing last fall, when a new pump curve was generated (where was the certified version from MWI?), the shortfall was in some cases over 5%.

In field tests just two months ago at London Avenue, the numbers were much worse, with capacity shortfalls of over 50%. The Corps researcher guessed that the April, 2007 London Avenue results were due to poorly calibrated flowmeters or perhaps bad placement of the flowmeters, and used the pump curve generated in the fall testing to determine flow (using the pump curve, the Corps researcher again called out about a 5% flow shortfall for the April test). But that pump curve itself has errors and scatter, meaning the error in the April test was compounded.

[Special aside on flowmeters:

The London Avenue tests used acoustic flowmeters placed on the 9'-0" manifolds, just as I had suggested was the best place for them in my May 1 post. Even so, the Corps researcher got pretty cruddy results.

It is instructive that the Corps researcher in April, 2007 (just two months ago) did not depend upon the readouts from the individual flowmeters already installed on 5'-0" discharges of the pumps, but instead installed new flowmeters just for the test. That says to me that the meters on the 5'-0" discharges are - as I suspected - worthless. Keep in mind those are the meters whose results give Bedey the Wheaties to make his pronouncements that everything's shipshape. Thus, Bedey's full of it.

The poor showing of the meters on the 9'-0" tubes also reinforces the notion that the ones on the 5'-0" discharges aren't giving good data. If the Corps researcher couldn't get good results with 4 diameters of runup to the meters (the approximate amount at London's 9'-0" manifold pipes), how in the world could meters with just 1 diameter of runup (those that are installed on the individual 5'-0" pump discharges) be expected to give anything but garbage?

Besides all that, though, there remains the question of whether the meters in the April test were way off - and their results can be dismissed - or whether the Corps researcher was just engaged in guesswork as to the results of a 50% capacity decrease. He refused to express confidence in any flowmeter readings at the floodgates when I spoke with him recently, and said it was still a subject of much debate. He also said he didn't want to seem like he was "covering anything up," but that he needed to go through channels before he discussed anything further. I told him that we already deep into hurricane season, and the time for "channels" was long past.]

Frankly, the Corps has no real idea how much water is flowing through the pumps (because they don't trust the flowmeters that are in place now - where have I heard that before?), but they are sure it's less than what it's supposed to be. In any case, there's no real basis for Colonel Bedey to pronounce that the pumps are working as designed.

Spread over the 18 units at 17th Street, a 5% capacity decrease means there's effectively only 17 pumps there. A 50% capacity decrease ... well I think you can figure out how bad that is. That's hardly good value, and is indicative of very poor engineering and even worse stewardship over the funds of the taxpayers.

Any other reasons the Corps wouldn't want full testing?

There's the obvious one: when subjected to similar testing for a week in Florida, the pumps failed at a spectacularly high rate.

They also failed to meet the specification for flow and head in that testing.

All of that is laid out in Ms. Garzino's memo, the shop inspection records, and in the internal report. All of them describe numerous failures throughout use of the pumps over a long timespan. Again, that's hardly good value.

Pulling the threads together

Colonel Bedey lied repeatedly in his interview with CNN's Anderson Cooper on June 11th. His biggest whopper - that the pumps would be performance tested in New Orleans instead of Florida - is undercut by a mountain of direct and indirect evidence. In fact, not only was the performance testing not moved, it has never been performed at all.

Bedey's lie also unspools - and perhaps help explain - a disturbing series of events.

1) After just one week of full performance testing, that testing was stopped on the authority of individuals who were not allowed to make such a change.

2) Then, for the last year, the Corps New Orleans District (NOD) has decided not to enforce the provision of the contract requiring full performance testing of all pumps, and they have not documented such a substantial change in the normal Corps way - i.e. they are operating outside of normal Corps channels without authorization to do so.

3) Instead, the Corps NOD is holding public "demonstrations" of the pump systems which are portrayed to the public (through the media and the New Orleans District Public Affairs office) as adequate performace tests in conformance with the contract.

4) The few semi-rigorous, non-"demonstration" tests done on the pumps since their installation show them falling below the required head and flow as called out in the contract. The first testing that showed the shortfall happened April, 2006. And then there was testing in November, 2006, and then more testing in April, 2007. Shouldn't they be following up on this?

6) The New Orleans District is making millions of dollars in poorly documented payments to a supplier that - according to the Corps' own hydraulic scientists and engineers - did not deliver what was specified.

7) The New Orleans District contracting office is acting as extremely poor stewards of public funds, leaving out - or perhaps not even creating - the required justifications for many of the over 30 modifications to the pump supply contract. They then refused to turn over a complete accounting of the project to the internal investigators:

"The Contracting Officer was requested on 11 April 2007 to provide a total dispursement screen from CEFMS to verify the actual expenditures to date on this contract. This was not provided to the Artman Team by the Contracting Officer." [Adobe pages 21-22, internal report]

Does any of this sound like the normal way to run a project? No, of course not.

But it does sound like the way potential indictees act when they are trying to hide their actions.

All of this - the cruddy accounting, the poor recordkeeping, the willingness to turn a blind eye to supplier inadequecies, the lies to the public - it all points to a cover-up of likely motivations - people in the Corps (Bedey, St. Germain, Bradley) were so interested in getting MWI's equipment on site and portraying it as non-defective that they would do anything for over a year to dupe the public into believing they were protected. Surely they should lose their jobs for this.

How in the world can the people of New Orleans, the federal government, those within the Corps, and anyone else trust these folks to build a functioning flood protection system when they engage in unethical behavior and skirt the law and their own regulations?

In light of the lives and property at stake in New Orleans, what could possibly motivate a person so strongly to do such things? I'm pretty sure only subpoenas are going to let us know the complete truth. I hope the Justice Department is paying attention.

If the Corps wishes to dispute this, then they need to make the involved persons (specifically Maria Garzino, Jim St. Germain, and Dan Bradley) available to do so. Anything else is just bloviation and spin.

Cover up, Part 1

2007-06-23T16:18:00.000-07:00

Introduction

Today I'm starting a multipart series of reporting and analysis of the facts revealed in the Corps' internal report on the floodgate pumps and drives. Today's part focuses on what I believe is the latch to Pandora's box: a lie by Colonel Bedey on national television.

That lie on CNN leads to a horrible conclusion: the Corps has deliberately avoided enforcing its own contract to keep egg off its face and to protect its supplier. Doing so is, well, crazy, and possibly illegal. One would think this is enough for people to lose their jobs.

The Quote

On the June 11th edition of CNN's Anderson Cooper 360 (transcript here) there was the following exchange between Cooper and Colonel Bedey:

"COOPER: There was a report last year by an Army Corps of Engineers engineer who said it's not working. The pumps won't work. And just this past Friday three engineers backed up that report. So do the pumps work?

[...lots of other stuff about how much Bedey loved and appreciated being investigated...]

BEDEY: "...And in doing so [putting in the floodgates and pumps], we had to make some hard decisions. Some of those decisions were to forego all of the factory testing and get the pumps in the water, knowing that we would have to do all the testing and commissioning here on site.

"We've done that. We've been successful."

I've added the emphasis, because it opens up a can of worms. It is a slight variation on the standard line the Corps New Orleans District has used to explain away the installation of defective pumps last year, events revealed by the release of Ms. Maria Garzino's memo this past March 13th. Usually they say they wanted to get pumps in the water, and then "troubleshoot" them later. However, this time Bedey switched it up a bit and mentioned the testing that was so thoroughly eviscerated during the pumps' and drive units' time in Florida.

Bedey's used this line before, though not in such a prominent forum. In a piece of Corps propaganda from April of this year, he said, "We decided we would work out the final testing on the pumps in place."

The language for that testing in the contract calls for full performance testing of every pump system in accordance with Hydraulic Institute standards. This would mean running the pumps through their entire range of flows and pressures under as close to "real world" conditions as possible. This was the segment of the testing which was unofficially "eliminated" by unauthorized Corps personnel and MWI in Florida after just a week of running the pumps last April. The exact day of the decision was April 18, 2006. The modification to the contract dealing with changes in the testing did not come until two weeks later, on May 2, 2006. As we will see below, the official modification did not eliminate the contractual requirement for full performance testing.

Bedey is saying the Corps had/has every intention of still doing that contractually obligated full performance testing; they would just do it on site in New Orleans.

What's interesting is that, unlike many of the other standard excuses the Corps uses, this one can be truth-squaded with documentation now available courtesy of the internal investigation. So in this part of the report, we'll examine the truthfulness of Colonel Bedey's statement. In the next part of the report, we'll take a look at its implications and consequences...

We'll look at three sources, all from the Corps themselves: 1) the text of the internal report, 2) the relevant contract modification, and 3) contemporaneous internal emails.

Exhibit 1: The internal report

The report has extensive passages on the elimination of contract-required full performance testing. It has this to say about the current state of affairs:

"Modification P00004 revised the static test procedures by clarifying the actual steps to follow but no where did it delete the requirements of the full size water testing for each pump. This testing was not performed in accordance with the contract requirements and a significant credit is due to the Government for the nonperformance of it."

So the requirement for full size testing has yet to be fulfilled. That's odd, because Bedey said on CNN they've done that testing and it's been "successful." Hmmm...

Exhibit 2: P00004

As the quote above from the internal report shows, Bedey was talking about a May, 2006 modification to the floodgate pump contract called P00004, which dealt with testing of the pumps and drives. The entire text of P00004 is included as Attachment #11 to the internal report. It is the only modification given such treatment, which attests to its importance.

For your convenience, I've broken it out as its own file. We know that P00004 is the only modification relating to changes in testing because the internal report acerbically notes, "Modification P00004 (See Attachment #11) was the alleged modification that revised the testing requirements of the contract."

Let me cut to the chase on this one - there's no mention in P00004 of eliminating the complete factory testing and replacing it with complete field testing in accordance with HI standards.

Assumedly, if Bedey were telling the truth, some mention of changing the testing from the factory to the site would have been included P00004.

So the only relevant modification to the contract does not mention what Bedey said it mentions. Hmm...

Exhibit 3: The emails

The internal report also contains contemporaneous internal emails among Corps personnel assigned to the Pump Team. Were Colonel Bedey telling the truth, the movement of the full performance testing of all pumps from Florida to New Orleans would surely show up in these emails.

Instead, three critical emails from late May, 2006 (when these pumps were supposedly - according to Bedey - to be fully tested) show no evidence of a testing plan for the site. They also show no evidence of a paperwork trail to back up Bedey's assertion.

The emails are on Adobe pages 122, 123, and 124. I have broken them out as their own link here.

First is a May 17, 2006 email from Joe Thomas to Maria Garzino and Dan Bradley. It states:

"Would like to establish a QC/QA program for field testing of the engines and pumps. Need some help to determine what testing is needed; parameters of the field test and what should be recorded for these tests and what is deemed an acceptable field trail [sic] run. What field tests will be run? How long? Acceptance criteria? Flow measurements? Can anyone develop a document so that we may record pertinent data and information. Such items as pressure; temperatures of system ??? Any help would be great."

Mr. Thomas appears unaware of Colonel Bedey's supposed (read: fictional) plans for full performance testing of all pumps.

Next is a May 17, 2006 response email from Maria Garzino, the Pump Team Leader who was then on site in New Orleans, and would obviously be the person best qualified to know if there had been any plans to move the factory testing to the field. It is sent to a bunch of people, including Jim St. Germain's cohort, Dan Bradley. The tone of the email is almost pleading.

"I do not have any information on how we are to proceed with field testing – I am unaware of any direction given by MWI as to what they envision for field testing (I have asked on many occasions for this info and have not to date received it – if it has been sent, possibly Dan [Bradley] has it then?) – I am also unaware of any direction that Task Force Guardian has given on how we are to proceed on our own as far as field testing of the Pumps and Drive Units – I have heard there is discussion in this area, but I am unaware of any direction given."

The person who would be responsible for overseeing field testing that should mimic the factory testing (which she herself witnessed) had no information about field testing.

There still remains the remote possibility that Ms. Garzino just didn't have the right memo in her inbox. That possibility is foreclosed by the email to which her email is attached. That email, dated May 31, 2007, is from Jim Bartek to Steve Farkas and Bob Hoffman (why in the world is Ms. Garzino not included?). It states,

"I have not received a memo from Jim [St. Germain] regarding testing. All I have received is the email below that I think you sent out [Ms. Garzino's May 17th email]: I would agree with the recommendations except to add monitor vibration. Keep in mind this was a supply contract and the spec is very vague regarding field testing. It just says 'The pump manufacturer shall provide for final inspection and testing of the system and shall make necessary adjustments to the control system prior to actual start-up tests. Start-up tests and demonstration shall be performed by the pump manufacturer’s representative and the Contractor, and witnessed by the Government…'"

Notice what Mr. Bartek is citing when referring to field testing: the original contract! He's not referring to this phantom modification that Bedey is making up that somehow justified speeding up and eviscerating the factory testing. Bartek's citing the original contract.

So the contemporaneous emails show no evidence of what Bedey asserts.

The exhibits lead to one conclusion

Thus, we have the internal investigation, the relevant contract modification, and internal emails all saying the same thing: the Corps New Orleans District willfully ignored a vital part of its own contract; after P00004, there was no plan for full performance testing of the pumps, and there still isn't a plan for full pump-and-drive performance testing to this day.

Put another way, Colonel Bedey has made up a fairy tale to cover up something far more disturbing. The Corps was flying (is flying?) by the seat of their pants when it came (comes?) to the on-site testing, working without the paperwork to back themselves up and safeguard the government's (and the taxpayers') interests. Bedey, St. Germain, and Bradley know this. This is a major, major violation of construction contract administration. But it gets worse.

In the next part we'll look at why they're doing this and the consequences of not conducting full performance tests.

AC360

2007-06-13T09:50:00.000-07:00

Colonel Bedey popped up on CNN's Anderson Cooper 360 Monday night. You can find the transcript here. It was his first interview after the release of the Corps internal investigation into the floodgate pumps. Cooper interviewed him while standing on the deck of the 17th Street gates (talk about home field advantage).

He put out more than a few falsehoods in the interview.

Lie #1: [speaking about the internal investigation] "BEDEY: That report -- that report that was just published last week was actually a snapshot in time of September of last year."

What? Um, the date on the front of the report is May 11, 2007. There are pieces of correspondence attached to the report which are as recent as May 7, 2007. There are ten technical recommendations for work which (according to the report) have not been completed, including raising the hydraulic reservoirs - which was supposed to be done over a year ago.

My guess at what Bedey meant? I'm not really sure. Frankly, it sounds like something a crazy person would say, like pointing at the sky and saying, "What a lovely shade of green!" Maybe he slipped and let out an old talking point that referred to a previous draft of the report which only examined September conditions (the report shows a flurry of work last September, including site visits and document collections). But the report now is most definitely written in the present tense, not the past.

Lie #2: "Well, we're absolutely stronger today than we were pre-Katrina. This pump, this interim closure structure is an example of that."

Again I say, huh? While the gates themselves might be stronger than the outfall canal walls (emphasis on might, since the system has yet to be battle tested), everyone locally knows there's less pumping capacity at the gates than at the Sewerage & Water Board pumping stations. That's why there's more pumps being put in at 17th Street and London Avenue, and even then there won't be enough pumps to match the interior stations' outputs. So in a fundamental way, we're weaker today than we were pre-Katrina. We can't evacuate rainwater during a hurricane as fast as we could pre-Katrina.

It's easy to slip stuff like this past reporters that are only in New Orleans for the day.

By the way, what's this "we?" Bedey lives on the North Shore. He should really refer to "New Orleans" rather than grouping himself in with the city.

Lie #3: "The commitment of the administration of the United States and the Congress of the United States was to provide the 100-year level of protection for the people of this great community and of the nation."

I really don't like straying into political territory, but I have to just put a toe in here.

This is what President Bush said in Jackson Square on September 15, 2005:"City and parish officials in New Orleans, and state officials in Louisiana will have a large part in the engineering decisions to come. And the Army Corps of Engineers will work at their side to make the flood protection system stronger than it has ever been."

Now, if the flood protection system was supposed to be able to handle a 100 year storm (the basis for the FEMA flood maps), then wouldn't "stronger than it has ever been" refer to greater protection than a 100-year storm?

I know I'm being a little obtuse here. I recognize the realities of Congress, the Administration, budgeting, and politics have whittled the stated goals of improving New Orleans' flood protection down to, "We're going to give you what we said you had for 40 years." But it's still wrong.

There's more to come from this interview...

Credibility

2007-06-11T19:56:00.000-07:00

The Corps internal investigation is the gift that keeps on giving. And, as with any government report, the tasty tidbits are in the appendices. Here's three that I've found so far.

Note that I refer to page numbers by "Adobe page so-and-so" That's the page number in the box at the bottom of Adobe Reader.

1) The hydraulic oil is not environmentally friendly.

Back on September 16, 2006, when there was hydraulic oil spilling everywhere, Colonel Bedey was quoted in the Times-Picayune as saying, "We're using environmentally friendly hydraulic oil."

That may have been true back then. The National Response Center spill report for the September 14, 2006 spill shows the material spilled as "MOBIL 67 BIODEGRADABLE HYDRAULIC OIL." (the 67 is probably a typo; the next nearest ISO grade is 68). Possibly, it was one of the lubricants on this Mobil page, all of which are advertised as environmentally friendly. But with such a paucity of information, I could never narrow it down, or even verify the claim.

Since that time, I've been curious to find out exactly what kind of hydraulic oil is being used, in order to find out if it really is "environmentally friendly." Thanks to the internal report we now know. On Adobe page 100 is a May 7, 2007 memo from Denison to the Corps' Dan Bradley. It calls out the specific brand and type of hydraulic oil. According to the memo, the hydraulic oil is Exxon-Mobil Nuto H 68.

Taking a look at the product information page for Exxon Mobil Nuto H, I can't see anything in there that mentions "environmental friendliness." In fact, it appears on a webpage with a family of products that is notable for its lack of "environmental friendliness." Such a quality is a big marketing tool, and if it were part of the makeup of the oil, Exxon Mobil would definitely mention it. It appears the Corps and MWI are no longer using environmentally friendly oil. Someone should tell Colonel Bedey.

2) Water in the hydraulic oil

Buried inside a field report by a contractor about a visit this past February to the hydraulic motor manufacturer, Rineer, is a yummy morsel about the hydraulic oil having water in it.

On Adobe page 88, Corps contractor John Ehlers describes the teardown of three Rineer hydraulic motors. These are three of the motors which were vibrating severely, and which have had stronger springs installed in them since then. This was, for months, the only problem to which the Corps would admit when it came to the floodgate pumps. What we have been told publicly was that springs in the motors were replaced with stronger ones at MWI's rep, Associated Pump & Supply in Houma, LA.

What has not been publicized until now is that the motors were not in as-new condition when they were reinstalled at Associated. Specifically, Mr. Ehlers writes this:

"The writer notes that the coil spring material is 'piano wire', which is a high carbon alloy. Any high carbon alloy is a candidate for rust when exposed to water and warm temperature and other contamination. The rust evidence on the timing plates suggests that there was some water contamination within the motor. Since the motor seals were all good, it is probable that the water was introduced by hydraulic fluid which had been exposed to water or moisture."

He also wrote in his conclusions section:

"All 3 motors had slight signs of rust on the timing plates. There was evidence that the hydraulic oil was less than perfect, in that a contaminant, most likely water, was present in the hydraulic oil."

So the hydraulic motors were put back together with rusty parts inside them.

But what's more interesting is the fact that at some point between installation of the pumps in the May-June-July timeframe and their initial startup testing during the same period, the hydraulic pipes were contaminated with water.

This has large consequences. The following is just theory, but it's what I think would normally happen in the case of contamination of the hydraulic fluid. There's only one source for water to get into the hydraulic lines. It was there to start with.

You might say, "What if the system had leaks?" A system running at any appreciable internal pressure is not going to allow water in while it's running. And while it's conceivable that water leaked into the system, it seems unlikely to have done so in three separate pump systems in exactly the same concentration (Mr. Ehlers writes of similar "slight" amounts of rust in each motor). So I'm going with the theory that the water was in the pipes to begin with.

As everyone knows, water and oil don't mix. When the pipes arrived on site they were supposed to have been pickled and oiled, according to the specification. The pickling process involves dipping the pipe in vats of acid and other chemicals to passivate it (I think - I'm a little hazy on pickling). Then I believe the pipes are supposed to be coated in oil and then capped and shipped. Theoretically (I think), the lines should be flushed completely before startup to ensure there was no debris or other crap in there before the final charging of hydraulic oil was placed. Feel free to correct me on this.

I'm betting that the lines were either not pickled and oiled properly or not flushed properly. What would this lead to? Once it was determined there was water in the oil (it would have showed itself in other ways during startup operations), all of the pipes would have had to have been completely flushed and refilled with new hydraulic oil. The change in oil described above - from the "environmentally friendly" grade to the Nuto H grade - could be explained by this. This work would have been at great expense and would have been backcharged to MWI if they didn't do it themselves.

All of the above is theory, but I like to think of it as deductive reasoning based on the available facts. Anyone who has evidence to back up or dispute my theory about flushing of the lines is encouraged to contact me.

3) When was it decided to raise the hydraulic reservoirs, the so-called "critical fix?"

The critical fix that remains undone is the raising of the hydraulic reservoirs. With that modification, hydraulic oil would flow downward from reservoirs into the Denison hydraulic pumps, instead of having to be sucked out the top of the reservoirs. Pulling the oil out the top of the tanks guaranteed entrainment of air in the lines. Denison pumps do not tolerate air very well. They were failing like crazy in the factory with the lines coming out of the tops of the reservoirs. And apparently, they continued to fail after they arrived on the sites (from Adobe page 24):

"The hydraulic oil pumps, since the delivery to the site [emphasis mine] have been removed and inspected by the manufacturer to determine premature failures, similar to the ones that were happening at factory testing of the pumping units."

This single sentence is huge in its implications. It totally rebuts the Corps' entire defense that they were installing "some capacity instead of none." Colonel Bedey and others have repeatedly used this line with the press, with the public, with policymakers in Congress, and even got the GAO to buy it.

That defense is BS. How could you have any capacity if the most critical part of your system was continuing to fail even after you installed it? It is very important that we find out the extent of the damage to these Denison pumps after their arrival on site. 74 (two on each of the 34 skids) were installed across the three canal sites. How many were determined to have failed after they were installed in New Orleans? 3? 30? 60? Knowing that would blow a giant hole in the Corps' credibility, because it would mean that there really wasn't any capacity to begin with - and there probably still isn't any, because the reservoirs haven't been raised in order to put the kibosh on the air-induced failures.

Yes they can turn them on for a little while, but how about for six, or eight, or 12, or 24 hours? What happens when air gets in the lines and the Denison pumps fail and send their bits and pieces down the line to the Rineer motors to gobble up? What then?

Anyway, as far as the time that the real solution (raising the reservoirs) was reached, it appears to be in an email found on Adobe page 114. It was from Jim St. Germain and addressed to nearly everyone on the project. It is dated June 18, 2006 and says,

"A meeting was held with MWI to discuss weld repairs and the Denison hydraulic motor. MWI has proposed to redesign the hydraulic tank and hoses to form a flooded suction for the two Denison hydraulic pumps. MWI will raise the hydraulic fluid tank and reposition the hoses to the Denison pump. This fix should prevent air from entering the motor and eleminate [sic] the need for the priming valve. MWI will also remove the suction strainer.

The proposal says that MWI will perform the work at no cost to the Government provided Government onsite contracts load and inload the skids. They propose to take the engines skids at 17th that are not installed and the spares first and then swap engins [sic] at other sites to try to avoid pump outages. They estimate a week to do a set (6 to 10) units.

This presents some problems. First, the preliminary time frame is not going to allow completion without impacts on scheduled pump completion dates. Second, once the fencing on the engine deck and building is installed it may be difficult to remove the engines. We should immediate[ly] have MWI retrofit the engines at 17th that are not installed. Coordinate with the three contractors on swap outs to minimize impacts to buildings. Avoid any impacts to pumping capacity. Have MWI develop a plan to retrofit in place."

Well, that work never took place, at least on the original 34 pumps. The roofs were put on the buildings and the fences were erected. Somehow, the reservoirs were raised on the extra six pumps puchased long after the original 34.

This June 18, 2006 email shows that Jim St. Germain - the guy responsible for driving the project - and the manufacturer - MWI, who apparently proposed the idea - had a clear appreciation of what needed to be done. Heck, St. Germain's the author of the email. Yet it took over a year to get this work going (we're still not sure what its status is), never mind how the public was in the dark about it. Why didn't it get done? Unfortunately, the report is silent on the why's of many things, including this mystery.

And it's not like they were pulling this solution (raising the reservoirs) out of the air. Two weeks earlier, the New Orleans Corps had discussed how they didn't like the quick fix of evacuating the hydraulic lines during startup. This comes from a May 31 email from Jim Bartek to Jim St. Germain, Dan Bradley, and Steve Farkas (but oddly, not Maria Garzino, the Pump Team leader on site):

"Just a few thoughts...It appears to me that MWI is just trying to 'make it work' to get by. Everyone agrees that the Denison pumps are operating in a dry run condition. While the priming procedure described below may work as a one time or short term fix. I feel, and I think Steve agrees, it will not hold up over the long term. There will be entrained air that will percolate out and also air leaking in through the tank etc that will get trapped. They need to look more at root causes than quick fixes of the problems. This could be pump sizing or relocating the suction intake to a more suitable location. Why has this design successfully worked on other pumps but not these units?"

The priming procedure referred to by Mr. Bartek over a year ago is the same procedure under which the Corps is now operating the pumps.

Mr Bartek goes on to presciently rebut Colonel Bedey's current assertions that the field testing the Corps is doing today (turning the pumps on and then turning them off in less than hour with low water in the canals) is adequate:

"These pumps must be field tested with design water levels to ensure acceptable operation."

Like I said, this report is the gift that keeps giving. I'll be back with more analysis in future posts.

Testing, Testing

2007-06-11T13:58:00.000-07:00

Updated 6/22/0, 7/28/07, and 8/13/07. See below.

The Times-Picayune actually managed to put together a sensible article on the upcoming London Avenue canal tests this past Saturday, June 9. I say that, because it's the first time that anyone in the media or at the Corps New Orleans District has admitted the following obviousness:

"Corps officials say they don't know if the load test will provide enough information to change the safe water level and, if it doesn't, what they might suggest doing next.

[Geotechnical engineer for the Corps' St. Louis District Pat] Conroy won't even guess at the outcome.

'We reserve the right to make no prediction at all,' he said. 'We may recommend raising the water level or not changing it or even lowering it.'"

Finally! Someone at the Corps actually admitted they don't know the results of the test two months ahead of time. It's better than the near constant pablum from Bedey, Wagenaar and others that implies that the test is a formality and that the Safe Water Level will be raised to 5 feet.

The article mentions that a contract has been issued for the test. You can find the official notification of the contract award buried on page 5 of this notification.

The article didn't really have anything to say about further testing after the localized test. However, a June 2, 2007 article in Engineering News-Record about General Van Antwerp's "truth and victory" tour through New Orleans at the end of the month provides the necesary information:

"The Southeast Louisiana Flood Protection Authority-East, one of the consolidated levee boards formed in January, has been very vocal in questioning the safety and validity of the London Avenue load test. Antwerp gave assurances that the Corps will have ample personnel on the ground to monitor the test. If the test doesn't provide enough evidence to support increasing the SWE, 'any additional testing will not be performed this hurricane season,' Bedey says."

Considering that the testing won't occur until well into hurricane season, and also considering that it took the Corps over six months just to get to the point of contract issuance, it's a guarantee there won't be any further testing.

[The link inside the quote comes from ENR's webpage. ~~Unfortunately, one must be a subscriber to get access to most of their articles. That's too bad, because they've done a ton of coverage on the London Avenue canal testing.~~ Update, 7/28/07: All the ENR articles are linked below.]

One other note: this article says that the four foot safe water level has led to shutdowns of PS#3 pumps "at least once in recent months." This is just the T-P getting snotty over my breaking news that they missed. Of course, we all know that pumps at PS#3 on the London Avenue canal have been shut down twice because of exceeding the Safe Water Level, once on December 30, 2006 (that's the one I broke, and which the T-P took days to catch up to), and once on May 4, 2007.

[Update: 6/22/07]
Today the T-P ran an article about the East Bank Levee Board (officially known as the SLFPA-E) meeting yesterday (see 8/13/07 Update below for minutes of meeting), at which the independent experts brought in the review the Corps' plans for the London Avenue canal testing presented their findings.

While the experts gave a thumbs up to the existing plans to examine the canal walls' susceptibility to gap formation, they also strongly recommended expanding the testing to look for problems with seepage:

"...Specialists brought in at the behest of levee authority members for a second opinion reported Thursday that they are even more concerned about a second potential mechanism for failure, one in which water would seep through sandy soils in the bottom of the London Avenue Canal, causing heaves or blowouts in the earthen levees on the land side of the floodwalls.

The pair told authority members on Thursday that both failure mechanisms should be examined during the same test period. "

This is smart. Seepage - contrary to the article author's totally incorrect assertion that gap formation led to both London Avenue canal breaches in Katrina - is what caused the blow out at the London Avenue South breach just north of Mirabeau Avenue. It is also suspected as a contributing failure mechanism in the London Avenue North breach. It is wise to look at seepage in this test.

However, one has to wonder why the testing plan didn't call for seepage testing to begin with, since top engineers in the Corps' New Orleans District were already citing seepage last year as a problem on London Avenue. Here's a quote from a September 12, 2006 Times-Picayune article (by the same author as the others - one has to wonder if she even reads her own clip file):

"Although most public attention since Katrina has focused on the 17th Street Canal, [Corps chief engineer Walter] Baumy said protecting the London [Avenue] canal from surge is more problematic because sand layers are closer to the surface there and seepage -- with potentially catastrophic results -- is a threat."

Also, that Corps guy Pat Conroy from St. Louis was quoted extensively in a Corps article just last June (yes, that's post-Katrina, post-IPET report release) about ... wait for it ... seepage! Here's one of those quotes:

"In fact, investigations into the occurrence and control of underseepage are a major consideration of the geotechnical design of a flood protection system"

Why wasn't control of underseepage even a factor in the Corps' testing of the London Avenue canal until outside, non-Corps experts brought it up?

I get the idea that these guys at the Corps really don't know what the hell they're doing.

Anyway, the Corps said they would incorporate the additions to the testing plan, and the SLFPA-E (that's tough to type!) approved the testing plan.
[end 6/22/07 update]

[Updated 7/28/07]
Engineering News-Record did a fabulous job in April and May covering this and other canal wall-related issues with a series of articles. They did much better than the Times-Picayune. Here they are:

Corps Wants Coffer Dam Test Nola Flood Board Wants Action - 4-12-07
Levee Board Gives Corps Conditional Permission for Levee Load Test - 4-20-07
Levee Board Holds Corps To Peer-Review Standards - 5-2-07
17th St. Canal OK Corps Says But Not So for London Ave - 5-25-07
Load Ratings of Drainage Canals Become Sore Point in New Orleans - 5-30-07

My favorite quote from these articles comes courtesy of Tom Jackson, then-president of the SLFPA-E. In the May 25th article, he says,

"'London is a problem,' says Tom Jackson, SLFPA-east president. 'Taxpayers paid for +12ft of protection and we got four,' he says. 'The whole thing about this exercise is the calculations for safety are 1.3, and they want to perform more tests to raise levels on levees that are absolute junk at best. The problem with this is this is not a storm-event-situation but every-time-it-rains situation. That’s why I keep telling them to fix the damn levees. If the peer reviewers are worth their salt, they will have the same reaction we do. Fix the damn levees.'"

Hmmm... where have I heard something like that before?

Previous posts that I've written about the London Avenue canal testing:
London Avenue canal tests, originally posted March 27, 2007
It's May in New Orleans - that means rain , originally posted May 7, 2007
Prepare for more PS#3 pump shutdowns, originally posted June 1, 2007
[end 7/28/07 update]

[Updated 8/13/07]
Here's more information about the London Avenue load test.

First, here's the minutes from the June 21, 2007 SLFPA-E meeting. That's the meeting at which official approval of the test was granted. The minutes contain all the nitty-gritty technical details of the testing plan. Since they are a primary document, they are far more informative than media reports.

Second, the Washington Post wrote about the test in its August 11, 2007 editions. The article treads much of the territory I've covered here, but it's still good for those folks in the rest of the country to get an introduction to real-time oversight of the Corps. The article by Peter Whoriskey, titled "Engineers to Test Flood Defenses In New Orleans," is here.

Finally, the Corps Hurricane Protection Office put up a "fact sheet" on its website this past Friday, August 10, 2007 (they always sneak this stuff out on Fridays). It too includes much previously reported information, with one important difference. It gives the start date of the test as Friday, August 17, 2007. It also reports the test's duration to be two weeks. The Corps fact sheet is here. Of course, in the section on "Why Test?" they don't mention that they don't have the money to actually fix the canal the way it's supposed to be.
[end 8/13/07 update]

Bombshell

2007-06-08T14:30:00.000-07:00

Updated 6/8/07 and 6/9/07, see bottom of post

The Corps internal investigation into the floodgate pumps is out. Big surprise, it comes on a Friday afternoon. But let's move past that bit of PR manipulation... The report is stunning in its breadth and depth. Things are way worse than I knew.

First off, I have to applaud General Crear (of the Corps Mississippi Valley District) for convening this team and allowing them to go as far as they did. The amount of crap this investigation uncovered is breathtaking. I know a lot of people don't like the Corps investigating themselves, but when put up against the GAO report, there's no comparison. It appears the GAO report was a total waste of time, and is pretty much useless in the scheme of things. Were I in Landrieu's office, I'd be calling on the GAO Inspector General to look into how an investigation that had to have looked at the same information the Corps did could have come up with such radically different (and obviously wrong) conclusions. We can only hope they now go back and expand their investigation, or maybe the FBI will look into things...

I never would have guessed that of the two investigations, the Corps internal one would be the one with the most shocking dirt. But that's simply the truth. The Corps independent investigative team - as well as Ms. Garzino and some of the Jacksonville District QA's and probably some others for who we'll never know names - have really tried to do the right thing here, and I salute all of them for it. This can't have been easy.

Frankly, it's easy for me to sit back and lob rhetorical bombs at the Corps and pat myself on the back (even if my neighborhood would suffer some of the worst consequences when the pumps fail). It's a lot harder to uncover this stuff from the inside, and to have to confront the fact that people to whom so much has been entrusted appear to be massively incompetent and corrupt. I've been on a few internal forensic investigations in my time. It's gut-wrenching stuff, but it is necessary, especially in this case. In fact, this case screamed for it. The folks involved in this effort deserve a ton of credit.

However, I must also take General Crear to task, because his cover letter doesn't really match the actual words in the report. He downplays the deeply, deeply serious findings on both the technical and commercial side of things. Honestly, one must look at his cover letter with a jaundiced eye, because the text of the report is very critical, but his cover memo is not. It reads more like a press release, and it even seems to contradict the report in one key respect. More about that below.

So on to my initial impressions...

1) Colonel Bedey, Jim St. Germain, and Dan Bradley have been lying to us in New Orleans and Jefferson Parish, all of the media (local and national), and the entire nation for a year, right up to and including Bedey's assurances during General Van Antwerp's "Truth & Honesty" tour last Thursday. There is a critical fix (as in, the pumps could fail without it) that needs to be applied to the drive units. The New Orleans Corps has known about it and has not performed that fix since June of 2006! I don't see how these men can keep their jobs.

2) That critical fix involves raising the hydraulic reservoirs on all 34 drive skids so that hydraulic fluid flows into the Denison hydraulic pumps without introducing air. Air is deadly to these pumps (a ton of pages in Denison's troubleshooting manual is devoted to just such problems). As a reminder, the Denison hydraulic pumps - which are supposed to raise the pressure of the hydraulic fluid to 3000 psi or more - were one of the centers of gravity that Ms. Maria Garzino was identifying as seriously flawed in her memo from May, 2006. They make numerous appearances throughout the shop inspection records as well. They were failing left and right during the factory testing.

What this report shows is that the Corps (specifically Jim St. Germain and Dan Bradley) were keenly aware of the fix as early as May, 2006, and have done nothing to implement it. There's actually a change order to do the work that's been in place since July, and it remains unfulfilled! It is a strong possibility that those Denison pumps remain at risk for failure, since the flooded suction fix has not been implemented. A temporary fix is in place (which I'm sure Colonel Bedey will try to hang his hat on), but the internal investigators eviscerate its effectiveness:

"The flooded suction intakes have only been accomplished on the new additional contracted 6 pumps, the original contracted 34 pumps have not been revised to add the flooded suction intake as was agreed upon to be revised in June 2006. MWI has only provided a vacuum type check valve for priming the suction to the Denison Hydraulic Pumps. This is only a façade in addressing the real issue and requires the operation of vacuum equipment to prime the hydraulic pumps. If the vacuum is not drawn properly, then the pumps will aerate and create irreversible damage to the components of the pumps. The bilateral contract modification to change the intakes to a flooded suction at no cost to the Government has been in place since 12 July 2006 and has not been accomplished as of this date on any of the original 34 pumps."

Yeah, that's right. MWI and the Corps were so aware of the problem that they implemented it on the six extra pumps at 17th Street, but never bothered to retrofit the other 34, leaving them in the same danger of failure that they've been in since they left the factory.

So according to the report (dated May 11, 2007), this flooded suction inlet work has yet to happen. Oddly, General Crear's cover letter (dated June 4, 2007, or four days ago) says the following:

"...Work is currently underway to modify the plumbing so that the hydraulic pumps can never experience a 'dry start up condition'"

Either way, the work has waited for over a year. Why should we trust them? They lied to us for a year by not even mentioning a critical flaw in the pumping systems, let along not mentioning they weren't implementing their own agreed-upon fix.

I'd like to mention a couple of other things about Crear's cover letter. The report itself mentions the following ten problems on the technical side of things:

1) Lack of flooded suction inlet on Denison pumps

2) Unreliable hydraulic piping

3) Lack of automatic clutch on drive units (a clutch would allow for true unloaded starting)

4) There are still welds that need to be checked (only critical ones have been fixed, and it was at government expense)

5) Lack of confidence in flange bolts (the flanges might not be able to hold the weight of the pumps)

6) Inlet strainers should be removed on drive units

7) Installation of hydraulic cones at London Avenue & Orleans Avenue

8) Lack of hydraulic monitoring device on drive units

9) The flowmeters are unreliable

10) MWI still hasn't supplied certified information, over a year after the scheduled end of their contract.

General Crear, in his cover letter, only mentions TWO of these ten items (flooded suction inlets and welds), and even then in a tangential fashion. He gives absolutely no indication that the remaining eight recommendations will be implemented, or even if they will be acknowledged as problems.

In fact, other than his cryptic mention of work on the drive unit "plumbing," the only action he says he will implement is formation of a team to bring the contract file up to date (as if the contracting improprieties enumerated in the report are simple clerical errors, not misappropriation of millions of dollars). No other actual actions are mentioned in his letter. He even has the gall to say the pumps "are working well." I doubt it.

Let me put my doubt about that this way: if you were in the Tour de France, a grueling 21 day, 2000 mile race, would you consider a brief toodle around the Champs Elysses the day before the race enough of a test for your body and your bike? No. You'd train over the actual course for months beforehand, maybe even climb L'Alpe D'huez a couple of times. Well, the 45 minute pump tests the New Orleans District has run are the equivalent of that toodle around the Champs Elysses. We still have no idea how they will perform in Hurricane L'Alpe D'Huez. If the investigative report is any indication, I'd say the answer is "not good."

By the way, the rusty pipes come in for it too in this report. The report basically says they can't be trusted and need to be seriously examined from stem to stern. That's 17,000 feet of pipe they're talking about.

3) The Corps' New Orleans District's contract administration has been a travesty - perhaps even criminal - and there appears to have been an active effort to cover that up. MWI was writing change orders - including prices - and the Corps was sending them back to MWI with a bow on them. Duplicate payments have been made to MWI. There appears to be no written justification for most of the payments. There appear to be millions of dollars unaccounted for. Cynthia Nicholas, the Contract Officer on this job, has to be squirming today. One has to wonder about the 50 or so other Task Force Guardian contracts administered by the New Orleans District, some of which were much bigger than this one, and how much "impropriety" there was in them. Where have our tax dollars gone?

4) Curiously, there are no names named in the body of the report. But it is obvious who is on the hook, from the emails that are attached: Corps New Orleans District engineers Jim St. Germain (mechanical) and Dan Bradley (electrical). They need to be hung out to dry, as well as their boss, Col. Bedey. After reading this report, there is absolutly no way these folks should be trusted with fixing a bicycle pump, let alone pumps upon which tens of thousands of people are depending for protection of life and property. When it comes to the New Orleans District and the Hurricane Protection Office, houses need to be cleaned, grand juries empaneled, and perp walks televised.

5) A cursory investigation was performed last May in the immediate aftermath of Ms. Garzino's memo. It was called the "Farkas" investigation, after the guy who led it. It apparently didn't even involve Ms. Garzino, it took MWI's answers to its questions on faith and hardly anything came of it. There is an extensive section in the new report which rebuts most of MWI's responses and (implicitly) calls the Farkas investigation and those that received it (who were also in on the investigation) to task for not following up strongly enough. The Farkas investigation appears to have been a sham from the start, and its lack of enthusiastic follow-up from those most closely involved (Messrs. St. Germain, Bradley, & Bedey) points to a cover-up. At the very least, there was a shocking lack of honesty with the public.

6) Colonel Bedey has been tasked with implementing the recommendations of the report. But he's the guy under which all this crap happened, so why in the world should he be trusted? It's nuts. Through his inaction, he has shattered the relationship between the region and the Corps, a relationship which was already severely damaged. One has to be wondering what his, St. Germain's and Bradley's motivations have been for the last year in not performing the critical fix which would actually make the pumps run properly for the amount of time they need to run (12 to 24 hours in hurricane conditions).

7) Finally, one of the most important things on this report is its date. Why in the world are we only hearing about these problems now, over a year after they were initially identified? Does anyone really understand the massive breach of trust this evidences? It's concrete proof that the Corps New Orleans District and the Hurricane Protection Office has withheld, dissembled, and flat out lied to almost everyone, including the Corps' independent investigators.

These issues were very clearly evinced last April and May. Confronted with them, there should be evidence of New Orleans Corps folks moving heaven and earth, but instead there appears to be mostly improprieties and squandering of opportunities. Why did it take General Crear four months to start an investigation? Why has the investigation - by an agency which knows itself better than any outside body - taken over nine months to finally wend its way out to the public? And most importantly, how in the world could there still be critical work to be done, this long after the problems were identified?

Obviously, I'll be writing a lot more about this in the coming days, but let me leave you with this. It appears there have been a few guardian angels inside the Corps watching over New Orleans. Ironically, it appears none of them actually work in New Orleans. The Corps is not some massively evil monolith. It is made up of thousands of people trying to do their jobs. But there are also some bad apples. And it appears we've got them running our work. How this situation has festered for so long, I'll never understand.

Update: Senator Mary Landrieu has called for further investigation. She's asked the Justice Department to look into the contracting administration irregularities, and she's told the GAO to go beyond what they put out a couple of weeks ago.

News coverage of the report release:
WWL-TV (video from 6 PM news): Report says flaws still exist in outfall pumps
Times-Picayune (inexplicably not on the front page, instead buried on the front of the Metro section): Landrieu asks Feds to probe pumps
Engineering News-Record: Internal Corps Report Cites Procurement Flaws
Baton Rouge Advocate: Landrieu calls for review of contract
AP: Corps report casts new doubt on reliability of New Orleans pumps
Fox 8 has not put up a print version of their story

I feel I need to write a little bit about how the Corps played possum with the release of this report. The chronology is important, in order to gain an insight into how these people think.

May 11: The report itself was done.

May 24: The report was "released," according to the cover page, but one has to wonder what that really means, since the public didn't see it until June 8, 2007.

May 25: The national media started arriving in New Orleans to start putting together stories for the start of hurricane season.

May 29-31: General Van Antwerp visited New Orleans and had a big, whoop-te-doo, "We're going to tell the truth, no matter what" press conference at the 17th Street floodgates on May 31. Every news organization in town was there. Joining Van Antwerp at that press conference were (from the Times-Picayune's coverage of the press conference):

"Brig. Gen. Robert Crear, who directs the corps' Mississippi Valley Division in Vicksburg, Miss.; Col. Richard Wagenaar, outgoing commander of the New Orleans District office; and Col. Jeff Bedey, commander of the Hurricane Protection Office in New Orleans."

The Corps actually inconvenienced its own contractors that day by forbidding them from parking their cars all day on the section of the Old Hammond Highway bridge where the presser took place. The press conference took place at 4 PM, but no one could park there all day long.

Also notable about the May 31 press conference: there were a lot of other Corps folks there besides the brass described above. Those were just the ones arrayed behind Van Antwerp during his comments. In the substantial crowd of Corps folks was Jim St. Germain, as seen below.

According to reporters at the press conference, Van Antwerp actually singled St. Germain out as a guy he'd be talking to a lot because he's responsible for the pumps. Oddly, St. Germain didn't show up for the press availability yesterday that bears directly upon that responsibility. That matters.

June 1: Hurricane season started. National media starts heading home.

Monday, June 4: General Crear (or more likely one of his staff, possibly in coordination with Public Affairs) finished his cover letter (it is stamped June 4, 2007)

Thursday, June 7: After repeated inquiries all week, only local reporters are emailed the report. Many public officials are not given a copy. The reporters are told not to report on it until after 2 PM Friday - a so-called "embargo."

Friday, June 8: Early in the morning, public officials at all levels of government receive the report, but not from the Corps.

Friday, June 8: Early in the afternoon, far from the 17th Street canal and its photogenic backgrounds (where TV stations could actually point their cameras at the hydraulic reservoirs that have yet to be raised and easily compare them to the ones that were) - in a conference room at the Corps' building on Leake Avenue, the Corps holds an invitation-only round table to release the report. Colonel Bedey and Task Force Hope head Karen Durham-Aguilera are in the room, while General Crear joins by phone. Walter Baumy, a top engineer for the Corps and former Wheeler Medal winnner, is in the room, but sits to the side.

Six reporters are in the room. They represent the AP, the Baton Rouge Advocate, WWL-TV, Fox 8 (TV), the Times-Picayune, and Engineer News-Record (an industry organ, which has turned up some good reporting, but is still mainly funded through advertisements from the construction & engineering industry). No reporters, other than from the AP, represent national organizations. Two of the local TV stations - WDSU and WGNO - are not in the room. None of the investigators on the report are made available to the press. Ms. Garzino, the one responsible for kicking all this off, is not made available. None of the Corps New Orleans District prime actors (Jim St. Germain, Dan Bradley, or Cynthia Nicholas) are in the room. Notably - in what can only be expressed as a sign of no confidence - no one from MWI is made available to defend their product.

By the way, its very notable that the report does not appear on any Corps websites. Not New Orleans District, not Hurricane Protection Office, not Mississippi Valley Division, not Corps Headquarters, not the Army. These cowards are afraid to post their own report. That hardly seems like the soldierly, honorable, truthful way to do things.

Let's also take a look at the Corps' efforts at stagecraft, because the visuals are just as important as what they say. Here's General Van Antwerp's very public (cars were driving by the whole time on the bridge behind the media) little victory dance on May 31 (courtesy the Times-Picayune):

And here's the Corps literally in their bunker on June 8 (from WWL's report):

That's Col. Bedey on the left and Task Force Hope head Durham-Aguilera on the right. Baumy's in the white shirt on the extreme right. The other folks are the reporters.

These things matter. The Corps New Orleans District (and apparently Task Force Hope, since Durham-Aguilera is right there next to Bedey propping up his dissembling) is desperate to control this story by minimizing the number of folks who are allowed to speak about it, as well as minimizing the number of reporters who could ask questions. They want to put their best face forward and avoid letting the truth out. If they were so concerned about truth and integrity and leveling with the people, why all the manipulation of the media? The answer is they are not concerned with any of the above. They are concerned with saving their own skin. That's going to be a lot harder with the Justice Department breathing down their necks.

Radios

2007-06-07T08:06:00.000-07:00

A little election-driven spat has been going between Orleans and Jefferson Parishes for the last few weeks. Jefferson Parish President Aaron Broussard, the man who sent JP's pump operators 100 miles away during Katrina - causing millions of dollars of damages when those pumps didn't run (there was no provision for remote operation) - wants the Corps to run PS#6 on the 17th Street canal if the gates drop. Orleans Parish's Sewerage & Water Board (S&WB), which has owned and operated PS#6 since it was built over 100 years ago, is obviously opposed.

Broussard's made a lot of noise and even complained to the governor. He's up for reelection this fall. As a result of his complaint to the governor, a meeting was held yesterday. The Times-Picayune reports on it today.

But first, let me remind you of part of my May 11 post, in which I was writing about the Corps communication plan (and the Times-Picayune's coverage thereof):

"Also missed in all the glorification of the Corps' communications plan is this actual quote from the Manual:

'One Corps of Engineers radio will be furnished to the S&WB for communications in addition to having the Canal Captains stationed at their respective S&WB pump stations.'

Yep, that's right: the agency in charge of pumping gets all the benefits of a single radio to communicate with the Corps."

Thankfully, this has now been rectified. According to today's Times-Picayune (in an article with no attribution other than "East Jefferson Bureau"):

"The Army Corps of Engineers agreed Wednesday to place two-way radios in pump stations along the 17th Street Canal to coordinate drainage into the waterway when the corps closes its floodgate for a hurricane, Jefferson Parish President Aaron Broussard said.

The radios, Broussard said, will allow corps officials at the gate to instruct pump operators at three stations on shutting down their pumps. The goal is to keep water in the canal from overflowing into nearby neighborhoods.

The move came as a small compromise to Broussard's request that the corps take formal control of the pumping stations -- one owned by Jefferson and two by the New Orleans Sewerage & Water Board -- during a gate closure."

This was a massive hole in the communications plan that everyone (including the Times-Picayune) had ignored until now. I'm glad to see it plugged. It's too bad it took over a year, but I guess you take successes where you can get them.

The legal agreement which governs who operates PS#6 (as well as a bunch of other things about the floodgates) is called Supplemental Agreement 2A. It was signed by the Corps, the Orleans Levee District, the East Jefferson Levee District, Jefferson Parish, and the City of New Orleans in late December. It lays out responsibility for operation of PS#6 (which is owned by Orleans), giving it (of course) to the S&WB in paragraph VI:

"The Sewerage and Water Board shall ... operate and maintain approach channels, drainage structures, drainage ditches, or canals under the control or within the geographical jurisdiction of the Parish of Orleans"

Supplemental Agreement 2A was part of the third revision of the legal framework delineating the responsibilities of each governmental body involved with the floodgates. That framework is called a Cooperative Agreement (or "CA"). Thanks to a FOIA request, I got the original and the two subsequent Supplemental Agreements ("SA") posted to the IPET website:

Original CA (October, 2005, covers all three canals)
SA 1 (January, 2006, covers only London Ave. & Orleans Ave. canals)
SA 1A (December, 2006, covers only London Ave. & Orleans Ave. canals)
SA 2 (January, 2006, covers only 17th Street canal)
SA 2A (December, 2006, covers only 17th Street canal)

The agreement was split up in order to recognize that Jefferson Parish only has an interest in the 17th Street canal, not London or Orleans.

SA 1A and SA 2A both make revisions in the language to the January agreements which reflect all the changes to the PIR's during 2006. SA 2A also clarifies the language regarding operation of the drainage pumping stations. There are other changes that I don't quite understand because I'm not a lawyer.

Should Pres. Broussard continue to attempt to get the Corps into PS#6 (an extremely unlikely event), it is SA 2A which will have to be renegotiated. Given Orleans Parish's (which includes the S&WB) status as a co-equal party to the agreement, as well as the S&WB's strongly stated opposition to any relinquishment of control of PS#6 (and the I-10 station), this renegotiation has a possibility of success that can be termed beyond remote.

Internal investigation release delayed further?

2007-06-06T13:15:00.000-07:00

I recently spoke with a very well placed Corps of Engineers official in Vicksburg. The official told me the following:

- the release of the report on the internal investigation into the floodgate pumps may be delayed into next week, due partially to the fact that General Crear is out of the office all this week. However, it might also make it out this week.

- when it is released, it will be released in its entirety and to the public.

- the exact form of the release is has not yet been decided (i.e. will it go out on the internet, or will folks have to put in requests for it through Public Affairs, or what?). My personal feeling is they should just upload the entire thing to http://www.mvd.usace.army.mil and just let it go at that.

Fun with FOIA

2007-06-03T22:45:00.000-07:00

In advance of this week's release of the Corps' internal investigation into the problems with the floodgate pumps, I thought it would be helpful for the public to have as much information as possible.

Therefore, below you will find every shop inspection record I have received through Freedom of Information Act (FOIA) requests. The shop inspection records were created by members of the Corps' Jacksonville District during their Quality Assurance (QA) efforts at MWI's facilities in Deerfield Beach, FL and Vero Beach, FL between March and May of 2006. They are the most comprehensive record we have of what happened as the pumps and drive units were assembled and tested in MWI's facility.

Some of these records were appended by Maria Garzino to her May 4, 2006 memo. However, the rest of them have not been publicly released before now. I obtained them through a FOIA request of every Shop Inspection made by Jacksonville personnel during their time at MWI's facilities.

They are a treasure trove of information, especially the pictures, none of which have been seen publicly before now. For anyone who wishes to understand the pumps, these records are required reading.

My personal favorite page comes from the April 25, 2006 report:

The top picture shows the testing warehouse in Deerfield Beach, about a month after the fire. As you can see, there was hardly any building left (no roof!). MWI attached the bridge crane to the fire-damaged steel of the building.

The bottom picture shows what is supposed to be clear water. Its actually the oatmeal-like goop that resulted from the numerous hydraulic fluid leaks during testing that are documented in the reports. The fluid and the water emulsified into something that just looks gross, never mind that it makes one wonder about how effective it was to test the pumps putting that stuff through them.

By the way, the Corps "released" the April 25th report previously in draft form. It was attached to Ms. Garzino's memo. Here's how the picture of the oatmeal looked in that release:

This is why it's important to get everything possible, and to keep asking for it. Otherwise, the Corps New Orleans District just releases the bare minimum, and hopes people are satisfied. It's always a game with these folks.

The reports are below.

(New) March 8, 2006
(New) March 17, 2006
(New) March 21, 2006
(New) April 3, 2006
(New) April 8, 2006
(Previously released as draft, final version withheld by Corps) April 10, 2006
(Previously released as draft, final version withheld by Corps) April 11, 2006
(Previously released as draft, final version withheld by Corps) April 12, 2006
(Previously released as draft, final version withheld by Corps) April 13, 2006
(Previously released as draft, final version withheld by Corps) April 14, 2006
(Previously released as draft, final version withheld by Corps) April 15 & 17, 2006
(Previously released as draft, final version withheld by Corps) April 18, 2006
(Previously released as draft, final version withheld by Corps) April 19 & 20, 2006
(Previously released as draft, final version withheld by Corps) April 21, 2006
(Previously released as draft, final version withheld by Corps) April 22, 2006
(New) April 23, 2006
(New) April 24, 2006
(New, because it is the final version, with color pictures) April 25, 2006
(New, because it is the final version, with color pictures) April 26, 2006
(New, because it is the final version) April 27, 2006
(New) April 28, 2006
(New) April 29, 2006
(New) April 30, 2006
(New) May 1, 2006
(New) May 2, 2006
(New) May 3, 2006
(New) May 4, 2006
(New) May 5, 2006
(New) May 6, 2006
(New) May 8, 2006
(New) May 10, 2006
(New) May 11, 2006
(New) May 12, 2006
(New) May 13 through 17, 2006

Prepare for more PS#3 pump shutdowns

2007-06-01T11:45:00.000-07:00

[Updated 7/28/07, see below]

[For the latest on London Avenue canal testing, see my June 11, 2007 post, "Testing, testing." It has been updated quite often.]

I like when reporters take advantage of opportunities. It looks like the Times-Picayune's Mark Schleifstein did so by chatting up Colonel Bedey at yesterday's press conference for General Van Antwerp. To the Corps' disappointment, he apparently didn't ask, "Isn't General Van Antwerp the dreamiest?" It looks like what he did ask was, "What's the latest on the London Avenue canal testing?" Here's what he wrote:

"Bedey said the corps will delay until August a planned test of a 150-foot section of the London Avenue Canal wall to determine whether the water level there can be raised 1 foot. Another corps official said the delay resulted from high bids received for the testing contract."

Notice that it seems Schleifstein had to go to someone other than Bedey to find out the reason the testing is further delayed. Further, that other person seemingly didn't want their name used in print. It's a tiny but telling example of how the Corps struggles on a daily basis to keep the message upbeat and suppress bad or embarrassing news. It's ironic this appears in an article about General Van Antwerp's press conference, at which he hammered home his commitment to telling the truth to New Orleans no matter what.

Anyway, the New Orleans District blew their budget and their schedule (remember that this testing was supposed to be complete by now). And the testing and analysis will not be complete this hurricane season. I say that because even if they finish the testing on August 1, there's no way they and their contractors could get reliable results out of the field testing in a matter of days - it would likely take weeks of number-crunching.

So the upshot is that the London Avenue canal Safe Water Level will very likely stay at 4 feet through this hurricane season. That's just great. Whenever it comes up again, watch for the Corps to kind of toss it off like it's just a fact of life they can't do a thing about. In fact, if heaven and earth needed to be moved, they could drive sheet piles around the weak section(s) to seal it (them) off.

The quote above was follow-up to Schleifstein's article in the May 31st paper, which said,

"The corps also has struggled to strengthen walls along the London Avenue Canal enough to test whether higher water levels can be allowed in that canal during hurricane season. Delays in completion of that work could delay tests of the walls until late July, just before the height of hurricane activity in the Gulf of Mexico.

If successful, the safe water level in the canal could be raised to 5 feet, which means more storage of rainfall if the gate at the end of the canal must be closed during a tropical storm or hurricane."

I do have one complaint about the standard Corps line on this testing. They say they are doing it to determine if the safe water level can go up. It bothers me that they never mention the possibility that it could go down. It further bothers me that the local media doesn't call them on it, enabling the Corps to bamboozle the public.

Statements like that introduce legitimate worries of bias into how the Corps is doing things. Are they pre-judging the results of the testing before they're complete? Fortunately, the East Bank superlevee board is looking over the Corps' shoulders in an independent fashion on this one.

Finally, I have to issue a correction on this matter. On May 11th I took WWL-TV to task over a May 10th report that Lee Zurik filed about the London Avenue canal testing. At that time, I said WWL showed the wrong location for the testing. I was wrong. The testing is planned for an area on the east side of the canal just south of Robert E. Lee, not south of Mirabeau. I had been proceeding on a false assumption - based on conversations with Corps and other officials - that the weak section at Mirabeau would be the testing location. That was a mistake that I should have caught even earlier than now, because the location had already been mentioned twice, once on March 24, 2007 and again on April 20, 2007 by the Times-Picayune. The April 20th article gave the location as,

"The test calls for slowly increasing the level of water against a 150-foot-wide section of the floodwall on the canal's eastern bank, just south of Burbank Street on Warrington Drive."

Burbank Drive is just south of Robert E. Lee, not Mirabeau. You can see the approximate location here.

I regret the error.

[Update 7/28/07]
See my June 11, 2007 post, "Testing, testing," for the most up to date information on the London Avenue canal testing. The Engineering News-Record articles linked there are far better than the Times-Picayune's coverage.
[end update]

How about briefing the citizens and taxpayers?

2007-05-31T13:24:00.001-07:00

[Updated July 11, 2007. See below]
[Updated June 9, 2007. See below]
[Updated June 1, 2007. See below.]

From the May 31, 2007 Times-Picayune, in an article regarding General Van Antwerp's visit to New Orleans:

"...The general will be briefed on the problems the corps experienced installing pumps at three temporary gates built at the ends of the 17th Street, Orleans and London avenue canals."

I assume this is in conjunction with the internal review of the pumps, which has been proceeding for nine months. Do we seriously think the general is going to receive the same patronizing BS that the public gets, like this May 24, 2007 gem from Colonel Bedey (talking about a pump going down during Orleans Avenue pump testing)?

"I equate it to an alternator going out on a car. You don't replace the car, you replace the alternator"

Thanks for fully explaining the problem to us. It's really inspiring that you think that none of the citizens are intelligent enough to absorb a complete explanation. You couldn't put out a press release the next day, or even in the week since then? Do we even know if the pump has been fixed? Do we know what the problem was? Do we know if the same problem can be found on the other 39 pumps? "No" to all four questions. It's typical of Colonel Bedey and his band of Merry Mendacity-Makers to not level with the public. Happy talk and collegial back-slapping has grown very tiresome. We deserve unvarnished, unspun facts. Apparently, Van Antwerp is far more likely to receive them than the people who actually need them. This is not openness.

Update, 6/9/07:
By the way, the Times-Picayune found out it wasn't a matter of a simple "alternator." Buried at the very end of their article about the release of the Corps internal investigation was this howler:

"During a news conference on the report Friday, Bedey also confirmed that one of the new pumps was replaced May 17 after a chunk of concrete was sucked into it, causing severe damage. He said the replacement took two days."

A chunk of concrete!

This also brings up the interesting question: where did they get a 60" pump in just two days? They didn't buy any spares.
[end update]

The citizens of New Orleans, who have to live with this stuff, should get the exact same briefing that a general who has been on the job for less than two weeks - and who does not have to live behind those gates - should get. Frankly, New Orleans should get that briefing before the general, even if it might embarrass the general. And local and national reporters should be crawling all over every office of the Corps (New Orleans, Vicksburg, or HQ in Washington) that might have access to the briefing and the internal review report in order to get them to the public. We deserve it.

General Van Antwerp's predecessor, General Strock, told Senator Vitter the Corps internal panel would get their work done by the end of May. Well, today's May 31. If there are recommendations and action items in there, wouldn't it be important to the citizens to know them, so they can make informed decisions about the risk they face this hurricane season? That season starts tomorrow.

By the way, it is very likely that there are recommendations and action items in the Corps internal report. The Corps doesn't investigate for the sake of investigating. They are a militarily-goverened organization, and as such, are action-oriented (at least in theory). They would not spend nine months on an investigation, and then say "Everything's fine!" At least I hope not. My instinct is that there are things in the internal report that show there is still work to be done on the MWI pumps, despite all the puffery from Col. Bedey on the local news.

Update, 6/1/07:
At the very end of the Times-Pic's coverage of Van Antwerp's 17th Street canal press conference is a tiny mention of the internal report:

"Van Antwerp also said an independent report on the problems experienced by the corps in installing pumps at the ends of the drainage canals should be released next week. The report was requested by [Corps Mississippi Valley Division Commander Brigadier General Robert] Crear after inspectors raised questions about the ability of the pumps to operate."

Separately, I have heard that Crear is writing/editing his preface to the report, and that is part of the holdup.

I did notice that Van Antwerp came out smelling like roses (mostly) in the TV coverage yesterday and in the print coverage today, so I guess that was "mission accomplished" for the Corps' Public Affairs operation. Though I must give the T-P props for calling him to task for a bit of cluelessness:

"Van Antwerp admitted he knew nothing about the long-delayed Hurricane Protection Decision Chronology being conducted under the auspices of the U.S. Defense Department.

The study was designed 'to enable the corps and the nation to fully understand the long history of federal, state and local decisions that led to the design and construction of the New Orleans-area flood and storm damage reduction system,' Maj. Gen. Don Riley, corps director of civil works, said when the study was announced in June 2006 as a companion to the nine-volume forensic report on the levee failures during Hurricane Katrina.

Corps officials and members of the independent team preparing the report have refused to say when it will be released to the public. The lack of such a report on the breaches -- detailing policy and political failures that accompanied engineering and scientific failures -- prompted criticism last year from the American Society of Civil Engineers."

In order to promote accountability later on (in case stuff goes to hell), here's a wrapup of Van Antwerp's comments during his visit:

Times Picayune: June 1
AP: New corps chief says he's personally committed to rebuilding New Orleans
WWL-TV (video only, report from 5 PM news): New head of Corps of Engineers in town
ABC-26: New Corps chief tours area
WDSU-6 (video only, report from 5 PM news): Corps Chief Promises to Tell Truth, Good or Bad
Baton Rouge Advocate: Storm risk report to be very detailed

[Update, July 11, 2007]
The Corps released the Katrina decision chronology today. The Times-Picayune did a big article that carried the Corps water quite adequately. Despite actually naming names within the article, the headline and the lede still say that blame can't be assigned to individuals within the New Orleans District. This is crap. Real people signed those drawings and wrote those reports and advocated those positions that led to Katrina. Some of those people are still working in the New Orleans District. So much for accountability.

Hopefully, all of that will come out in the civil suits.

GAO report released

2007-05-29T17:30:00.000-07:00

Updated 7/2/07. See below.

The GAO has released their report on procurement of the 34 floodgate pumps from MWI. It is available here (or here). The official title is "U.S. Army Corps of Engineers' Procurement of Pumping Systems for the New Orleans Drainage Canals" and the GAO's number is GAO-07-908R.

This is not the final version of the report, but rather the briefing slides presented to Senator Landrieu's office on May 17, 2007. A so-called "blue cover" (i.e. full-blown) report will eventually be issued.

I'm reserving judgement on the GAO's work until the results of the Corps' independent review are (hopefully) made public. In his May 2, 2007 letter to Senator Vitter, now-retired Lieutenant General Strock (Corps Commander) mentioned that the independent review was due to Brigadier General Crear (head of the Mississippi Valley Division, which is one layer above the New Orleans District) by the end of May, which is this Thursday.

Update
You can read my thoughts regarding GAO's work in my July 2nd post here.

I'm a weenie

2007-05-29T17:25:00.000-07:00

After a fiasco in the comments to this post a couple of months ago, I got a lot more selective in my moderation of comments. Some people were writing some really scurrilous dirt, and I cut it off. I realized that I am under no obligation to give individuals a forum to sling personal insults or to make vague accusations with nothing to back them up. Also, I am not interested in injecting politics into what I write. In fact, I'm not interested in injecting anything personal into what I write. Finally, I'm interested in comments that are written well, without lots of SCREAMING or exclamation points.

Despite all this, and the fact that "Comment moderation has been enabled" is written on every comment screen attached to every post (and has been since I started blogging last September), some people persist in the belief they have an absolute right to see everything they write published on someone else's website. That's not how the real world works.

I should make clear that I don't discriminate against anyone in particular when moderating comments. In reality, unless a commenter identifies themself in the body of the comment, I really have no idea who they are. Most people do not attach their names to their comments in the body of the text; a few who have webpages fill in the webpage field. That's fine. There's no obligation to identify oneself when criticizing another person, though I believe it is the polite thing to do if one is interested in dialogue. From a practical standpoint, it also helps to distinguish oneself from all the other comments which have "Anonymous" as their author.

To those of you whose comments I've deleted, I'm sorry. If you feel it impugns my credibility not to publish everything that comes across my desk, you are entitled to that view. I read every comment and judge it on its own merits before I pass it through to the great, wide world.

However, as I said in the comments to that March post, this is my sandbox. Like every other blogger, I reserve the right to publish whatever I like. You are free to agree or disagree. I view the comments fields as more of a "Letter to the editor" forum (call me a Luddite for still reading newspapers). Newspaper editors do not publish every letter they receive, nor do I publish every comment I receive. That's what comment moderation is about, and it shouldn't really be a surprise. I've had moderation turned on from my very first post. There are more than enough other forums on the internet where people can complain about anything under the sun with impunity. Heck, starting a blog literally takes five minutes.

If anyone has a deep problem with what I'm writing, and wants to really engage in a rational discourse, they're free to email me. My address is in my profile, which is linked at the upper right of this page. Due to a mistake on my part, for a very long time, my address wasn't there, but I've since fixed that oversight.

In an exception to my policy of not posting stuff that's personally insulting, loaded with politics, and not at all on topic, below you will find the latest comment I received. According to the timestamp, it came at 8:55 this morning. It's attached to my post about the flowmeters. It is typical of stuff I've avoided posting, though I'm sure someone will have archived all their comments and will post them somewhere to show what a horrible person I am. Judge for yourself whether this is on topic and germane to a rational discussion. There was no name attached.

MATT -

You are such a little weenie. I just came back to your blog and you deleted all of my critical comments.

You don't really want to post the peer-reviewed REALITY here do you?

I can't stand the Bush mobsters, but I REALLY despise dishonest engineers like you that post their own opinion, declare it to be absolutely gospel science, and then remove/dimiss all respondent commentary!

YOU are as bad as the BUSH PEOPLE!!

Flow

2007-05-22T08:27:00.000-07:00

UPDATE, 6/1/07: Make sure you take a look at the comments to this post. There's more information there.

Each pump at the floodgates has a flowmeter on it which measures the flowrate of water through the pump units. The meters are ultrasonic units manufactured by Endress+Hauser. Specifically, they are model Prosonic 93W meters. What that means is that the transmitter (the box with all the electronic guts and the display) is a Prosonic 93. The "W" refers to the type of sensor. Endress+Hauser's flowmeter page is here. At that link, the specific page for the Prosonic 93W can be found by clicking on the box corresponding to "Ultrasonic Flowmeters" and "Conductive Liquids". Then, along the left hand side, you will find a menu entry for "Prosonic Flow 90W, 93W." Here's a picture taken from the Prosonic 93W page:

The way these things work is pretty simple. One of the sensors (the two little cylindrical things on the left in the picture, and which are about the size of a small can of tomato paste) is clamped to the outside of the pipe. It sends an ultrasonic waveform out through the pipe and the water flowing through the pipe. That signal is picked up by the other sensor, clamped to the outside of the pipe 180 degrees opposite to the first sensor. The other sensor sends the signal to the transmitter box (the blue and white box in the picture above, which is actually a model 90 transmitter - the model 93 has a bigger display and more plugs on the bottom) for processing. Depending upon how the waveform has changed as it has traveled through the water, an average velocity of the water can be calculated. If one wants a more accurate measurement, a second set of sensors can be added and the two signals can be averaged. The floodgates installations use just such a two-channel arrangement. After averaging, the transmitter can then send the flow data to a controls system for display elsewhere. The results are also displayed locally on the transmitter's screen.

It only makes sense that one would want as steady a flow as possible for measurement. Turbulence would scramble the signal, increasing the error in measurement. For this reason, the manufacturer recommends installation of the sensors a specified distance away from anything that could disrupt flow, i.e. introduce turbulence. Such things could be elbows, pumps, valves or pretty much anything other than straight pipe.

The distance is measured in multiples of diameters of the pipe. In the Prosonic 93 installation literature, which can be downloaded from this link, the specified distance downstream from a pump is called out as 20 times the pipe diameter. Here's the applicable picture from page 30 of the manufacturer's manual (which I have annotated in red):

The diameter of the pipe for each individual pump is 60". Those 60" lines - in groups of two or three - flow into 108" (9 foot) pipes that go straight out to the lake.

In the case of the flowmeters on each of the pumps, the sensors are located approximately 1 (one) diameter downstream from the pump. They are mounted pretty much right on top of the pump. Below are a few pictures to show the location of the sensors and transmitters.

Here's a picture of a removed discharge elbow at the Orleans Avenue site. These elbows fasten to the tops of the pumps once the pumps are in their frameworks out in the canal. I've annotated a few details on it. Inside the white box is the location for the sensors.

Below is an enlargement of the area inside the white box above. I've noted the locations for the sensors. As you can see from the photo above and the enlargement, they are only about four or five feet above the flange that bolts to the pump.

In the picture below I've noted the locations of the Prosonic 93 transmitters on the pump platforms.

Below is an enlargement of the transmitter on the left.

The picture below shows the Orleans Ave pumps on the east bank. I've highlighted the area which is enlarged in the next photo.

The picture below shows the highlighted area enlarged from above. As you can see, the flow sensor is just above the discharge of the pump.

In the photo below (an enlargement from another photo) you can see the orange straps on the adjacent pump which hold the sensors in place.

The photo below is taken from a short version of May 25, 2007 AP story about the testing of the Orleans Avenue pumps the previous day. The short version appeared in the Miami Herald, and had three pictures attached to it. The full version is here, sans photos.

On the picture, I've highlighted the bands that hold the sensors with red arrows. Inside the white box, one can see one of the sensors placed at exactly the location it was before the refitting of the pump units with hard pipe and new hydraulic motors.

And in the picture below, a screen capture from a May 15, 2007 WWL-TV report about pump testing at Orleans, you can see that the flow meters are definitely in use. The gentleman on the right appears to be a contractor from Prime Controls, according to the "PRIME" sticker on his hardhat. Prime is the contractor responsible for the entire SCADA system. I'm not sure what agency or company the gentleman on the left is from. But as you can see, they're both checking the flowmeter.

There's two problems related to the flowmeters here, one that may be solvable, and one that needs to have questions asked about it.

The first problem is this... in this installation, the flowmeters are pretty much in the worst possible spot to measure flow because of the highly turbulent nature of the flow immediately downstream of the pump discharges. As such, there is a large error being introduced into the flow measurement. In a perfect installation, the Prosonic 93 with W sensors has about a 0.5% error, which is pretty good. In this installation, which is far from perfect, the error could be significant. I can only guess, but I'm sure the Corps guys running the floodgates project know (at least I hope they do).

Is there a way to solve this? Maybe. There aren't many good spots on the individual pump discharges on which to place these sensors, due to the numerous sources of turbulence. For example, at 17th Street, all of the pumps shoot the water up into elbows, which are themselves bolted to elbows, which are welded to the 9 foot pipes. Thus, the individual discharges for each pump are nothing but sources of turbulence. It would be the worst place in the site to put flowmeters, but that's where they are. Here's a picture of 17th Street from last fall, showing the pump discharges:

Given a choice - and they are admittedly few - the best place to look for a location for the sensors would be on the straightest piece of pipe possible, as far away from sources of turbulence like the pump discharges or the tees where the 5 foot individual pump lines drop into the 9 foot pipes. The best place, given those parameters, would be on the 9 foot pipes on the lake side of the gates. According to the manufacturer's literature, the "W" type sensors can be placed on a 9 foot pipe and still work just as well. One would not have accurate flowmeter readings from individual pumps, but the existing flowmeters could still be used strictly for indication of flow ("It's flowing, but we don't know if the rate on the display is actually true."). However, one would have a better feel for the overall flow for the set of two or three pumps going into each of those nine foot pipes. Here's a picture of the lakeside location at 17th Street, taken from NPR's May 18th article about what's going on with the pumps:

Here's an overhead shot of 17th Street taken from a May 4, 2007 WWL-TV report, which shows the substantial length of straight pipe available:

The choice at Orleans isn't so great, since the straight section of the 9 foot pipes aren't nearly as long as at 17th Street (picture also taken from the May 18, 2007 NPR article):

As you can see, there are no sensors currently strapped to these locations.

London is about the same as Orleans, since it was designed by the same firm - URS (screen capture from April 30, 2007 WWL-TV report, sorry about the quality):

A closer view of the 9 foot pipes can be found in this USACE photo, previously shown in my October 21, 2006 post:

The second problem is more logistical in nature, but no less severe. One might think that the measurement of the flow is merely an academic matter. It's nice to know, but don't we just want to turn the pumps to "11" and let 'em rip? Well normally, yes. But remember that there's not enough pumps at the lakefront to match the what can be put into the canals on 17th Street and London Avenue. In addition, the Safe Water Levels must not be exceeded (By the way, that's going to be real tricky on London Avenue, considering that the pumps were specified to run at around 4 feet above mean lake level, which is also the safe water level on the canal. Modifications to the discharges of the 9 foot pipes have supposedly allowed the pumps to run at a lower height - perhaps 2 or 3 feet, but who knows if that's actually true?).

When one takes both these circumstances - too few floodgate pumps and the Safe Water Levels - into account, it becomes clear that there is a delicate balancing act to be peformed between the New Orleans S&WB pump operators in stations 6 (17th St), 7 (Orleans), 3 and 4 (London Avenue) and the Corps pump operators at each of the gate structures. The way that that balancing act is performed is by relying - in part - upon the readings from the flowmeters, which are piped through the SCADA system to the computer terminals in each pump station (except pump station 4 on London - I assume messages to the operators in that station will be relayed via the folks in station 3).

The Floodgates Operating Manual recognizes this. Here's the section titled "Step 9 (T - 6 hours)" (sorry, but the Corps isn't big on paragraph breaks. This is exactly as it appears in the manual):

"Conditions will rapidly deteriorate in the final six hours before landfall. Activities at the structure should be completed and all team members should be in safe and secure positions. The H&H Team will continue to monitor the storm and the predictions. The Canal Captains will need to closely monitor the canal levels since rainfall will likely begin to increase, leading to increased pumping at the canal pump stations. Through the working relationships established in the Water Control Plan, the Canal Captains will primarily be observers at the Local Partner pump stations but if the canal level rises to one foot below the Safe Canal Levels they will notify the pump operators and continue to communicate with the operators to verify they are adjusting pumping accordingly. If canal levels rise to the Safe Canal Levels then the Canal Captains will notify the pump operators and continue to communicate with them to ensure they adjust pumping to not exceed the Safe Canal Levels. The Canal Captains will remain in contact with the Operations Chief throughout the storm and inform the Operations Chief when the pump station pumps have either been throttled back or shut down. The procedure will reverse when canal levels lower and the Canal Captains will inform the pump station operators when canal levels are below Safe Canal Levels so they can adjust pumping accordingly. For steady flow conditions, the targeted level for the canals will be in the range from 3 to 4 ft. (NAVD88 Datum) this will provide desired suction head to maximize flow for Interim Control Structure Pumps while not approaching the Safe Canal Levels. Flow meters will provide discharge flow rate data for each Interim Canal Structure Pump. Canal Captains will coordinate with Local Partner Pump Stations to achieve steady flow under these conditions."

A few notes are necessary. First, let me explain the numbers on the "targeted level" in the above paragraph.

The only copy of the manual that has been released is from July of last year. At that time, the Safe Water Level was 5 feet at both 17th Street and London Avenue. Since then, the SWL has changed to 6 feet at 17th Street and 4 feet at London Avenue. Elsewhere in the manual, levels one foot below the safe water level are noted as the activation depths for the floodgate pumps. Thus for London Avenue, the steady flow operating range is probably 2 to 3 feet now, not 3 to 4 feet. And for 17th Street, the range might be 4 to 5 feet, or it might have stayed the same.

Also, the Canal Captains are the Corps Operations representatives in the S&WB pump stations. They'll be huddled over the readouts from the SCADA system, checking the canal depths from the level gauges and the flowmeter readings from the floodgate pumps. As one can see, the flowmeter readings will be part of the data considered by Corps personnel in their decisions to advise S&WB personnel to turn off the city's pumps.

For storms below category 3, Corps personnel are supposed to remain at the floodgates. Supposedly, they'll button themselves up in the little bunkers on the drive platforms, but considering the courage it takes to volunteer for such duty, I seriously doubt they will stay inside if there's a problem or if a request comes over the radio to check a flowmeter out on the pump platforms. Thus, there will likely be a backup for the flow signal received in the S&WB pump stations in the form of Corps Operations person who can look at the local display at the gates, as well as look at the pumps themselves.

Problems arise, however, when the storm is a category 3 or above. In that case, the operators are supposed to evacuate the floodgates. Now, I think it's questionable that they will do so. After all, these guys are probably pretty dedicated, and would most likely have to be escorted off the drive platforms at the point of a gun. I would compare that to the situation exemplified by the S&WB drainage pump operators who stayed during Katrina.

However, sometimes orders are orders, and you must leave. That situation is demonstrated by the evacuation of Jefferson Parish's pump operators during Katrina. With the operators gone, and with no automated system to control or adjust the pumps, the lake backflowed through the JP pump stations into most of the east bank of Jefferson Parish, a totally manmade disaster.

So for truly nasty storms (cat 3 and above), there might be no crews out at the floodgates. In addition (and this is according to the manual) for category 4 or 5 the Corps also pulls the Canal Captains out of the S&WB stations and relocates them to the Corps headquarters on Leake Ave at the river. The S&WB determines whether to leave the S&WB operators in the S&WB stations.

What this means is that all the data to determine whether to turn off more city pumps than will already be turned off (and they will be turned off after the gates drop; otherwise, they'd overwhelm the Corps pumps) will be received via the SCADA system. This includes the data from the flowmeters, which is most likely inaccurate, possibly greatly so.

So two questions need to be asked:
1) Assuming that the use of ultrasonic flowmeters was the correct choice in this application, does the Corps know how accurate the data from those flowmeters actually is? That is, has anyone done detailed baseline tests to determine whether there really is an error due to the current installation position of the pump flowmeters?
2) Does the Corps plan on running drills in which there are no operators interacting with the machinery at the floodgates, in order to determine whether there are kinks which need to be worked out? After all, their plans call for the floodgates to be evacuated in heavy storms.

Troubles with the trench again

2007-05-21T11:15:00.000-07:00

The following is unconfirmed, but it's from a pretty damn good source.

During the Corps' gate closure drill on May 15th, 2007 which took place at all three floodgate sites, apparently they had some trouble closing the gates at 17th Street. Why? There was construction debris in the trenches into which the gates seat. Apparently, at least one and possibly more gates stopped about a foot short of actual closure.

There's lot's more information about the trenches at this earlier post.

Problems

2007-05-21T08:52:00.000-07:00

With hurricane season approaching, I thought I would summarize all the problems that I've discovered with the floodgates pumps over the past few months. This list is somewhat scary.

1) Only one type of backflow prevention, instead of two as called for in Corps engineering manual for this type of installation. [blogged September 29, 2006]

2) Durst gearboxes undersized for anticipated thermal load, increasing potentiality of overheating. [blogged March 8, 2007]

3) Hydraulic reservoirs undersized by factor of three, increasing potentiality of overheating. [blogged May 11, 2007]

4) No electrical backup for diesel engines, despite availability from chosen vendor, decreasing reliability. [blogged May 11, 2007]

5) Undersized hydraulic fluid transmission lines, increasing potentiality for catastrophic failure. [blogged April 17, 2007]

6) Unpainted, rusting hydraulic fluid transmission lines, increasing potentiality for catastrophic failure. [blogged April 5, 2007]

7) Rusty hose fittings on pump units. Note that this one has supposedly been addressed by the replacement of most of the hoses and fittings at the pump units with hard piping. [blogged April 5, 2007]

8) Flowmeter sensors in locations which will amplify measurement errors [blogged May 22, 2007]

There are others that I just haven't gotten to as yet.

GAO, testing, and real problems

2007-05-18T10:00:00.000-07:00

Yes, the GAO has presented their findings to Senator Landrieu's office. I'll have more to say about this later. However, I urgently need to zero in on the high-profile testing the Corps has been doing after refitting of the pump unit hydraulic motors. They are also replacing much (not all) of the hosing at the pump units with hard pipe. They are continuing to act in the exact bad-faith manner called out in the GAO report.

This is from the press release about the GAO report found on Senator Landrieu's website:

"By June 1, the GAO told Sen. Landrieu, the Corps plans to have completed reinstallation of 40 pump systems that have each been tested for 45 minutes to two hours."

Two hours?!? 45 minutes?!? Are they serious?

The most recent test, at Orleans Avenue on Tuesday, May 15, 2007, only went for less than an hour. Why is the media, our government officials - hell, anyone else but me - not noticing the tremendous disconnect between these dog-and-pony shows the Corps is putting on and what is really needed to determine if the pumps work?

These pumps and drive units may need to run for somewhere between 12 and 24 hours during a storm. Remember that the original factory testing called for full performance testing. Then that was downgraded to a five hour endurance-and-reliability (E/R) test for the drive units only, with no performance testing for the pumps themselves. Then that was downgraded to a three hour E/R test. And by the end of the testing (according to the complete batch of inspection reports I got via FOIA recently) , some drive units were being sent to the field after just 45 minutes of testing. Only a tiny handful of the pump units were verified as meeting the specifications for flow and head.

Shockingly, one of the findings in the GAO investigation is that the Corps has been making up testing requirements as they go along. It's shocking because it seems they continue to do so to this very day, and are actually using the latest batch of tests as a defense that they're now doing the right thing. It's up-is-down thinking, and so brazen in its attempt to flim-flam the public and policymakers that I can't believe it. Why are these systems not being run for legitimate amounts of time? How can we be sure that they will work as designed if they haven't been tested under the conditions they are likely to see?

I understand that due to canal levels, it's impossible to run the entire systems (i.e. five or six pumps at a time) simultaneously for an extended period. However, there is no excuse - NO EXCUSE - for not running each individual drive-and-pump combination for a real world length of time. The Corps is continuing - to this day - to leave us in jeopardy of pump failure.

The walls

2007-05-16T10:53:00.000-07:00

I've been thinking a lot about the big picture, and how we've gotten to the atrocious situation we're now in for flood protection. We now have a set of conditions which are more likely to cause flooding in New Orleans than before Katrina. How did this happen?

One part is the Corps' New Orleans District's moronic decision - yes, decision - to not fix the outfall canal walls, instead putting up the floodgates. The walls were supposed to protect against storm surge, and they were only put up within the last fifteen years. Why could they not be made to work now? It's hardly rocket science. After all, since Katrina the Corps has reinforced or improved walls and levees along the Duncan Canal in Kenner, along the Gulf Intracoastal Waterway by the NASA-Michoud site, and at plenty of other places around the region. Why were the walls along the outfall canals any different?

I'm really not sure of the "why." But I am sure of where we are: no wall improvements in the near future, and possibly ever. This has deep consequences, some of which we've already seen (pumps at PS#3 getting turned off twice in the last five months for one). Nearly everything that is a problem with post-Katrina drainage and flood protection in New Orleans (on the east bank, west of the industrial canal) can be traced back to the Corps' foolish, foolish pair of decisions to go with the floodgates, and to not repair the walls. This post talks about the consequences of, and possible reasoning behind, those decisions, and where we are as a result.

Let's start with evidence backing up my premise that the walls won't be repaired...

Evidence the walls will not be touched
Of course, the plain evidence (which the Corps always tells us to ignore, but which my gut tells me is right) is right there. There has been no work on the outfall canal walls outside of the breached and heavily damaged areas since the storm. Whenever I tell this to someone from outside the region, they seem amazed.

There was also this April 25, 2007 story from WWL-TV, in which Corps Protection and Restoration Office Chief Tom Podany is quoted as telling the Jefferson Parish council that the west walls of the 17th Street canal "meet all the requirements we have." Remarkably (or perhaps not, when you consider the sub-par caliber of their recent Corps coverage), the Times-Picayune devoted nothing to this story until Senator Vitter let the Corps know such statements are way out of bounds, since the canal walls are part of the Lake Pontchartrain and Vicinity project, and are thus required to work as designed, no matter what malarkey the Corps builds at the lakefront. His letter can be found here, and the Times-Picayune's story is here.

Then there's the other story from the April 13, 2007 Times-Picayune about the west walls on the 17th Street canal, in which the Corps "revealed" that the sheet piles on the west side are very short, but still expressed strong confidence in their floodworthiness. I put revealed in quotes, because that particular article is so poorly composed, and full of such inconsistencies, that I am convinced it should have never been written in the first place. It will be subject of a future post about what happens when bad stories happen to worse writers. But it's still helpful in revealing the Corps' feelings about the walls, which are that the walls are just fine. The April 13th article also mentioned the delivery of the Corps' study of the walls' stability by the end of April. We're still waiting. Finally that article mentioned finishing emergency repairs on the east side of the 17th Street canal by June 1. Those repairs were first announced April 3, 2007, in another Times-Picayune article which included the usual head-patting from the Corps.

That work just started in the last few days, but my reaction is: Huh? I thought the walls were fine. That's what we've been told for over a year. Another reaction is: why did it take over 21 months, if the job can apparently be accomplished in two weeks (I sincerely doubt this is a comprehensive fix, especially since the area covered is so small), and what does that say about the condition of the walls in those spots up until now? It seems the Corps is talking out both sides of its mouth, which certainly doesn't breed trust. At the very least, there's been a serious lack of urgency. Why wasn't this work done before last hurricane season?

[Update, 7/25/07]
As far as the present condition of the walls along the 17th Street canal, we can now know a bit more about that.

On May 24, 2007, the Times-Picayune reported on the release of a Corps study of the 17th Street canal. The study looked at how stable the walls are under various pumping and lake stage scenarios. They looked at the envelope between practically no flow to maximum output from all three pump stations (PS#6, I-10, and Jefferson Parish's tiny Canal St. station). They also looked at the envelope of the lake level from 0 feet to 5 feet, in one foot increments.

They found that the six foot safe water level can be justified, though the water rises dangerously close to it in certain regions of the canal under many circumstances. Those regions (near I-10 and Veterans) are the ones that got the emergency repairs mentioned in the April 3, 2007 article.

Well, I have received a large chunk of that report, and now everyone can read it for themselves. You'll find the first 58 pages of the report here. The Appendix, which contains the raw data, a May 2006 inspection report, a fascinating November 2005 Linfield Hunter & Junius internal memo, and more good stuff is here (warning: over 80 megs). The rest, which includes the actual analysis, should be forthcoming soon.

There's lots to take away from the report. Here's a few of my impressions.

1) The sections of the east wall immediately adjacent to the breach are among the weakest in the whole canal.

2) The Corps' method of calculation for all the governing Safe Water Levels (the Method of Planes) was superceded by Corps HQ shortly before the study was issued by something called the Spencer Method. They say they will go back and redo the calc's using the newer authorized method, but there's no indication whether it would give different results. I'm no geotechnical engineer, so I have no idea whether this is significant or not.

3) There's no correlation between the various "reaches" into which the canal walls were broken for calculation purposes (there were about 16 on each side) and the placement of the depth sensors which are tied into the SCADA system (I think there's only five or six of them). I guess the Corps operations folks just watch the particular sensor nearest the weakest spots.

4) With the lake at five feet, the water gets above or gets within six inches of the governing safe water level near I-10 for total flows exceeding 7500 cfs. This means if the lake level gets close to five feet and the gates haven't been dropped, PS#6 will have to cut back on three of their big pumps. Of course, that's kind of moot, since there's not going to be more than 7500 cfs available with the gates down anyway (and most likely a lot less, due to failures of the MWI pumps).

With lake stages lower than five feet, it appears there is no significant need for pumping cutbacks at PS#6. However, the trigger for gate closure is six feet in the lake. And also, as noted above, these calculations are based on a now-outdated method. In addition, they rely on the Corps' ridiculously low safety factors. Combine all that, and there's more than enough uncertainty to make one believe that pumps will be cut back in extreme cases, whether the gates are up or down.

Overall, however, the report appears to present mostly good news. However, it's important to note that there was no outside review of this report before its compilation, and there were no outside resources utilized by the Corps. This is unfortunately still the norm.

However, now that the report is public, researchers and scientists outside the Corps can now critique it.

[end update]

So it's pretty clear that the Corps New Orleans District, which has had 21 months to do something with the walls, has chosen to do nothing (excepting some minor patching). Why? According to them, it's because the floodgates are there, and they "take the walls out of the equation."

Out of the equation?
When asked about the walls, the Corps says, "Don't worry, the floodgates take the walls out of the equation." Podany said it in that Jefferson Parish Council meeting:

"'Because we have that temporary gate at the lake, those walls don’t need to function during a storm,' said Podany."

They said it in the April 13th T-P article:

"'We've done the analyses, and the results convince us that underseepage wouldn't cause the walls to fail anywhere in the 17th Street Canal because of the steps we're taking, which includes limiting the amount of water in the canal,' said Walter Baumy, chief engineer of the corps' New Orleans district."

They said the same thing as recently as May 9 in the Times-Picayune, at the end of the long overdue article about everything the Corps has done wrong since Katrina:

"Despite the weak spots, Bedey said, the corps has been successful in dramatically reducing the risk from storm surge in a number of locations, compared with before Katrina. That includes the installation of gates on the 17th Street and London and Orleans avenue canals, which will eliminate the threat to flood walls along the canals. Problem pumps designed to move rainwater in the canals over the gates into Lake Pontchartrain have been repaired and successfully tested at the 17th Street and London Avenue canals, Bedey said, and are expected to be repaired and tested at the Orleans Avenue canal by June 1."

That's nuts. All these statements rest on the giant, unstated assumption that the gates and floodgate pumps will work as designed. To no one's surprise, I have serious doubts about that assumption, and so do many within the Corps. As I pointed out in my last post, if the gates don't seat properly in the trenches, they'll probably fail. If the gates' hydraulic winching mechanism doesn't work, and the winds are too high for a crane (i.e. over 30 mph), they won't close. There are plenty of ways for this system to fail. We also know that the MWI pumps and drive units have deep problems.

In addition, the walls see water all the time from normal rainstorms when the gates aren't down. The gates most definitely do not take the walls out of the equation, unless the equation only counts tropical storms in which the gates don't fail (more about gate failure below). Otherwise, the walls are very much in the equation. This knocks out a central premise in the construction of the floodgates, forcing the Corps to retreat...

The Corps has a "system" to save the walls, and drown us
When the fallacy of the "walls are out of the equation" is pointed out, then the Corps says, "Don't worry, we have a system to keep the walls from seeing too much water during a rainstorm." In point of fact, they have no such system. They have a system for monitoring the water level in the outfall canals during a rainstorm, which is an entirely different kettle of fish. That system is a set of electronic level gauges installed along each canal which is connected to a few computer terminals scattered around the city, including within the three pump stations at the southern end of each outfall canal. Surprisingly, the T-P actually described the system correctly in their sloppy wet kiss to the Corps on March 25, 2007 (keep in mind, the following describes what happens with the gates closed, not during a non-tropical system):

"...The canals themselves have been lined with a Supervisory Control and Data Acquisition, or SCADA, system that uses sensors to transmit a wealth of information via computer screens, including changes in water levels along the length of all canals.

It is the SCADA [system screens] inside the pump stations that canal captains will monitor as they synchronize the pumps near the gates and the bigger city pumps situated hundreds of yards -- sometimes a mile or more -- closer to the heart of the city.

With the gates closed against a surge, city operators must power down their stronger pumps to keep from overwhelming the lower capacity of the dozens of pumps set up at the mouths of the outfall canals. It is the pump operators who will make adjustments in response to information the canal captains collect from SCADA readings, Wagenaar said.

So critical is SCADA to protection of the canals that there are special backup generators in the pump houses just to keep the data system running. In addition, fiber optic cables backed up by a microwave system have been installed to keep SCADA data flowing as long as possible to pump stations and to the corps' riverfront headquarters and emergency bunker."

What's notable is that - other than this article - it is rare that the distinction between monitoring and control is actually made, especially when the Corps' New Orleans District is quoted directly. It is an important distinction, and its omission is typical of the self-aggrandizing puffery the N.O. District has adopted as an unbreakable habit. The Corps doesn't control anything outside the gates, but they act like they do.

During any rainstorm, including non-tropical ones, if the water level gets too high in any of the outfall canals as it is being pumped out of the city (that is, above the Safe Water Level - or SWL - for each canal: eight feet at Orleans Avenue, six feet at 17th Street, or a measly four feet at London Avenue), then there are only two solutions to avoid another breach:

1) Lower the level of water the city's pumps must push against from the lake. That is, the Corps shuts the floodgates and turns on the floodgate pumps. God help us.
2) Limit the amount of water going in the canals. That is, the Corps tells the Sewerage & Water Board, a local agency over which they have no official jurisdiction, to turn off the city's drainage pumps for the specific canal for which the SWL has been exceeded. This has already happened twice, on December 30, 2006 and May 4, 2007.

That's the "system." Force the city to drown itself to save the walls. Talk about bass-ackwards. So the unfixed walls are not only a problem not remedied by the floodgates, their unrepaired state is actually driving decisions on flood protection. The walls still loom very large.

What if the gates fail?
In choosing to spend over $200 million in construction, repairing, changing, and expansion of the floodgates and pumps, the Corps deliberately left fixing the walls in some fuzzy future. They cast their entire lot with the floodgates, as the quotes above show. But what if the gates don't work?

Such an idea - despite all the evidence that the New Orleans District and its mostly locally-based consultants can't be trusted to design a block of wood - still seems anathema to almost everyone. But think about it. This same crew was trusted for 40 years to do the right thing, to follow their own rules, and they failed to do so. They even decided - decided! - to construct the levees and floodwalls to improper benchmarks, resulting in shortfalls of two feet when one compares actual heights to design heights! Now, we are supposed to believe that structures conceived and designed during the same time the Corps was denying responsibility for Katrina's devastation (from September to December of 2005) are going to be stronger than anything else they've ever put out there? We're supposed to believe that - despite a documented history of non-functionality for over a year - the MWI pumps are all of sudden going to be better than sliced bread? We're supposed to believe that the same bunch that allowed the trenches at the gates to be filled and sealed with two feet of grout during the height of the 2006 hurricane season somehow now knows their ass from their elbow? No, there's been too much wrong with all of this to believe it can work. And if the gates don't work, then we're back to relying on the walls - the same walls that are known to be underdesigned and poorly constructed.

So what if all that money had instead been applied to fixing the walls? Could it have been possible? Would New Orleans have been safe during 2006? Maybe, maybe not. But to not even consider fixing errors which have been reported very widely and have been analyzed by at least three teams (IPET, Cal-Berkeley, Team Louisiana), is absolutely nuts. Heck, Team Louisiana's report even tells the Corps exactly how deep the sheet piling needs to be along London Avenue. The design work is already done.

How the floodgates - combined with New Orleans District incompetence - have made things worse in new and innovative ways
There are other real consequences to these really bad decisions. Let's look at one in particular.

In the floodgate scheme, there is not enough pumping capacity at London Avenue and 17th Street to match the pre-Katrina drainage capacity (approximately 9300 cfs on 17th St and between 7000 and 8000 cfs on London Ave.). There never has been, and I don't think there ever will be. Let's look at that situation in a little more detail.

My assertion that there probably never will be enough pumping capacity doesn't come from thin air. The Corps says as much in Rev. 4 of the PIR, issued in October of 2006:

"Based on a cursory review by ERDC [the Corps' drainage gurus in Vicksburg], their belief is that large discharge capacities required at 17th Street and London Avenue may be difficult to achieve. This concern is based on a large number of pumps installed in a very small canal area with an erratic distribution of flow. A definite recommendation cannot be made at this time by ERDC."

I'd say we know the recommendation now, seven months after issuance of that report. The plain evidence is this: the Corps is spending $135 million to get pumping capacities up to levels that are still less than pre-storm values (about 7500 cfs at 17th St, and about 5000 cfs at London Ave). A lot of that money had to be routed through an extraordinary Air Force contracting instrument outside of normal Corps contracting procedures. Does anyone really think a similar, uniquely funded sum will be spent any time soon, considering how much has already been thrown down this hole? I'd say "that's all folks" on extra pumps. We're stuck with what we've got coming this summer, assuming they all work.

In addition, we've also got evidence that they've also unofficially put the kibosh on more pumps at 17th Street, as mentioned in a little noticed article in the Times-Picayune on May 6, 2007.

The article was about the townhouses which line the west side of the 17th Street canal floodgate site. They are collectively known as Mariner's Cove West, and are part of the Mariner's Cove development, which went up in the 1970's. Before Katrina, they were very valuable properties.

The Times-Picayune missed the entire point of the article, focusing on the homeowners and their high incomes. But Mariner's Cove West has been key to this entire pumps-and-gates story, without anyone noticing since the beginning.

Mariner's Cove West has actually been used by the Corps for over a year as an excuse NOT to expand pumping capacity at 17th Street. In Rev. 2 of the PIR, issued in May of 2006, the Corps laid out their case for adding about 4000 cfs of additional pumping flow at the 17th Street floodgates. Most of that flow in the Rev. 2 scheme (which was eventually mostly scrapped) would come from hydraulically-powered pumps along the west side of the canal. Why not put more pumps in along the east side?

"Adding capacity beyond the additional 4,000 cfs would require considerable additional rights-of-way. On the east side, sub-alternatives are limited by townhouses."

Yeah, they were afraid to tear down Mariner's Cove West. This despite every owner of every unit wanting a buyout since right after the storm. But very early on, the Corps New Orleans District (specifically Fred Young) told Mariner's Cove West townhome owners that the Corps had no intention buying them out. They were told this as early as February, 2006 by Mr. Young, and told the same thing many times after that. The Times-Picayune did a big story about Mariner's Cove West on April 19, 2007 (about a year late, in my estimation), mostly focusing on the struggles the townhouse owners have faced with the Corps' N.O. District Real Estate Division.

Let me make clear exactly what I'm saying here, because this is new. The Corps, knowing they needed more pumping capacity at 17th Street, refused to make way for that capacity - even though every person who would lose their home was willing to be bought out. Instead, the Corps dithered for over a year, leaving the Mariner's Cove West folks in limbo, and - more importantly - leaving everyone who drains into 17th Street without their prestorm pumping capacity.

Now, back to that May 6th Times-Picayune article, because it gets worse. It's about the Corps finally buying out Mariner's Cove West. Except - and you're not going to believe this - they're not going to use the land for anything! Here's the key quote from the article:

"'I just wanted out, but this helps others, who were unsure, to move on,' said [Mariner's Cove West resident Mike] Frank, who has left his native New Orleans for northern Florida. 'They're telling us none of the property will be a project; they will make it green space, so that's good news for the rest of Mariners Cove.'

I nearly spit up my drink when I read this in the paper the day it was printed. I also was amazed that the T-P didn't make the connection between pumping capacity and Mariner's Cove. The folks at New Orleans CityBusiness missed it too when they reported the same basic facts, but with a little more depth in a May 14, 2007 article. They wrote one sentence about the use of the property and also got a vague quote from Col. Wagenaar:

"The Corps said it was necessary to acquire and demolish the buildings to continue current work and provide maintenance space for future projects. 'Because of the ongoing and future work at this location we made the decision to acquire this property,' said Col. Richard Wagenaar, commander of the Corps' New Orleans District."

You have to read the tea leaves and between the lines with these folks. Sometimes, what isn't said is way more important than what is said. And what isn't said - between the two articles - is that the Corps is buying the property in order to fit more pumps into the 17th Street site. That's significant. Do we really think they would hold that information back? Combine it with a) the already stated objections by the local Corps folks that the site is too small; b) what I know the Corps has been telling homeowners for the last year (yes, I've talked to many of those homeowners myself); and c) ERDC's worries, and I only see one conclusion.

So here's the connection between the Mariner's Cove West buyout (specifically its timing) and 17th Street pumps, for those too dim at the paper to make it themselves: there will be no more pumps at 17th Street, despite a bunch of real estate opening up at the site. We're not getting back to pre-Katrina pumping levels for a very long time.

So that's another consequence of placement of the floodgates in the first place: they couldn't actually provide a system that has the same level of protection as we had before Katrina. And in not repairing the walls and enstating the Safe Water Levels, they've constrained that level of protection even further, to the point where we're affected even when there is no hurricane. The two decisions - building the floodgates and not fixing the walls - have had deeply troubling consequences for our city.

The really big consequence - Flood Insurance
Last April, everyone in town was in a tizzy because FEMA had yet to release its post-storm guidance on flood insurance and BFE's (base flood elevations). The ABFE (Advisory Base Flood Elevations) for Orleans Parish was finally released on April 12, 2006. These have allowed the rebuilding to proceed by allowing residents and businesses to purchase flood insurance.

The ABFE's (they were issued for all Katrina- and Rita-affected parishes and counties), which were legally adopted by Orleans Parish on August 3, 2006, are conditional documents. That is, they are based on the Corps doing certain things by certain times. Specifically...

"In addition to the recent USACE storm surge modeling, FEMA has also developed these recommendations based on the height and integrity of the levee system expected to be in place by September 2007."

So what if the levee system - which includes the floodgates and pumps - is not what FEMA expected to be in place by September, 2007?

"Although FEMA is confident in the results from this current assessment, the agency will continue to monitor progress made with regard to levee improvements, findings from other ongoing studies, and enhancements to the agency’s understanding of the probability of flooding in this area. FEMA will adjust the recommended flood elevations as necessary as the agency prepares updated FIRMs [Flood Insurance Rating Maps, popularly known as "flood maps"] for Orleans Parish and its incorporated areas."

And what does that mean? You only have to go one paragraph backwards to find out...

"For the Parish Advisory BFE (ABFE) inside levees, this Guidance is similar to NFIP rules for areas protected by levees being restored to provide 1-percent-annual-chance base flood protection. Should the requirements needed for application of these rules fail to materialize, flood elevations in this area would be based on a "without levee" scenario and could exceed elevations of 8 feet (west and south of Mississippi River) or 13 to 14 feet (east and north of Mississippi River) referenced to the National Geodetic Vertical Datum of 1929 (NGVD29)."

Now a little explanation is in order. Last year, the Corps was saying they would have a certain level of work done by September, 2007. According this Corps press release, issued the same day as the Orleans ABFE, that level was,

"September 2007 - Completion of restoration of undamaged and subsided areas; completion of previously unconstructed portions of authorized projects"

It's unclear if that's going to happen by this September. There are so many projects included, and I don't know the status of every one. It's also unclear if FEMA is accounting for the floodgates and unrepaired walls in their calculations.

Restoration of the system to the 100-year level authorized by Congress was supposed to be done by 2010. However, the Corps has pushed some of that work out to at least 2012. It is that work which is supposed to allow for true FEMA FIRM's to be issued. But since the region can't wait until 2010 or 2012 or whatever for final maps, FEMA will work off of the Corps long-delayed risk and reliability study, which will supposedly include all the 2010 work.

My question is this: if the current gates and pumps and the lack of repair to the walls are shown to be failures, what happens to flood insurance in the city the day after that failure is confirmed? That is, will FEMA follow up on their statement in the ABFE to "adjust" the flood maps, possibly adjusting them to a "without levee" condition?

I'm certainly no expert on all this flood map stuff, but I can read. And I think the (admittedly very hard) question needs to be asked of FEMA's mitigation folks and the Corps, although we already know the Corps' answer - "everything's fine!"

It needs to be asked because everything in the recovery is based on the availability of flood insurance, and I mean everything. We're talking about billions of dollars in Road Home funds, as well as everyone's mortgages, which require flood insurance, all of which could be placed in jeopardy if the gates don't work, simply because the walls haven't been touched.

Conclusion - the Corps has royally screwed up
None of this would even be happening if they hadn't put in those stupid gates in the first place cutting off New Orleans' drainage capacity. But then they decided to not do anything to the walls, doubling down on their earlier stupidity. So they've made things worse for the past hurricane season, for the coming hurricane season, and for every season until 2013. That's a total of seven hurricane seasons, never mind the problems with normal rainstorms and the safe water level limitations. Like their actions with the datums and the sheet pile lengths (well chronicled in Team Louisiana's report, specifically chapter 5 and chapter 6), the Corps has deliberately bungled their mission of protecting the citizens of New Orleans.

The Corps will not fix the walls. They will rely on their beloved yet untested, unproven, and defective floodgates (we won't really know they work until a storm hits), leaving New Orleans in a more perilous condition than before Katrina. Residents should think long and hard about relying on the gates, the MWI pumps, and the unrepaired walls to save them. The situation has become unbearable, and I hope people realize it before the worst happens, rather than when it's too late.

Enough

2007-05-11T22:29:00.000-07:00

Updated quite often. See below at various spots.

In light of some very cloying, cowardly coverage by the local media...

a) WWL-TV does a report on London Avenue canal testing on May 10, 2006, and only interviews Corps personnel, and then shows the wrong spot on the canal where the test will take place [they showed the repaired breaches at Robert E. Lee Ave, which is over a mile north of the area where the tests will really take place - just south of Mirabeau Ave.]!

[Update, 6/1/07: I have issued a correction for the above paragraph, which is erroneous. The correction can be found here.]

b) WWL-TV does a special series of reports titled "Closing the Gaps," from April 30th to May 3, 2007, supposedly about all the problems we still face with levees and pumping, problems brought about by the Corps. So who do they interview as the main source of information... the Corps! Part 1 here (video here), part 2 here, part 3 here, I couldn't find the print version of part 4, but the video is here.

c) On May 9, 2007, the Times Picayune republishes a Corps press release about the new head of the New Orleans District as an "article" without mentioning that the current commander - Colonel Wagenaar - is retiring a year before his usual 3 year tour is up.

d) This one reaches back a bit, but on Sunday March 25, 2007 the Times-Picayune published what was portrayed as a "big" front page article, a real love letter to the Corps, detailing how ready the Corps was to save us all. In reality, it was just a huge rewrite of the Floodgates Operating Manual, a document which has been publicly available here - due to my requests - since last August. Also the article included only quotes from Corps of Engineers personnel (a hallmark of a lot of Times-Picayune coverage of the Corps these days), including a tremendously cheap shot by Wagenaar:

"...And although Wagenaar applauds the 'heroic' work of local emergency responders to the 17th Street Canal breach -- especially West Jefferson Levee District employees who engaged in road building -- he said he welcomes the streamlined chain of command achieved through levee district consolidation. 'What is not streamlined is dealing with the Sewerage & Water Board. They don't seem to work for anybody.' Wagenaar said."

Of course, the article's author leaves that comment - which is the height of hubris from an agency that seems to answer to no one, including Congress - unrebutted. She apparently didn't even bother to call the S&WB for comment. Also missed in all the glorification of the Corps' communications plan is this actual quote from the Manual:

"One Corps of Engineers radio will be furnished to the S&WB for communications in addition to having the Canal Captains stationed at their respective S&WB pump stations."

Yep, that's right: the agency in charge of pumping gets all the benefits of a single radio to communicate with the Corps.

[Update, 6/7/07: The radio has been addressed as part of the kerfluffle over control of Drainage Pumping Station #6 on the 17th Street Canal. See my June 7th post here.]

... So in light of all that (and much, much more) crappy coverage of what the Corps has done and isn't doing over the last two weeks, allow me to let you in on some real stories that anyone with a modicum of investigative sense would have ferreted out by now:

1) The hydraulic fluid reservoirs on MWI's engine skids are undersized by a factor of 3. MWI's own catalog calls for a 900 gallon reservoir. FPI's 60 inch pumps use a 900 gallon reservoir. The MWI models at the floodgates have 300 gallon reservoir. This decreases the capacity to get rid of heat generated when the pressure of the hydraulic oil is raised up to 3000 psi (theoretically - though it's probably closer to 2600 psi), and then shares some of that energy with the Rineer hydraulic motor in the pump unit. If there's too little room in the reservoir, the fluid doesn't have enough time to cool, meaning the oil stays thinner than it needs to be. Less viscosity means less ability to transfer energy means less ability to pump water.

Why did MWI do this? Maybe to save money on hydraulic fluid, which they were responsible for supplying for startup and commissioning. Such a savings could have been in the range of $50,000 to $150,000, money that goes straight to MWI instead of some supplier.

According to my latest batch of testing reports, received via FOIA request (and held back by New Orleans since January), during testing MWI actually was pointing fans at the drive units to keep them cool until the Corps QA's told them to stop it. Overheating of many components - not just the hydraulic system, but also the Durst gearboxes - on the drive units is still a big, unaddressed concern.

2) The Corps has yet to issue three Orleans Parish pump station repair contracts, over 20 months since Katrina. These contracts are part of the original $40 million set aside for pump station repairs. The three contracts - Electrical-Mechanical Repairs, Building/Structural Repairs, and cleanout of the PS#17 Discharge line and the inlet basins of three or four (it's unclear if Elaine Street's basin cleanout is included in this contract) other stations - appear to be on the very slowest track possible, since the Corps has had the money since January of 2006. The following is a quote from a February email sent to me by one of the people working on the Discharge Line solicitation (W912P8-07-R-0027), which has been indefinitely postponed since February 12, 2007:

"The reason we suspended the solicitation was due to the Right-of-Entry agreements we are working on getting from the Port of New Orleans and Project Permits from the Orleans Levee District and DOTD. We were under the assumption that we could do the work under the ROE and CA with the New Orleans S&WB. We are doing this so that everyone knows what is going on and we don't have any delays once the project is awarded. The ROE from the Port of New Orleans has been obtained and we are working on getting the two project permits."

Sure sounds like urgency to me!

[Update, 7/9/07] And the ugency continues on this one. The PS-17 Discharge Line/PS-6 & PS-7 Basin Cleanout solicitation was revived on June 5th, and has since gone through numerous changes and delays, still without award of a contract. The latest modification (number Seven), issued July 9th, changes the response date to July 11.

Also, one of the other remaining Orleans Parish repair contracts was finally issued a couple of weeks ago. According to this Corps contract award announcement, a $3.25 million contract for Electrical/Mechanical Repairs at the S&WB drainage pump stations has been awarded to Healtheon, Inc on June 19, 2007. These repairs will happen at nearly every one of the 23 stations.

While the award announcement gives a local addess (Suite 2500 in the downtown skyscraper at 201 St. Charles Ave.), the Louisiana Corporations database shows that Healtheon's principal office is in Birmingham, Alabama. A Google Maps search of that address shows it to be a private house. Its webpage shows its focus to be in "healthcare construction," but it appears to have a division which just happens to deal in "disaster recovery." Healtheon does not appear to be registered as a contractor in Louisiana, according to the Louisiana Licensing Board for Contractors website.

Healtheon also received the contract for Electrical-Mechanical repairs to pump stations in St. Bernard Parish, as noted in this January 28, 2007 award announcement. The value of that contract was about $786,000. How is it there was only one company anywhere that could handle these two contracts? Why wasn't this open to competitive bidding?

The Orleans Parish E-M repair contract, like the St. Bernard Parish E-M repair contract, was yet another sole source 8(a) set-aside. Sole source 8(a) awards have become the pattern for the Orleans Parish pump station repair contracts. Almost all of them, except the motor rewinding contracts, have been sole source set-asides. There's no competition required, and I've yet to find where one can even find solicitations for them. They stink to high heaven. And what the heck took so long to issue this latest contract? I'll tell you - no urgency.

Speaking of which, two other Corps pump station repair contracts still remain unissued. They are the Building & Structural Repairs contract, and the Elaine Street Station Reconstruction contract. The Elaine Street Station reconstruction contract is somewhat new, having appeared on the docket within the first few months of 2007.

[Update, 7/24/07]
As I surmised above, the PS-17 Discharge Line/PS-6 & PS-7 Basin Cleanout, the Building & Structural Repairs, and Elaine Street Reconstruction contracts are still unawarded, according to the July 19, 2007 schedule of Orleans Parish Pump Station Repairs from the Corps.

Also extremely ~~disappointing~~ embarrassing is the fact that the repairs are moving at a snail's pace. As we approach the two year anniversary of the storm - and as we just passed the 18-month anniversary of the signing of the bill which appropriated the money for these contracts (the 3rd Katrina supplemental was signed December 30, 2005) - three contracts remain unawarded and fewer than half of the 14 contracts have been completed. In addition, those that have been awarded are now taking even longer than previously scheduled.

For example, contract OPS-6A, (awarded to IPS of Louisiana and then subcontracted to Conhagen; Corps award notice from 12/1/2006 here), intended to replace bearings in vertical pumps at five pump stations, was previously scheduled to take 243 days, according to the Corps' November 1, 2006 repair schedule. The latest schedule shows that work now taking 290 days and dragging into the end of September, 2007. And if one takes a close look at the percent completion (42%) versus the percent of time passed since the contract started (75% of the anticipated 290 scheduled days) it appears that the work on this contract is even further behind than indicated. I doubt the contractor will be able to finish 58% of the work in 25% of the time. The delays seem to be concentrated in the three New Orleans East stations (14 and 16 on the lakefront, and 15 way out east near the end of the world), where construction progress is respectively 19%, 12%, and 26%.

Looking at the big picture, the scheduled completion date for all of the repair work is February 9, 2008. That's 34 months after the storm hit!

They really don't care, do they?

3) [Update, 5/16/07] Has anyone asked about the Corps' plans to evacuate workers from the floodgates for anything over category 2 (see the Floodgates Operating Manual)?

Also see this 5/16/07 article from the Times-Picayune:

"The other three members of each corps gate team will stay at the gate for storms up to Category 2 hurricane strength. For Category 3 storms, they'll relocate to a safer structure nearby. For Category 4 and 5 storms, considered worst-case hurricanes, one three-man team will retreat to a bunker at the corps headquarters complex on Leake Avenue at Prytania Street, while the other two teams will evacuate the area until the storm passes, Wagenaar said."

According to the manual, that nearby safer structure is the Louisiana Dep't of Transportation & Development building at 7252 Lakeshore Drive, shown here. It's about a quarter mile from the 17th Street gates, which makes it quite far from the gates at Orleans and London.

There's no way to remotely control (other than on/off) or adjust the floodgate pumps; that has to be done at the drive skid. The pumps will very likely need manual adjustments - which will be impossible with no one there. This was buried at the bottom of the T-P March 25 article, written by a different, less rigorous author than the May 16th article, and made to sound not very important:

"If the area is threatened by a Category 4 or 5 storm, the men probably would be pulled off site once the gates were down and the pumps running. They would return as soon as possible after the storm passed, Accardo said. 'Why have them in harm's way? These aren't really safe houses in the sense that Jefferson Parish is building safe houses,' he said. 'These are equipment houses designed to handle a hurricane, but not a major storm surge.'"

Well, what if there's a problem with the fuel lines from the 20,000 gallon diesel fuel tanks? By the way, the Corps could have gotten dual drive units, with electrical motors as well as diesel ones - MWI offers them in their catalog, which is the basis for the Corps specs after all. Or they could have specified a drive system that could actually be controlled remotely, rather than one that can barely be turned on and off by remote. But why bother going for backup power or a truly robust control design? They were in a hurry!

In any case, the Corps knows leaving the pumps alone is a problem, but they just skirt over it in their own manual. From "Step 8 (T - 12 hours)" on page 10 (Adobe page 14):

"Due consideration must be given to what storm conditions are expected to be after the gates are installed and the pumps are initialized to permit the personnel that are part of the closure operation will be out of harm’s way as they travel to local safe locations or are being evacuated to remote safe locations."

That is, the Corps plans on "initializing" the pumps, i.e. turning them on and hopefully maximizing their flow, and then leaving them. What happens if they shut down and can't be restarted remotely (a very likely circumstance if one reads how to start them in the first place- which definitely requires someone being physically in front of the drive unit)? Well, I guess then we're out of luck. The Corps says, "c'est la vie."

I wonder if they plan on rehearsing leaving the pumps unattended the entire time they're running in their drills for this hurricane season? I sincerely doubt it.

4) Does the entire city know and understand that we're depending on a contract to a diving firm to guarantee the gates actually work? When the gates are lowered, they seat in a trench, which a firm called H.J. Merrihue is responsible for keeping clear during operations. If the gates don't seat in that trench, they'll fail. I'd like to know exactly how the diving firms do that clearing while a tropical storm is coming into the lake, and the city is pumping rainwater out along the canals. Plus, the Corps says the diving firm is responsible for evacuating their own personnel after the gates are lowered, despite the Corps getting the Corps folks out on helicopters under a separate contract. Again, see the Floodgates Operating Manual for the details on this one.

These are the same trenches that the Corps' contractors filled with grout during the height of hurricane season last summer without realizing it. They had to chip the stuff out mechanically at 17th Street and Orleans Avenue, meaning had they tried to lower the gates before the trenches were clear, they wouldn't have seated, and the city would have been defenseless against storm surge. I broke this story last August and posted about it to a number of local newsgroups. You can find my postings here, here, and here. I also posted some post-game analysis, including catching the Corps in some lies about the importance of the trenches, here. My concerns about the gates seating still exist, and have been echoed in National Geographic's latest feature, from which comes this nugget about the floodgates:

"A Dutch engineer recently visited some of the new floodgates and pumps installed at the mouths of the city's three main drainage canals. His verdict: They may be 'doomed to fail' in the next big storm. The engineer, who asked not to be named because he sometimes collaborates with the corps, notes that the gates have no mechanism to remove sediment and other debris that might keep them from closing as a storm approaches. Instead, the corps says it will rely on divers to check for obstructions and clear them away."

This is very troubling. Supposedly, the Corps was supposed to construct some metal pieces designed to fit in the trenches to keep them clear until they're needed, but of course we never heard whether that happened, or if it did, if that idea was successful. Plus, I'm pretty sure the trenches themselves are pretty beat up from all the jackhammering last summer, so I doubt they could build pieces that would completely seal against silt, rocks, and other debris building up. It's just another example of how what appears to be robust could in fact be very weak.

[Update, 7/24/07]
The grout in the trench problem was apparently very serious. It was noted in a PowerPoint slideshow given June 27, 2007 by the consulting engineers on the floodgates job at last months' SAME Infrastructure conference. The main conference webpage is here, and the presentation on the gates - given by guys from Linfield Hunter Junius and URS - is here.

Here's the exact text regarding the grout problem, from slide 43. Note the final bullet point.

"- Soil Mixing
• Soil Improved to an Average Shear Strength of 1,000 PSF minimum
• Allowed Fewer Piles
• Critical Item in Construction Schedule
• Soil Mix Residue Issues – Grout Silted in Gate Foundation and Pump Sumps – Major effort to remove soil mix residue with divers"

There are also pictures of the grout that flowed all over the site and is still there (slides 73 and 74). The grout went everywhere. It looks like someone stepped on a giant tube of toothpaste.

The presentation is fascinating, since it contains information never before revealed, like alternatives considered to the final needle gate design. One choice was to sink barges!

The presentation appears to have been thrown together fairly hastily; the pictures of the grout from 17th Street are actually dated two days before the slideshow.

And of course, they are still putting lies about the pumps out there. There is a mention of them on slide 47: "Hydraulic Pumps – Only Type of Pump Available in Allotted Time." This fails to mention that the pump specification was taken straight out of MWI's catalog. It also fails to mention all the problems before and since the installation of the pumps.

One other thing that isn't mentioned: the sill at London Avenue was completely screwed up and delayed startup of that floodgate for over a month. The sill problems were mentioned in Times-Picayune in a May 12, 2006 article (back when the T-P actually cared what was going on around them). There were more details in this July 3, 2006 article from something called "Construction News:"

"Complications with the sill prevented M.R. Pittman from completing the London Avenue Closure by the June 1 deadline. 'We expected to have the sill complete, but we ran into difficulties with the sheetpiling,' said Leroy Smith of the USACE. 'Some split due to hard driving. When compressing sheetpiling, we hit hard material at elevations of minus 58 or minus 60.' Smith says the piling started racking and fell out of alignment. The sheets, separated at the bottom, had to be pulled up, and Pittman had to start all over.

Ranjit Gujja, USACE project engineer for the London Avenue Closure, said the same strength of sheetpile was used when driving the second time around. 'We used a double template instead of a single template to give more balance and alignment.' Sheetpiling was driven between the two templates the second time around, to prevent racking.

The sill, a crucial element in the design of the canal, held the job up. On June 1, divers were cutting down the sheetpile on the west side of the sill, which was complete, and Pittman was in the process of completing the east side. The closure is expected to be complete by July 1. The cells were expected to be completed within a week. Discharge pipes were set, and the pumps were scheduled for a test run the next week."

[end update]

There's tons of other questions which must be asked, and which are not. This is just a sample.

Documents

2007-05-09T23:05:00.000-07:00

Headlines:
- Corps admits to copying & pasting, but says it's not really their fault. Their paid consultants did it.
- Newly revealed: Pump contract has been under Corps investigation for over eight months, report due this month
- MWI tried to downgrade pump testing before testing even started

Anyone who really reads my blog carefully knows that almost everything on it relies on primary source information - either my own observations or original documents. Today, I've got two more documents.

1) The Corps defends copying and pasting - kind of

The first document is the Corps' May 2nd response to Senator Vitter's April 13th inquiry regarding the copying & pasting of MWI's specs into the Corps' January 13, 2006 bid solicitation for the original 34 floodgate hydraulically-powered pumps, as well as the same practice getting employed for the Corps' June 9, 2006 follow-on solicitation for 23 more hydraulically-powered pumps.

The public doesn't usually see the responses to letters from members of Congress to various agencies. That's why this is so interesting.

I personally find it a fascinating glimpse into how the Corps acts when their hand is caught inside the cookie jar. Their defense basically boils down to, "Sure we copied the specs, but we copied them from both MWI's catalog and FPI's catalog. And also, it wasn't us doing the copying; it was those consultants we hired." So tell me URS and Linfield Hunter (and any mechanical subconsultants hired by you), how does it feel getting thrown under the bus?

The Corps letter makes no mention of the June specification being copied & pasted, despite its explicit inclusion in Vitter's original letter.

What's also realllly interesting is that the letter also reveals the existence of a previously unreported Corps internal investigation which has apparently been underway since September, 2006. That is, all the time the Corps has been reassuring the public that everything was fine, they themselves have been looking into the pump problems. Objectively, this is a good thing. When problems crop up, it is a good thing to look into them. Except that the Corps was aware of serious problems since at least April, 2006, almost six months before the official internal investigation launched. Also, they've never mentioned their internal investigation to the public, and we wouldn't know about it without my releasing this letter.

The letter says that the internal investigation is rolling toward a conclusion at the same time as the GAO investigation - this month. I hope some enterprising reporter goes after the results of the Corps internal investigation...

The letter also shows evidence of General Strock or his staff not being given complete information. It claims that FPI wasn't technically strong. That's not completely true. FPI was told directly by the Corps during their debriefing (the meeting after the contract award that was quoted in the AP article) that - other than problems with their engine suppliers' delivery dates (the suppliers' dates apparently didn't match FPI's assurances for delivery) and the fact that FPI had specified three different engine manufacturers (the Corps thought that could lead to maintenance and spare parts problems down the line, even though there's nothing in the spec specifically forbidding different engine suppliers) - the technical aspects of their proposal were perfectly adequate.

Let me also address the contention that FPI's specs are just about the same as MWI's...

FPI makes hydraulic pumps very similar to MWI. Both companies have a complicated, intertwined history. And it is true that the specification that FPI uses for their hydraulic pumps with skid mounted engines, found here on FPI's website, bears a striking similarity to MWI's spec in many respects, probably due to that intertwined history. However, the FPI section on piping actually has a callout (left blank for the customer to insert) for the pipe diameter (MWI and the Corps did not call out the piping diameter), its testing section only includes two of the five paragraphs found in MWI's spec (dynamic factory testing is not even mentioned in FPI's spec), and the FPI spec is missing many other passages found in both MWI's and the Corps' spec. It is extremely unlikely that FPI, or any of the other bidders, thought the Corps' spec was anything but based on MWI's spec.

For reference, you can read many other FPI specs at this page on their website. The specs are found in the left hand column.

2) Shop inspection reports

I got a big manilla envelope in the mail yesterday. It was the results of a FOIA request I made last December. I requested all the shop inspection reports - including photos - written up by personnel from the Corps Jacksonville District while they were stationed as Quality Assurance personnel at MWI's facilities in Florida during 2006. Some of those reports were attached to Maria Garzino's memo, but the photos were low res, and not every report was included as an attachment by Ms. Garzino.

I now have all the reports (well most of them - the Corps may have held a few back - I've still got to check). They're pretty dense, but the pictures are pretty amazing. Also, it seems MWI was looking to back out of their own testing requirements even before the testing started.

What follows is a quote from Shop Inspection #5, which took place April 8, 2006. The first tests on the pumps and engines were on April 11, 2006.

"3. The Contractor's Testing plans for both the Dry (Static) testing and the Wet (Dynamic) testing discussed. On Friday, 7 April 2006, Mr. [redacted] requested a variance to the 90-minute dry (static) test to be changed to 90 seconds. he has requested this variance through the New Orleans District. As of Saturday, 8 April 2006, he has not received a decision. No dry testing will be conducted until further direction from New Orleans is received..."

This particular change was never implemented. However, one has to wonder why MWI was looking to downgrade their own testing specs even before testing started. One also has to wonder whether the Contracting Officer (Ms. Cynthia Nicholas) was who MWI was contacting for changes, since the Contracting Officer is the only person legally authorized to approve such changes.

Update, 6/3/07: The shop inspection reports are now available online in this June 3, 2007 post.

It's May in New Orleans - that means rain

2007-05-07T13:54:00.001-07:00

[Updated 5/12/07 & 7/28/07]

[For the latest on London Avenue canal testing, see my June 11, 2007 post, "Testing, testing." It has been updated quite often.]

We had a downpour in New Orleans on Friday, May 4. During the storm, I went out to the London Ave canal around 1:30 or 2 PM. The safe water level (SWL) in London is 4 feet above lake level. Whenever there is a major or minor rainstorm, the place to check for problems west of the Industrial Canal is along London Avenue, since its SWL is so low, and can be exceeded with a heavy rain combined with a wind out of the north. We had both of those circumstances on Friday.

I looked at three points along the canal.

At Pumping Station 3, the staff gauge read 4.5 feet at the discharge basin. At the time I was there, all five major pumps (A & B at 550 cfs each and C, D, & E at 1000 cfs each) were running. The Corps has given the Sewerage & Water Board special dispensation to let water rise to five feet in the pump station discharge basin.

At the Mirabeau Avenue bridge, just north of the ~~weakest area of the canal walls~~ [Correction, 6/1/07: I got the location of the load test wrong. I assume the load test location is the weakest section, but that's not necessarily the case. See the detailed correction, with the load test location, here.] the staff gauge read 4 feet and the water was rising.

At the Leon C Simon bridge, just south of the floodgates, the staff gauge read approximately 3 feet. The floodgates were all up.

The staff gauges are basically yardsticks stuck in the mud of the canal. It's unknown how reliably they measure depth since they may have subsided over the years (though based on my observations and the timeline reported by the media, it's looking like they're pretty close). The safe water level is officially measured based on the new automated sensor system installed under the Corps' contract with Prime Controls of Texas. There are five or six depth sensors along London Avenue (just as there are sensors along the other two outfall canals), all of which are tied into the SCADA system set up to monitor the pumps at the floodgates. If you've driven across any of the outfall canals, you've seen the solar panels sticking up from inside the walls. Those are the locations of the level gauges. Their output can be seen on computer terminals in pump stations 3 (London Avenue canal), 7 (Orleans Avenue canal), and 6 (17th St canal), as well as in the Corps' building on Leake Ave.

As we found out later on May 4, (through some pretty confused reporting from WWL-TV and the Times-Picayune), at least one pump was turned off at PS#3 as a result of exceeding the SWL of four feet. The level got to 4.3 feet at the sensor across from Dillard University (roughly described by the Corps - via WWL - as "near Harrison Avenue," even though Harrison actually doesn't cross the canal) before the pump shutdown. This came as no surprise to me. It must have happened right after I left PS#3 around 2 PM.

We also found out that the Sewerage & Water Board lost 25 cycle power for somewhere between a few minutes and 40 minutes (the reporting was pretty cruddy on this), when a natural gas compressor at their 25 cycle power plant went down and they had to switch to diesel backups to get power back on line. About 60% of the S&WB pumping capacity across the city runs on 25 cycle power, so this was a big deal.

What a mess.

Update, 5/12/07:
The Times-Picayune's Michelle Krupa sorted out the whole mess in an article today. She did a good job giving details of the 25 Hz shutdown on May 4, as well as giving tons of background on how fragile the S&WB power system really is.
[end update]

I'd like to focus my comments on London Avenue, a topic I've been hammering on for over six months now.

The Corps New Orleans District keeps shrugging off permanent pumping capacity decreases along London Avenue (which drains a huge area that includes the French Quarter) as no big deal. The first couple of times the canal depth either got close (December 21, 2006) or exceeded (December 30, 2006) the SWL, I was the one that broke the news because the New Orleans District didn't have the guts to admit what happened, even when they were given the opportunity to do so in an open, public forum (New Orleans City Council Public Works Committee hearing of January 8, 2007).

I guess they got tired of having their dirty little secrets exposed. So this time they decided to get out ahead of the story, but not without some typical, mendacious spin. Check out this quote from resident Corps spokesbot Vic Harris in an AP article about Friday's rain:

"Now everything is copacetic," Harris said. "This is not an uncommon thing. We make those adjustments commonly when the water goes up."

First off, the Corps doesn't make the "adjustments," the Sewerage & Water Board does. Secondly, the only reason the S&WB has to do this is because the New Orleans District hasn't done any-damn-thing to address the weakened walls along London Avenue. Sure, they have an idiotic plan to do "testing" at the weakest section of eastern canal wall south of Mirabeau Avenue, but we've been hearing about testing for literally months, and nothing has happened. And thirdly, it is a very uncommon thing to have infrastructure and personnel that have performed wonderfully for nearly a century suddenly told to stop working because another organization can't get its act together. I don't appreciate having my local tax dollars, which fund the S&WB, wasted by the Corps because they can't fix their own blatant mistakes.

The latest information we've received about the testing came courtesy of the T-P on April 20th, when the paper decided to do some investigative reporting by ... having a reporter attend a meeting where all the details of the testing plan were spoken about at length. The article (as with anything really important about the Corps other than cutting down trees, which gets front page coverage) was buried inside the local section.

That article mentioned that the plan called for a 50 day schedule. On the day the article appeared, we were already smack in the middle of rainy season, and less than fifty days from the beginning of hurricane season. Now we're even closer, and still nothing has been done. An earlier article quoted the Corps as trying to pin the blame for delays on the S&WB and the East Bank levee board for not giving rights of entry. The S&WB has already given the Corps the R.O.E. The East Bank levee board has not (to the best of my knowledge). However, the East Bank levee board's objection is not to the idea of isolating the weak section, but to inadequacies in the Corps' testing plan. NO ONE is against driving sheet piles to isolate the weak section, and the New Orleans District needs to stop playing games by insisting on their stupid testing as a condition for blocking off the most dangerous section of levee in the city west of the Industrial Canal.

Why are they doing this silly testing instead of something a little more comprehensive or common sense? Since the Times-Picayune reporter didn't ask the question on April 19th (or any other day for the last nine months since the four foot SWL was announced for London Avenue), I am forced to guess the New Orleans District doesn't have the money to do anything else, and they suspect that major repairs are necessary along the entire length of the London Avenue canal. They might be using this testing as justification for more funds (many more funds) from Congress. This might or might not be backed up by this quote from the T-P's confused coverage of Friday's storm:

"The corps hopes to eventually raise the limit on the London canal about 2 feet, but first must conduct tests of the walls by filling a portion of the canal and then the entire canal with water. Those tests have not been scheduled."

It's impossible to tell if the T-P really means that there will be whole-canal testing, or if it was just a writer not knowing the facts. But considering that the subheadline of the article was blatantly wrong, I tend toward the latter.

[Updated 7/28/07]
See my June 11, 2007 post, "Testing, testing," for the most up to date information on the London Avenue canal testing. The Engineering News-Record articles linked there are far better than the Times-Picayune's pathetic coverage.
[end update]

The subhed was wrong? Yes. Here's the headline:

DOWNPOUR
New pumps fail a major test as a strong storm knocks out power and downs trees, but Jazzfest takes it all in stride

There were no "new" pumps involved in Friday's storm. I can only assume that someone very clueless threw the word "new" in there because of the recent AP coverage of the floodgate pumps. I tried to call a correction into the T-P newsroom three times on Saturday, but no one picked up. I guess there was no news on Saturday.

But on the point about the floodgate pumps, let me make this very clear to the Times-Picayune, local video editors at our TV stations, and everyone else in the media:

THE FLOODGATE PUMPS WILL ONLY BE TURNED ON WHEN THE GATES DROP DURING A TROPICAL EVENT.

So when the story is about a non-tropical rainstorm (as it was on Friday), please stop showing archive footage of the floodgate pumps and stop mentioning them as if they play a role in non-tropical systems. This is a basic fact that the local electronic media seem to be incapable of grasping. I constantly see chyron graphics over anchors' shoulders - including this past Friday, Saturday, and Sunday - showing the floodgate pumps whenever there is a drainage story. Such sloppiness only serves to confuse the public and obscure the very real problems we will face when a real tropical storm approaches.

Anyway... my point - other than frustration with local reporting - is that the Corps New Orleans District continues to delay fixing the London Avenue canal walls for what can only be described as insane reasons. We've already had New Orleans' pumps in PS#3 turned off twice (that we know of), and have been close a third time. How much more of this idiocy do we have to tolerate? Do they really, really want a levee breach? Just drive the damn sheet piling, and fiddle with the testing or whatever later. For God's sake, it's not hard.

By the way, repairs on the 17th St canal, meant to bolster the east walls (read the article, it sounds like more than just bolstering) also seem to be delayed. I've watched that section of canal every day, and I've seen no progress at all. I'd be very surprised to see that work done before June 1. So much for urgency.

Copy. Paste. Repeat.

2007-05-01T13:51:00.000-07:00

UPDATED: see bottom of post.

I've been hammering away at the original bid specifications for the outfall canal pumps quite a bit. That's because they are one of the root causes for the dilemma in which we now find ourselves. They are also a legal document, which also carries all kinds of implications.

I had admittedly been proceeding under a somewhat polyannish assumption regarding those specs, based on my own experience in writing technical specifications. I assumed that the personnel in the Corps' New Orleans District who crafted those specs did so based purely on the technical requirements for the job, and that a lot of work went in to them. I assumed they were working from a considerable base of knowledge about the specific equipment they were writing about. That's always been the case on my end, as well as for any other engineer I've ever known.

Well, that's apparently not what happened. As reported by the Associated Press, the Corps used MWI's spec to write the original bid solicitation from January, 2006. Well, that's understating it a bit. The Corps and its archtectural/engineering contractors (Linfield, Hunter & Junius and the local office of URS) copied and pasted MWI's catalog specification word for word, sent it out as a competitive bid solicitation, and then awarded the job to MWI. When he found out about it, Louisiana Senator Vitter asked the Corps what the heck was going on in this April 13, 2007 letter.

Among the revelations in the article is that the testing requirements which were downgraded by the Corps during testing in Florida (as documented in Ms. Garzino's memo and the independently authored reports attached to that memo) were actually MWI's own testing specs. They couldn't even meet their own spec!

However (and I hate to mention this) there's even more to the story.

On June 9, 2006, the Corps put out yet another bid solicitation, for 23 more hydraulic pumps (16 at 17th St, and 7 at London Ave). While there were significant changes from the January spec, the new solicitation still included large, word-for-word chunks of MWI's catalog specs. It was postponed (on June 15, 2006), and then cancelled (on June 20, 2006) before it could get to the contract award stage. At the time, the Corps said they cancelled it because one of the potential offerors (i.e. bidders) was threatening to sue. This was mentioned in a New Orleans City Council meeting on June 22, 2006, for which I have the tape. When I asked why the unnamed bidder was threatening to sue, I never got an adequate answer, just a lot of handwaiving.

Now we know why. It is because the other potential bidders were sick and tired of seeing MWI's spec in print underneath Corps letterhead for the second time in six months.

But wait! There's more...

The June spec was pursuant to the May, 2006 Revision 2 of the Corps Project Information Report (PIR), which made an unconvincing case that hydraulically driven pumps were the best idea for expanding pumping capacity at 17th Street (the PIR Rev. 2 doesn't seem to authorize expansion at London Ave, but the Corps still asked for seven pumps there. Go figure). Lineshaft pumps - in which the engines are directly connected to the pumps via a gearbox (i.e. no hydraulic fluid, no piping, etc.) - were deemed to not meet the Corps' requirements. The logic in PIR Rev. 2 boiled down to a very weak argument for why hydraulics should be used instead of lineshafts:

1) Hydraulic pumps would have a delivery time of 12-16 weeks, while lineshafts would have a delivery time of 16-20 weeks.
2) The cost per cubic foot per second was less for hydraulics than for lineshafts.

There was no detailed discussion of the technical merits of lineshafts over hydraulics (which are substantial), the long experience of major lineshaft manufacturers, or why the particular style of pump was even an issue. After all, the question on the table in May, 2006 (and all throughout 2006) was how to get more pumps out to the floodgates. Why it would matter whether the pump was driven by a direct drive or a hydraulic drive was irrelevant, and pretty anti-competitive.

In fact, the PIR Rev. 2 seems geared toward one objective: finding any argument to justify the use of hydraulic pumps. After all, "cost per cfs" - rather than just overall cost - is a strange way to measure the cost of a system which will only be running for a tiny fraction of the entire year. Who cares if it costs a little more to build and run the system? Isn't the real objective to get as much water out of the city as possible?

By the way, on that most important count - by the Corps' own admission - hydraulics wouldn't have done as good a job as lineshafts. They said they'd be able to get an extra 400 cfs from the lineshafts vs. the hydraulics (in the PIR, they proposed 12 lineshafts @ 300 cfs (3600 cfs) along the west side of 17th St, while they called for 16 hydraulics @ 200 cfs (3200 cfs) along the west side. Both plans called for two more hydraulics on the east side. Those two hydraulics were eventually installed). Yet they still proceeded with hydraulics.

As far as the other part of the Corps' argument that delivery times were four weeks faster for hydraulics, I am extremely suspect of where they got their delivery estimates. How do we know that they weren't also getting that information from MWI, and not lineshaft manufacturers? Also, why did it take so long to get the PIR Rev. 2 and solicitation out? After all, everyone knew what was needed: more pumps. And they knew it right from the moment the original solicitation went out in January. If they had put the revised PIR and second solicitation out a month earlier than they did, their argument about faster delivery times would have been moot. It just doesn't track.

What I'm saying is that the argument of hydraulic vs lineshaft seems to be a setup to make a particular company look good. It shouldn't matter what the drive arrangement was, and the bid solicitations shouldn't have made any mention of how the pumps would be driven. As long as the pumps hit the flow and head performance requirements, what else mattered? That is, why not open up the process to everyone, rather than narrowing it to a field of about three companies?

So how do we know that there was no necessity to pick a particular drive arrangement? Because the last revision of the Corps PIR that we have, Revision 4 (from October, 2006), contains no such discussion. Revision 4 is the justification for the current set of pumps now being placed at 17th St and London Avenue at a combined cost of $135 million. Those pumps are of a lineshaft design. And Revision 4 makes absolutely no mention of hydraulic vs. lineshaft. In fact, the word "hydraulic" doesn't even appear in Revision 4.

So we have the Corps New Orleans District from about November, 2005 until at least June, 2006 acting in internal documents and conversations like they are in love with hydraulic pumps (specifically MWI's pumps). But they were also receiving messages from other bidders and their own project engineers that there were serious problems with the hydraulic pumps (both on the technical side and on the contracting side) during that same time period. But they still plowed ahead. I think the message finally made it through in June (when the second solicitation was cancelled), but the reassurances to the public still kept coming, even after the disastrous testing in August, 2006 when the pumps nearly shook themselves apart.

It is that total lack of transparency that is tremendously galling. For most of 2006, long before the public was aware there were problems, the New Orleans District was totally aware of what was going on, but they never let any of us know. And even today they put out their bland reassurances. Their credibility has always been in the toilet, but at this point I'd say it's been flushed into the river.

UPDATE:
For my readers in Florida, there's a reason this may all seem familiar:

Primed for success

According to the St. Petersburg Times, practically the same Kabuki show played out in 2001 and 2002 in Florida, with the South Florida Water Management District playing the role the Corps is now playing. Specs were apparently rigged to favor MWI, who won the jobs.

Tip of the hat to da po blog. Go there to read a little more.