Contracted to fix the fiasco, IBM estimates costs at $1.25 billion: sources. There’s even talk of divorce, after just 3 years of corporate marriage.
By Don Quijones, Spain, UK, & Mexico, editor at WOLF STREET.
As the blame game intensifies over beleaguered UK lender TSB’s never-ending IT nightmare upgrade, now in its 12th week, relations between the bank’s management and that of its parent company, Spain’s Banco Sabadell, are beginning to sour. According to sources quoted by the FT, things are so bad that TSB executives are even considering whether it has a future as part of the Spanish group after a bungled IT upgrade resulted in operational chaos:
At the heart of the row is the dominant role that Sabadell’s in-house IT provider Sabis played in the botched IT upgrade which left hundreds of thousands of customers unable to access their bank accounts and piled pressure on chief executive Paul Pester.
Mr Pester told MPs in a letter last month that the bank was looking to “insource” its IT infrastructure from Sabis. Such a move would sever TSB’s main operational link with its owner and make any future separation easier, three people close to the bank said.
“If Sabadell sold, that would clearly be for the best,” said one.
It’s a damning indictment of just how bad things have got at the UK’s seventh largest bank. The IT debacle has already been branded the “biggest IT disaster in British banking history.”
Hundreds of thousands of people have been left unable to access their bank accounts. Countless standing orders, payrolls, mortgage installments and other payments and transfers have failed. Thousands more have fallen victim to fraud attacks. Even when the bank tried to apologize, it managed to send the wrong apologies to the wrong people, in the process breaking the EU’s new data protection laws.
The prime cause of all of this mayhem was Sabadell’s determination to get the new IT system up and running as quickly as possible, in order to save millions of euros in monthly fees it was having to pay each month to TSB’s former parent company, Lloyds Bank plc, to use to use its old IT system (our articles on this fiasco). An insider at the bank alleged that the disastrous roll-out of Sabadell’s Proteo4UK system at TSB had been eminently foreseeable, and yet the management went ahead anyway, adopting a hope-and-pray attitude.
In a letter to MPs TSB’s CEO Paul Pester laid much of the blame on Sabadell, whose “written attestations” about the readiness of its new IT platform were “instrumental” to TSB’s decision to proceed with the migration of customer data. But as a report released in late June confirmed, TSB and Sabadell were anything but ready.
The report featured a brief presentation by IBM, the firm appointed by TSB to identify and resolve its IT problems once the crisis had begun. In its preliminary work plan Big Blue’s technicians suggested that the bank’s testing was not up to scratch, saying it had not seen “evidence of the application of a rigorous set of go-live criteria to prove production readiness”.
Underscoring the scale and complexity of the project, IBM said that a firm would need “longer than normal to prove the platform through incremental customer take-on to observe and mitigate any operational risks.” It also cautioned that such projects often bring up a dizzying array of hard-to-diagnose technical and functional problems.
“To address this risk profile, IBM would expect world-class design rigour, test discipline, comprehensive operational proving, cut-over trial runs and operational support set-up,” it said. There was very little evidence TSB took such precautions. While TSB’s CEO Paul Pester told a parliamentary committee in June that the issues at the bank were with middleware, the IBM team detected additional problems with custom and package applications as well as the network.
A source close to the matter told WOLF STREET that IBM’s estimate of the cost of rectifying the issues and accounting errors was £955 million ($1.26 billion). This excluded the fraud instances as they are not regarded as being part of the IBM remit, and are being treated as a normal banking function of fraud prevention.
The source, whose statements could not be independently verified, told WOLF STREET that the migration was started before the final Lloyds backup runs were complete and verified; and given the time gap between the “last known good” backup and the cumulative transaction data to date, TSB was told that there is no question of a roll back, even if Lloyds would agree, which they have indicated is out of the question.
IBM believes that the bandwidth calculations for the UK-Spain data link was out by 50% and did not take in to account that when the TSB customers were allowed to start using the system, the migration of a large part of the historic data was still ongoing, the source said. Sabadell IT staff allegedly throttled the bandwidth available to the TSB Internet Banking clients in order that the migration was not interrupted, but a router failure/malfunction in Spain meant that a large part of the historical data lost their packet ID’s which scrambled portions of the data. Throttling the bandwidth would explain the delays that internet banking clients experienced in getting on the system, the source said.
According to the FT, tensions are already rising between the management of TSB and Sabadell. The two banks have only been together for three years, and they’re already talking of divorce, albeit largely through unnamed sources. Investment bankers in the City of London have been speculating whether Sabadell would be willing or able to keep hold of the business in the long-term.
Sabadell would be hard pushed to find another bank willing to take TSB off its hands for anything close to the €2.35 billion Sabadell paid for it in 2015. Even at the time of the acquisition, experts warned that Sabadell was significantly overpaying for TSB while underestimating the potential costs of integrating the new business into its existing IT platform. Some also cautioned about the potential risk posed by a vote in favor of Brexit, which has since come to pass.
A source close to Sabadell suggested it would be reluctant to jettison its UK business. Its purchase of TSB was a key part of the bank’s plan to pare down its reliance on its domestic Spanish market. “It always needed an acquisition to make it sustainable,” on industry figure told FT. “They definitely cannot buy anything at the moment.”
TSB represents roughly a quarter of Sabadell’s total assets, and the impact of a forced sale on Sabadell’s own financial health could be considerable; but so, too, would the bleeding of funds from TSB’s unending IT meltdown. By Don Quijones.
It’s payback time for the financial sector that was bailed out by taxpayers, Spain’s new government thinks. Read: Banks Squeal as Spain’s New Government Threatens to Do Unthinkable: Raise Taxes on Their Profits
Enjoy reading WOLF STREET and want to support it? You can donate. I appreciate it immensely. Click on the beer and iced-tea mug to find out how:
Would you like to be notified via email when WOLF STREET publishes a new article? Sign up here.
As they say in Spanish “Lo barato sale caro.”
That means that buying cheap or trying to save money by going everything to cut costs ends being more expensive that if you had not tried to save money to start with.
That’s what happened here.
What they should have done is the same thing many business do when moving to a new system. They keep the old system working for a year or two as they move stuff to the new one just in case of screw ups.
Just like companies used to keep and some still do, paper records beside the digital ones. In case the digital ones got screwed.
One of TSB’s major problems was that their IT system was actually owned by Lloyds Bank, who were selling TSB. TSB were paying a hefty fee to keep using the Lloyds system and there was a considerable financial incentive to move off the Lloyds hardware. From the date of sale to Sabadell, TSB were a “paying guest” on Lloyds system and Lloyds were a serious competitor to TSB in the UK banking market.
Lloyds had tried to build out a new IT system for TSB to enhance its sale prospects, but had abandoned the build because the costs were getting out of control. Sabadell thought their own fairly new IT system was “the dogs gonads” hence the disaster unfolding.
It is sad to see TSB in their current state as they were a good bank and their IT systems when Lloyds bought them, were streets ahead of the other UK banks in that they were among the first to offer “real time banking” and had a good niche market in same day money transfers for the house sale market which meant that most law firms had accounts at TSB.
Good banks don’t give their IT department the order to rush out work that if not done well spells disaster.
Is all about money isn’t it?
Had they delayed things a few months this chaos would not have happened.
Basically, nobody knows if this bank is still solvent because there is no way to know under the circumstances. Employees could take home bags of cash and nobody would really know it’s missing. Maybe that was the point, to blame the eventual failure on the IT dept.
They have to keep it running for nine more months.
Then they can blame BREXIT for the fiasco, pop their golden parachutes and sail off into a cushier and more rewarding position- destroying something else.
IBM knows a lot about IT screwups – like their Phoenix project in Canada. TSB couldn’t even pick a credible IT consultant.
SONY is still way worse. And let’s not start with Yahoo…
IBM – Indian’s Badly Managed
Idiots Briefing Management
A preview of coming attractions under our rapacious corporatocracy, where grotesquely overpaid yet clueless CEOs strive to maximize “shareholder value” (and their own compensation) by reckless cost-cutting, customers be damned. The result will be screw-ups on a vast scale and extremely angry customers – though whether the latter will finally get angry enough to vote out the corporate stooges who enable and abet such despicable greed and avarice, and put in true reformers, remains to be seen.
I am criminal according to the AUSTRALIA GOVERNMENT I like the freedom of using cash .As the above article proves relying on digital monies can only lead to chaos as no one can guarantee access 24/7 .Perhaps the whole digital financial system is the week point take down the internet, your ballistic missiles, air craft carriers etc are obsolete .
I love my cash too BUT you do know you have cash because the ATM dispenses them to you no? That’s electronic. Let’s say you don’t use ATM and go to the teller, you do know that the teller can give you money because the SYSTEM tells them you have a positive balance?
In other words, unless you store your cash under the mattress, you still rely on some electronic system?
My mattress IS lumpy Times are tough.
And to think there are still people rooting for AI and an even more complex and inter-connected world. What can go wrong? (hint, it usually does)
I mentioned a few months ago we switched out all our insurance coverage due to reaching voicemail one too many times with the agent. Tonight, I pulled my 40 year old Honda Trail 90 out of storage. I had it tucked away for over 1 year with a tank full of premium and some fuel stabilizer. A few kicks and vroom vroom vroom. I also have a 38 year old ct 110. Simplicity = resilience. If my computer goes down I’ll just go back to reading more books.
Good article DQ. It’s getting wobbly out there.
Wow. Gasoline should be less problem, but how did the oil, and battery fare?
I once had a trail 90, a 70 and a 50. Tough as nails, they are.
Does anyone believe the $1.25 Billion estimate? I don’t. Maybe double or triple that to get in the ballpark…
The usual ploy is to come in with a low estimate, low enough to get the contract. Then you work that estimate upward as you start digging and finding more problems, or the client wants to add this or that feature which wasn’t mentioned in the original spec.
Snark aside – I don’t think what you are suggesting is at all out of line in this case.
There is flat out no way a due diligence team could accurately assess how long and how much the cost is going to be to fix this Crap Show. So on something like this, you propose doing enough to find a path forward and make the client aware of the fact it’s all time and materials. You give them the tools to make a business decision to go or quit at each step. (With the added condition, that on something like this it’s pay up front before starting.)
—
My company does a lot of process/system recovery for various clients. Clients that have managed to “lose” their version control repositories and *all* backups, or that have fired all of their subject matter domain experts and scrapped their build servers… Only to find that they need to do another software release… Because %75 percent of their revenue is coming from some legacy project… (Oopsie…)
With the continued infiltration of MBA cockroaches into US businesses, you’d be surprised at how common this is. (Or maybe not.) Strangely enough we never seem to see these kinds of problems with our European or Asian clients.
Nothing we’ve ever been called in to work on approaches the size of this mess though…
Your estimate is probably low. ; )
They big question is: Is it possible to unwind the “scrambled” data at all? No one has mentioned whether or not the “scrambled” data was retained or dropped when the router “malfunction/failure” occurred.
Small rant:
This is what happens when Accountants take over IT. Everything becomes a line item on an AP ledger. Use the lowest possible cost workers because 3 for the price of one means more work will get done. Whether it gets done correctly and well is never addressed is this model.
There is a statement in IT.
1. Good (System itself)
2. Fast (Time to delivery)
3. Cheap (Cost of system)
Pick any 2, you’ll never get all 3 at the same time.
Good + Fast: It won’t be cheap.
Good + Cheap: It won’t be fast
Fast + Cheap: It won’t be good
I don’t have to guess which model they decided to use… ; )
They were running up costs of “millions per month.” But, now that they tried to rush changing the systems, they are facing an overall cost of over a billion to fix the mess.
Given those numbers, a good accountant would have been emphasizing caution and avoiding any risk of switching systems too early. Its far better on the accounting books to pay ‘millions’ for a few more months to avoid paying ‘billions’.
However, the sort of boneheads who make it into management these days would only see the ‘millions’ per month and demand a quick changeover. Probably after reading some ‘management book of the month’ about being a ‘decisive’ leader. The sort of thing that says silly stuff like make a bad decision quickly and decisively and with strength is better than a delay to make a proper decision.
Well, It *is* better not to think too much and too long, if the decision is reversible.
The problem is that these jokers cannot lose, either they get the golden parachute or they get the bonuses – often both – and there is always opportunities for the “smash & grab” management.
So for them, why waste a single calorie at all on thinking? It’s all win-win!
Driving the bus off a cliff at full throttle is nothing when “the invisible hand” is there to catch you. The passengers may not like it, but who cares about losers?
There are six phases to an IT project
Phase one – enthusiasm
Phase two – disillusionment
Phase three – panic
Phase four – seeking out the guilty
Phase five – punishing the not guilty
Phase six – rewarding those who made no contribution.
We are approaching the climax of phase three and the cast for the next phase are quietly taking their places
LOL. IBM is known to create mess rather than fix it, but nobody was fired to choose IBM. Had a misfortune to work with an ex-IBMer, so no surprise; lots of self-confidence backed up by incompetence . Good luck to Sabadell.
Bigger messes = Bigger billing; It’s a feature.
The price tag of 1B+ is ridiculous. Ten good programmers with current banking experience can fix this, but it will take management budding out, which is why it won’t happen the right way.
Agree. Once management gets in the way, the problem just becomes bigger. Especially when they are under the gun.
“Ten good programmers with current banking experience can fix this”
I wouldn’t be too sure about that. I’m betting the database is fundamentally corrupted. Software could be working just fine, just acting on bad data.
And, if that is the case, the only thing you can do is help folks find a new bank and close your doors for good.
Determining DB corruption is the easiest part. First you fix the software so that you can “replay” a day of data you can confirm. This means replaying a day before the disaster. Then you can apply transactions day be day until you get them all done.
The problem with this scenario is that you may get to insolvency before you finish processing all the transactions. But if “insolvency” is not relevant then you can ignore it, like they are already doing.
Kent,
I’m with you. The DB is toast. “the migration was started before the final Lloyds backup runs were complete and verified”. The word “complete” gives it all away.
Given that they started before the backup was complete, the new DB was already corrupted before the first transaction was executed. Allowing the system to run for days (or weeks,) just turned a big problem into a worst case scenario.
And with everything I’ve read about this, it’s likely that the software has been mishandling data as well.
I think it was in the book The Mythical Man Month https://en.wikipedia.org/wiki/The_Mythical_Man-Month) that it was postulated that good programmers are about 10 times as productive as an average programmer. In my experience the break is closer to 40 to 1.
Even if you could get 10 “Rock Stars” that could work well together, I don’t think you could right this ship.
Most bank/finance SW is built on layer after layer of old legacy code. Understanding all of that legacy baggage and moving it forward onto another platform is not a trivial task in the best of circumstances. Given that the underlying data is probably corrupt now, the legacy process/data is likely lost, and the new process went out the door half baked… There is no quickie way out of this mess.
I’m betting the best path forward is for the British Government to liquidate the institution and make the customers whole as of the day of the transition while transferring them to different banks. <<< I know nothing of the regulatory environment in Britain, so I have no idea if something like this is possible.
One of the primary reasons why Spanish Banks were interested in buying UK banks was that the UK banks traditionally have much higher deposit ratio’s for their client base. These higher levels of savings gave acquiring banks access to a big pool of virtually free money to help fund the banking business.
The BoE placed restrictions on Santander using UK deposits to fund their global ambitions when they first arrived in the UK as the BoE were nervous of global banks with uncertain governance structures and regulators in Spain who allowed Spanish banks to rapidly expand on very thin capital cover.
The new software is so massively deficient that it had to be obvious to all levels of workers and management at all levels of Sabadell. In a socialized banking system like we have today, they simply didn’t have to care. They will be bailed out by various governments regardless of their incompetence or criminality. The saddest thing is that with the accounts so mangled, the people who haven’t already gotten their money out are unlikely to ever get the money or get compensated for it. Since they can’t prove what is in their accounts the regulators will be unlikely to pay.
There is a Wells Fargo bank near me and people apparently still have accounts there. Despite it having been demonstrated that they are basically a criminal enterprise. Do you think if their depositors accounts “accidentally” got mangled like those at TSB that the FDIC would pay? I sure hope not.
All of which goes to show that no one ever went broke underestimating the intelligence of the people.
This makes me glad I always keep my last 3 years printed statements. A way to prove what is there and what was spent if ever needed.
A big mess for which the cure will be to sweep losses under the carpet.
A read about an euro insurance company that sold bonds decades ago that if redeemed under the terms would make the bond holder owner of the company and all it’s assets. Not sure if it still trades publicly (I believe it does?) but wow, existing shareholders could be wiped out overnight.
The backup technology must have been stone age. And only one copy?
I would think that the databases at Lloyd’s and TSB are different. So one cannot just do an “sql-dump” and “import”. They are likely huge data sets, so one has to break them down somehow into chunks that can be checksummed and resent-on-error in reasonable times.
So, To move the accounts, someone had to first write a compiler, which compiles the records stored in the source database into transactions that create the same records in the target.
A possibile shortcut is to use the API for managing transactions in the target system and use that rather than mess with data-base to data-base migration. With that approach there is only one compilation stage, that of creating the calls for the API.
But. Maybe the API wasn’t ready. Maybe some bright young thing thought that we can just do the sql-dumb, then some script can just patch the dumped sql and it will be much faster since the API has rulez and checks and sequential entries. They figure they can fix the API in parallel, every PM loves that! Maybe then they go live while this quick-fix script is running and now we have to things updating the same records. This is not so good!
Anyway, it is not a trivial job. Putting time as the primary objective above integrity is not the right way to go about it.
This is a really troubling situation and it is far beyond the competence of bank regulators to deal with. Imagine if TBS had been a SIFI!
I note that Google, Uber, GM and other self driving car developers must carefully evaluate and test their systems and NONE have been granted approval to be operational. Surely a bank should demonstrate any new IT system is fully functional before it is allowed to transfer its records and rely upon it.
A canary in the coal mine perhaps? I’m actually surprised that major IT breakdowns do not happen more frequently given the endlessly patched legacy systems and outsourced support.
The question arises what should one’s personal strategy be for ensuring assets don’t evaporate during someone’s IT or security snafu? Spreading out your assets across multiple financial institutions and keeping good records would be mandatory.
I think it is not so much that TSB was having to pay LLoyds Bank for use of Lloyds’s system so much as TSB wanting to introduce new ideas for its customers without Lloyds having advance warning or being able to veto the idea because Lloyds didn’t want to change the system.
In UK competition amongst banks for custom is intense. Early days but the new Metro Bank could give TSB a run for its money.