But Spanish parent, Banco Sabadell, proclaimed triumphantly to its Spanish audience that the botched IT transition was a resounding success.
By Don Quijones, Spain, UK, & Mexico, editor at WOLF STREET.
As bank IT system revamps go, few have been as disastrous as UK bank TSB’s latest system upgrade, which began on Friday, was supposed to be up and running by Monday morning, but is still down on Tuesday evening. It’s now Day Five of the crisis and an estimated 1.9 million TSB customers still cannot use the bank’s online service. According to customer testimonials featured on The Guardian, some cannot even withdraw cash with their ATM card or their credit cards.
One TSB customer was reportedly given access to another customer’s £35,000 savings account, £11,000 Isa, and a business account when he logged onto his online banking account on Monday night — an allegation the bank denies. “There’s been no data breach whatsoever. I can categorically say that customers’ data is safe,” said the bank’s CEO, Paul Pester. Judging by the tone of some of their tweets and comments, the customers are not wholly convinced.
TSB used to belong to Lloyds Bank, one of the UK’s Big Four lenders that was bailed out with public money during the financial crisis. After later acquiring HBOS, itself the product of a failed merger between Halifax and the Bank of Scotland, European banking regulators deemed the combined banks to be too big and concentrated. So Lloyds decided to spin off TSB, which was duly acquired by Spain’s fifth biggest lender, Banco Sabadell, in 2015.
For Sabadell the deal represented its first major foray into foreign markets. But merging banks, especially across national borders, can be a complex business, especially when the IT systems involved are completely different.
In the case of TSB, its IT system was clearly well integrated with Lloyds after almost 20 years of co-existence. But since the moment it was bought by Sabadell, the Spanish lender has had to pay Lloyds rent to use its old IT system. The cost of this “transitional service agreement” skyrocketed in 2017 from £91.8 million to £214 million — the equivalent of almost £10 million a month. Even for quite a big bank like Sabadell, that’s quite a lot of money — hence the rush to get it replaced. But as they say here in Spain, haste can kill (la prisa mata).
The project was initially scheduled to be launched in November but it wasn’t ready by that date. The extra delay cost Sabadell over £80 million. Now the project has been launched, serious doubts are being raised about just how ready Sabadell’s homemade, brand new, largely untested IT system really is. If it isn’t ready, that could be a very big problem, especially if, as some have posited, the changes can’t be rolled back.
This is one of the biggest bank account migrations Europe has ever seen. The upgrade involves moving 1.3 billion customer records over to a “state-of-the-art” (Sabadell’s words) in-house developed banking platform called Proteo4UK. It is also the first time a Spanish bank has attempted to develop a new platform from scratch for migrating customer records on such a scale outside Spain.
Sabadell’s management saw the launch of Proteo4UK as a historic moment. Even on Tuesday evening, when countless TSB customers were no doubt wondering how they were going to get hold of the cash in their account or make bank transfers, Banco Sabadell had a triumphant press release on its website proclaiming the IT upgrade as a resounding success, a message that was in turn echoed by certain Spanish media.
“Migration is an excellent starting point for the organic growth of the business and the improvement of TSB’s efficiency,” crowed Group Chairman Josep Oliu. The move represented “a new milestone” in the banking group’s history, particularly with regard to its international expansion. The message was finally pulled at around 6:30 pm GMT.
The very best Sabadell can hope for now is that the IT issues affecting TSB get resolved very soon — as in the next 24-48 hours. Even in such a rosy scenario, this episode will end up costing the bank dearly, both in terms of money and reputation. There will no doubt be fines and compensation to pay. Some customers will presumably close their accounts and take their business elsewhere. In other words, more lost money, none of which is good news for a bank that already has “a weak balance sheet” with over 11% of its loans being classified as non-performing, according to Merrill Lynch analysts.
And that is the best case scenario. In the worst case scenario, there’s really no telling where this could end. If the bank is unable to keep an accurate ledger or record of customer product holdings, this could mushroom into a serious crisis, not only for Sabadell and TSB but also their respective banking sectors. Whatever does happen, one thing is clear: Banco Sabadell will need to launch a massive charm offensive if it wants to undo the damage of the last five days. By Don Quijones.
UK regulators may be on the verge of doing something right, but doubts remain over how genuine their stated intentions are. Read… After a String of Corporate Scandals & Collapses, “Big Four” Accounting Giants Face Breakup in the UK
Enjoy reading WOLF STREET and want to support it? You can donate. I appreciate it immensely. Click on the beer and iced-tea mug to find out how:
Would you like to be notified via email when WOLF STREET publishes a new article? Sign up here.
“The very best Sabadell can hope for now is that the IT issues affecting TSB get resolved very soon — as in the next 24-48 hours.”
HAHAHAHAHAHAHAHAHA said every IT professional ever.
Gteat article. Does a regulatory body check or approve the system? Would have been smart to get another entity to sign off on the new system :) Hope it goes well but not looking great. Cant wait for the follow up article; it’s a cliff hanger.
Well, in the face of a £10 million a month expense, any entity signing off would be encouraged to sign off very quickly. Probably be as meaningful as one of those modern corporate audits.
Generally accepted procedure (assuming this system went through successful pre-installation acceptance testing) would be to quickly (ie few minutes to a couple of hours) fall-back to previous version of system.
Looks like somebody tried to cut a corner.
And the prize for getting to the solution goes to Javert Chip. This bank has no one of your abilities on it’s payroll. Smashing job old chum!
If this is how a bank handles technology, i.e. not hedging risk, imagine how they manage lending.
From the outside, this scenario at TSB looks familiar to me.
I have worked in IT for 25 years, finally as a senior technical infrastructure consultant. One day I quit in disgust and instead bought an old dilapidated farmhouse in the French countryside that I’m restoring myself. The main reason for this career move was the increasing tendency of the suits to interfere with the work, playing political games, harassing the professionals with impossible goals and deadlines and finally blaming the techs when things went wrong.
I can all too easily imagine what went wrong at TSB, I can almost “feel” the familiar scenario of upper management putting pressure on hapless IT personnel to “get things running yesterday”. Caught between a rock and a hard place, there is no doubt that the techs cut corners here and there. Now that the chickens have come home to roost, there is also no doubt that the inevitable naming and shaming will emphasize the incompetence of said techs who bungled the execution of the brilliant strategy conceived by the geniuses in upper management.
I can only hope that this SNAFU will give pause to all those suits in all those IT projects all over the world and make them realize that simply putting more pressure on is not the solution for getting good results quicker.
So that my ex-collegues will be taken seriously when they warn of impossible schedules with a reference to “remember TSB…”.
Happens all over the world, bureaucrats think that their ideology trumps diligent IT process. When their ridiculous demands and ideology demands fail, the army of spin doctors blame the techs. It’s a disgrace.
This is a regulatory failure. Having worked swift and chips upgrades, the regulators perform system tests on major upgrades or migrations in multiple steps over time. Platform migrations in a large bank take days, just to move the databases, and parallel systems are used during the transitions, because failure is a possibility and expected.
In your world and ours.
Sabadell is a SPANISH bank.
It is simply a matter of how much extra it costs, to get those testing and compliance sign-offs, this month.
A little history on the Trustee Savings Bank.
http://www.solhaam.org/articles/tsb.html
I hope their customers reward them with taking their business elsewhere in numbers large enough to resemble a run on the bank.
Also working in FIRE IT, I see the same as Jos. Unfortunately not yet able to retire myself!
A failed bank, due to cutting corners, might be what could make the bigwigs realize that IT is at the core of their business, and not a cost center to be minimized.
Having been a customer of Sabadell in Spain for the past few years, I’m surprised at this situation. Their integration of the various IT systems of the banks they consolidated here seem to have gone very smoothly. Think the problem here is the quality of UK IT staff – they are not exactly renowned for success in IT projects – NHS anybody? Still it will no doubt please the Brexiteers to have somebody non-UK to blame
The buck stops with Sabadell, so to speak. That’s the parent company. That’s who initiated this IT transition to its own system. That’s who is responsible. Not some underling in the UK.
Wolf,
I don’t know how Dodd Frank affects IT banking transitions now, if at all. But before that, no large US bank would have been able to do a migration on this scale without regulatory oversight. Any system upgrade/migration that could take a bank down for one day would have endangered their “charter” to do business, which requires them to transact normal business during every banking day.
Having worked on large scale transitions for banks, I personally wrote more software for disaster recovery than for the transitions. The ability to rollback FUs was the order of the day.
This is a regulatory failure.
No1 rule in running migrations is do it on a national holiday weekend! It gives you an extra 24hrs to stop the warm and smelly hitting that rotating thingy on the ceiling.
In this migration you were moving a complete bank out of a system owned by a competitor, to another system where most of the tech staff would have been operating under Spanish management. There would have been areas of the Lloyds systems that they would not have allowed Sabadel to poke around in! I would love to read the DR plan!! Non-DR folk never could understand why it took 30% longer time to restore data than to back it up! hey-Ho
Petunia,
This is not in my bailiwick, but it seems you would have to be correct. I have heard many stories about how hard it is to merge bank IT systems after a big merger — that it takes years and is very costly. In addition, just about all bank IT systems are very old to begin with. But it seems regulations are trying to make sure the transition doesn’t lock customers out of their accounts for days.
I have my own story about bank IT chaos. This was in Belgium (early 2000s). I was running a startup, and having already experienced the relaxed attitude of Belgians about these kinds of things (Belgacom notified us that it would install new email servers and that it would shut down our corporate email for a few weeks … like you’re kidding, right?), I had set up accounts with the two biggest banks — Fortis and KBC — just to be sure. Then one day, I tried to pay some people online, and I couldn’t do it. So I walked to the branch, and everyone was sitting around with nothing to do. The entire IT system of KBC countrywide had gone done, they said, and no one could withdraw money, make transfers, or do anything. This lasted for days. Everyone was really relaxed about it. Just wait a few days, they said.
I cannot imagine the regulatory fireworks in the US if this happened.
Wolf,
A bit of banking trivia. I was the one brave/stupid enough to ask “What happens if we can’t open for business?” I was informed that the definition of a bank opened for business was a minimum of one bank officer taking deposits and handing out minimum withdrawals. This requirement could and would be met by one banking official, some bank guards, and a truck full of cash. There would never be a day when they were fully closed down.
UK NHS IT projects normally came unstuck when the civil servants in London decided that there should be a “national” system rather than hundreds of local stand alone systems. In the early days of NHS IT every hospital did its own thing in terms of hardware and software. The hospital consultants would often decide what software they wanted based on what they had used elsewhere.
The first NHS Global email system failed because the consortium who were given the contract tried to bypass Microsoft Exchange and Novell Groupwise and bought a small US company who had written an email program. When the Hospital Consultants found out from their Secretaries that there was no migration path for years worth of appointment data sitting in their Exchange/Groupwise system, the Consultants threw a wobbly to keep their secretaries happy, and the whole thing collapsed.
Hi cuius, as a brexiteer it doesn’t please me at all. It should have never been sold to a Spanish bank in the first place. As for Sabadell share price dropped from 1.93 euros to 1.63 euros in a month, not great.
The Spanish banks were very keen to get a foothold in the UK banking market. The UK retail banks have much higher deposit bases than many continental banks, and these deposit bases help when computing the Regulatory Ratios so beloved by Regulators.
When Santander first arrived in the UK the Bank of England placed restrictions on their ability to use UK deposits in other parts of their international network. The BoE were also concerned about the control being exercised by the Botin family who ran Santander but did not own a comensurate equity position.
Carlo M. Cipolla once said “Non-stupid people always underestimate the damaging power of stupid individuals. In particular, non-stupid people constantly forget that at all times and places and under any circumstances to deal and/or associate with stupid people always turns out to be a costly mistake.”
Now replace “Stupid” with “greedy”.
Clinical definition of a regime in the USA LOL
For an ongoing domestic IT disaster, check out the $360+ million “UHIP” (United Health Infrastructure Project) in the smallest state of RI. Apparently designed to integrate forty-eight state and federal public benefit programs, it has left stranded many of the area’s most vulnerable. There is talk of tossing the system and starting over from scratch.
Sadly, your contention that the Lloyds and TSB systems were well integrated was a little wide of the mark. TSB was originally “the Post Office bank” which had quite a reasonable IT system and was used for money transfers for house sales because they were virtually the first UK real-time bank. Lloyds thought that when they bought TSB that integrating the IT systems would be easy. It was not and essentially TSB Branches ran on their own IT system out of Andover, whilst Lloyds wrote some link software so that both systems could talk to each other, but only at a high level.
When the EU decided that Lloyds had to divest branches, TSB was logical choice after Lloyds looked at many permutations, but even Lloyds failed to design and implement a new stand alone IT system having spent over £1bn trying.
Sabadel’s IT system dated from around early 2000’s and that meant it did not know much about internet banking and smartphones. The migration of TSB seems to have worked regarding the “leaving Lloyds” bit but seems to have stalled on arrival at its new home.
Not sure if this falls under S.N.A.F.U , F.U.B.A.R or ClusterF*ck as they say in IT :
http://www.macleans.ca/news/the-phoenix-pay-system-disaster-explained/
GIGO.
It sounds like a modern-day bank run. As a depositer, suddenly I can’t get my money? Prove it to me a guy in a suit didn’t clean out my money and take off in an airplane to Switzerland. I’m nervous as hell about banks as it is, I think after something like this happening to me I’d only use the Bank Or Serta (Or Sealy or Beautyrest).
Good News…………..It is Thursday night in the UK and TSB have called out the IBM Crash Team to find out why the internet banking system is still not working correctly.
However……… a worrying point is that numerous customers are reporting that when they can access their account they are seeing random overdraft limits showing up on their accounts and random credits. If true, then TSB is in a lot of trouble!!
When you do any form of migration in IT, the normal errors tend to be routing and access to resources. If you want to move data from A to C via B you might get an error message saying “Can’t find B”. or if you are targeting a storage unit with multiple data streams, it might slow down to snails pace because you have underestimated the bandwidth needed.
What does not happen in migrations is that the data is “amended” in a random manner and data fields added. To add an overdraft limit to an account that is in credit, is not logical from a migration standpoint.
There was a virus many years ago which targeted Sage accounting programs by adding £1 to every number, every 24hrs. It was not difficult to find and you had to restore your data back to a date when you knew the figures were correct, but the psychological impact on the accounts staff was profound, because they stopped trusting the figures on the screen!!
What happens to a Bank where the Internet customers stop trusting what is on the screen???
If you are correct in your assessment (as a fellow with it banking experience I tend to agree) TSB Sabadell is in very deep trouble, as the balance and limit changes, when limited access is possible, indicate, Corruption or Virus.
Either will do horrendous damage, in such a data transition event.
If they can do roolback they still have to wait for the Migration to complete and fully reboot, first. As it appears to be flawed that could take a while.
Hi DQ, talking of banks I see Deutsche Bank is taking a beating today reference the share price.