Payment chaos: For bottom-line-obsessed bank executives, IT systems are an expense to be slashed. The results are in.
By Don Quijones, Spain, UK, & Mexico, editor at WOLF STREET.
Internet banking has become a crisis-prone business in the UK, as the online platforms of big banks suffer regular outages and other forms of IT disruption.
Friday morning, the online systems of the Royal Bank of Scotland, Ulster Bank and Natwest — all part of the RBS Banking Group — crashed in unison, leaving millions of customers unable to pay bills or view their balance on their online and mobile accounts. The group has 19 million customers in the UK and Republic of Ireland and 5.5 million active mobile app users.
After around five hours of chaos, the RBS Group announced that the problems had been resolved. The failure had apparently been caused by a “technical glitch” — a word that is being used with increasing frequency by high-street lenders — in a regular update to their firewall. The bank emphasized that it was an “access issue” and there is no evidence that customer data was compromised. But then, it would say that!
On Thursday, it was the turn of the UK’s largest bank, Barclays, whose website and telephone banking service crashed for around seven hours, leaving frustrated customers locked out of their online accounts.
Fed-up customers took to social media to vent their anger, with some complaining that they were unable to access their accounts not only through the Internet platform but also ATMs. Barclays has around 24 million UK customers, though it’s not clear how many of them were affected by the outage.
The bank told customers that they should still be able to make payments to existing payees through mobile banking, though new payees weren’t possible due to the incident. It also claimed that payments into accounts were unaffected by the issues.
One alarmed customer begged to differ, complaining to the BBC that a payment due into his account had gone missing, while another customer reported the systems inside branches being down, preventing customers from carrying out transactions even in the old fashioned, pre-digital way. By mid-afternoon, the IT “glitch” — that word again! — had been “resolved,” though no explanation has yet been given as to what caused it.
A few days earlier an outage at online challenger bank Cashplus, which targets people with poor credit histories, left customers unable to access their accounts, make cash withdrawals, or make or receive payments. The problems prompted Nicky Morgan, chair of the Treasury Committee, to ask Richard Wagner, chief executive officer of Cashplus, for an explanation of what happened and how victims of the outage will be compensated.
In other words, over the last two days, dozens of millions of UK bank customers have been locked out of their online accounts at different banks.
In terms of RBS, this is not the first time this year its subsidiary Natwest has suffered an outage. Its banking app went down briefly in April and in July a glitch with its card payments left customers unable to use their cards in shops or online.
RBS, the largely state-owned lender that has cost British taxpayers almost a hundred billion pounds in bailouts, losses, fines and legal fees, also has a rich history of outages, including a major blackout in 2012 that lasted for over a week, disrupting customers’ wages, payments and other transactions. The outage was allegedly caused by an “inexperienced” RBS tech operative’s blunder. For the duration of the blackout, the only means many customers had of accessing basic banking services was to visit the local branch.
That, however, didn’t stop RBS from embarking on a branch closure rampage, blaming the growth of internet banking for its decision to close one in four of its branches. Now, it can’t manage to keep those web-based services up and running, leaving customers even worse off. Even as the lender has increasingly digitized its services, it has consistently downsized its IT services team. In 2017 it revealed that it planned to axe 900 IT jobs by 2020 and is doubling down on its outsourcing of IT roles to India to reduce costs.
This underscores one of the major problems high-street lenders have with technology. They never treat it as a mission-critical aspect of their business, even as that business becomes increasingly dependent on technological solutions to stay competitive.
Bottom line-obsessed bank executives are always looking for cheap, short-term shortcuts to IT issues, with the result that lenders — particularly, but not only, in the UK — have for decades under-invested in their sprawling, creaking, accident-prone legacy systems dating back to the primeval age of COBOL and mainframe technology. And if some banks had been thinking about trying to finally move off their legacy systems and drag their IT platforms into the 21st century, the recent botched IT migration at mid-sized TSB, which continues to sow chaos 23 weeks after it was supposed to be ready, will not encourage them to do so. By Don Quijones.
Time is running out. March 29 is the deadline. Urgent action is needed. But it’s not happening. Read… Disorderly Brexit Would Trigger Mayhem in Derivatives Market
Enjoy reading WOLF STREET and want to support it? You can donate. I appreciate it immensely. Click on the beer and iced-tea mug to find out how:
Would you like to be notified via email when WOLF STREET publishes a new article? Sign up here.
Whatever caused all this is irrelevant, as banking increasingly goes online, people need to have relationships with multiple banks, cross border is also a good idea.
Not just in the UK:
https://www.mypalmbeachpost.com/news/hundreds-riviera-beach-employees-unpaid-deposit-glitch/f7WXBFydq9K9bfCxuFz6eL/
->Whatever caused all this is irrelevant
And yet, if you look closely, you’ll notice it’s actually the subject of the article.
->For bottom-line-obsessed bank executives, IT systems are an expense that needs to be slashed.
Executive bonuses, golden parachutes, and stock price manipulation take priority, but diverting resources into programs to ramp up money laundering and defrauding customers probably should have waited until they could keep operations from blowing up in their faces.
I’m the last person on earth that would defend the big banks, but IT is not the only reason for systems to fail. I have lived through 5 hurricanes and a major flood. I saw these disasters wreck the electrical grid as well as phone systems, separating customers from their money. Having some cash is always a good idea, as well as another form of access to some money.
BTW, the serial nature of these problems looks like an upgrade problem of some sort or even a deliberate shutdown to stress test the system.
Unlike natural disasters, IT failures are entirely preventable.
->I’m the last person on earth that would defend the big banks, but IT is not the only reason for systems to fail.
It’s usually incompetent management, the kind that slashes resources needed for IT.
These British banks can’t use hurricanes and floods as an excuse.
They could reasonably claim that they were distracted by wallowing in pools of laundered money. That at least would be honest.
->BTW, the serial nature of these problems looks like an upgrade problem of some sort or even a deliberate shutdown to stress test the system.
Which is to say, incompetence, like I said.
Unamused,
While there is incompetence in every large organization, don’t underestimate the idea of deliberate action on their part. It could be a deliberate shutdown.
The problems with IT in big CORPS is that they try to make omelets without breaking eggs or, said in another way, they try to spend as little as possible with IT projects. Normally big consultancy firms are hired to do complex projects on a very tight schedule and for very little money. Go figure what happens: very junior teams and lots of corners cut. Consultancy firms are there to make money and, with that in mind, they do what’s possible with the available time and budget.
I have a very good relationship with my mattress.
Oh dear, that doesn’t sound good does it?
But really a well-stuffed mattress has always been the peasant’s best friend, and that’s what we all are.
I like your cynicism.
My mainstay is CSC, cynic, skeptic and critic. While it may seem quirky it provides a few layers of protection in an unpredictable World as a pawn in the game.
The article somehow managed to avoid using the words “cyber” or “attack,” which is quite a feat considering what happened a few months ago:
“Seven UK banks targeted by co-ordinated cyber attack”
https://www.ft.com/content/2e582594-48ab-11e8-8ee8-cae73aab7ccb
Also note:
https://www.americanbanker.com/cyber-attacks
Maybe it should be have been “self-inflicted hack.”
Cyber attacks on banks usually try to steal data or money. They’re not trying to shut down a bank’s ATMs or online payment system.
..it takes time to blame the Russians, they like to do that all at once.
Firewall updates typically cause this type of problem. They admitted it was a firewall update.
If there is typo in the file of ports to open, or ports to close, then you’re screwed. I have seen this happen many times, and often no on can find the cause of the problem until someone sees that IDP port 13831 should have been IDP port 13381.
“Well, it worked in acceptance test!”
“Yes, the acceptance test server uses completely different ports!”
Maybe that’s why the acceptance tests should be done in a replica of the production environment. Not doing so is wrong and prone to problems
I’ve been a contractor at a giant software company for about ten years. Working there and stories like this continue to amaze. Junior level mistakes that are never learned from. While I still believe the old adage that one should never attribute to malice that which can be explained by simple incompetence, the constant repetition of the same old dumb mistakes makes me wonder. How deliberately stupid can you be?
This might be an early gift to remind the people that Brexit is a bad idea. Come exit day, everyone will get to suffer, this is just an early reminder.
You know, I think the only sane people in the world is the Swiss. Neutral in everyone, they are smart enough not to join anyone’s crazy boy band.
Sorry MCH what’s Brexit got to do with it? I seem to remember a Spanish banks incompetancy caused TSB to crash. As mentioned by DQ it’s all about bottom line, cutting cost on the most important things in a bank, people and service.
Have you noticed how these “IT crisis” are only happening to the British banks? Not any of the continental banks. It does make one wonder if the IT departments at these banks are more incompetent than their continental counterparts or if there is something else going on.
Now it’s probably a bit of a leap to say this has anything to do with Brexit. But the situation seems just a little odd. Regular old mismanagement seem to occur everywhere, see Spain, the Germans, and so forth. But the Brits are apparently incompetent when it comes to IT and banking. How odd is that.
We have frequent ‘glitches’ in Netherlands causing customers unable to access thejr bankaccounts for hours and even days.
@Joe.
Interesting, did not know this, could I ask if this is at the same degree and scale as what happens in the UK? Some how it seems to be a bigger deal in the UK, at least that’s what it seems like.
I just find a lot of this to be incredibly coincidental. Or perhaps its widely known in the country of origin, but not as much outside. I’m curious if DQ is aware of these issues that you refer to in Netherlands. I wonder how many of the other readers here are aware of this.
Hi MCH, every European Bank is cutting branches, people, number of atms and blaming it on more people using online banking. What the banks have failed to grasp is that in the long run IT costs will be no cheaper than keeping all the branches open. You’re correct though British banks are terrible ref I.T. Regards Steve UK
->This might be an early gift to remind the people that Brexit is a bad idea.
It’s actually a good idea. The problem comes in when the profiteers can’t agree on how to split up the racketeering operations.
Maybe it is elements from the EU who are doing this – sabotaging the UK banking systems – because of BREXIT – in an attempt to stop BREXIT – maybe it is elements from within the European Central Bank that are – in desperate fear of losing control of the other 27 EU member nations & thereby falling into oblivion – who have commissioned expert hackers to wreak havoc within the UK banking system ??
Maybe it is the current band of DIGITAL BANK ROBBERS who are enjoying the current profitable romp around the global banking system.
Like in the movies – they set off the alarm a few times to conuse everyone & then they hit the joint ??
On cash we trust.
IT is not an expense, is a necessity.
There is a reason I haven’t had a credit card since 2002 and is because those things get hacked and or cloned quite easily.
I haven’t done an online transaction in ages and I plan to avoid them as long as I can.
When my bank insist I am always “And if they steal my data then what?” That shuts them up.
->“And if they steal my data then what?” That shuts them up.
Nobody’s going to steal your data. Your bank will sell it and pretend they got hacked. Call it a business model.
The problem is that BOTH things happen. Not only banks sell your data, it also gets stolen.
Problem is, your data is already online, and essentially you are just having a teller do your online transaction for you while you stand there.
Have we noticed that we are run by computer systems ??
That mere mortals do not know we exist until we front at their premises to keep an appointment ….. arranged by their computer system.
Of late I have weekly hospital appointments to attend & it dawned on me ….. I only exist on their computer system ….. the human element has been freed up from the responsibility of knowing I exist until I front to be treated … & then I vanish into oblivion again.
Ok. Who wants to go 100% cashless?
The banks are just testing how people adapt to being denied it maybe. Seriously, in Orwellian terms you can imagine a psychological profiling of people’s reaction, social tolerances….
Bank’s Revenge
A story of reverse engineering and stress tests as applied to the population on their own behalf, a not for profit philanthropic adventure outsourced to somewhere near you now!
Don’t miss it, guaranteed to stir the emotions of even the most apathetic!
It’s blown up in their face, hasn’t it.
But —– do THEY know this
Because —– if THEY do not realise this fact
We are still on the drawing board to be screwed.
The new ‘British Disease’ – starve and outsource IT until it collapses?
British Airways has had several major outages in recent years as well.
I’ve had direct experience of British Air’s IT-related dysfunction. Their website simply does not work, or rather works just enough to suck you in. You begin any process online and always end up trying to complete it with customer service. Which means a nice young man in New Delhi, apologizing every step of the way, has to in turn struggle with their trainwreck of a system. But even he (his name is Addison–we are buddies) is eventually defeated and has to call and explain that the problem really isn’t solved.
There’s something a little ironic here. Greedy, incompetent Britain is still being propped up by all those workers resident in its now defunct former empire.
And now we know how it all ends, “not with a bang but a whimper”. The further the conversion of banking goes from cash to electronics the more fragile the system becomes. And while attention is focused on the financial problems in Spain and other countries, it would appear that the UK is losing its ability to maintain a modern digital infrastructure. After the lessons of the last few months, the Icelandic banking collapse which hit the UK very hard, and the upcoming “Brexit” (will it really happen?) you’d think thinking people in the UK would keep their money in their mattresses.
Some of us do but if you are reasonably flush it might be wise not to overdo it!
This happened a few months ago and folks couldn’t get across the Severn-Wye bridge between England and Wales because their debit cards didn’t work at the toll booth.
Appantlly, they closed the bridge because of this and therefore traffic had to use the other bridge, this simply diverted the revenue generated to the other bridge.
it may have been a local “IT Glitch” as I would have thought both bridges used the same system….
Recently a Sainsbury Supermaket local to me had same issue and couldnt take card payments.
I was in another supermarket once, the week before Chritmas and same thing happened. It was carnage, like their world had come to an end. Stampede to the ATM’s outside which then emptied/crashed in 5mins and then probably a 100+ trolleys full of food and perishable goods just dumped in aisles as almost everyone exited as they didnt have cash.
The banks provide easy card payments and people use it all the time.
My generation used to say Cash is King.
It was an interesting observation and it will become more and more part of life because there are now youngsters who have lived their whole life electronically and the following generations will hardly touch cash . After all, they cant have cash taking up space in their pockets as reduces the space for their Iphones and coins might scratch it.
This sounds like the death kneel against the bank’s war against cash. Hoist with their own petard. :)
Well, if the UK problem make you cringe this headline for an article in Business Insider should scare you to death: “…Verizon and AT&T are banking on 5G to connect every device on the planet to the Internet” . That’s not a petard, it’s an A-bomb! Sounds like the potential for the black swan event from hell.
I love reading this website because you find out that living in the Philippines and always having downtime with trying to access the internet is not just peculiar to yourself but you find out that people even in so called 1st world tech savvy countries also have the same access problems in this digital age.
I guess we are all really Third world.
When you are glued to the screen of your phone you don’t notice the similarities , so definitely internet access has an effect on how developed a country is.
;-)
He who controls the computers, controls humanity.
About 10 years ago I worked as an IT contractor for a state level Blue Cross/Blue Shield health insurance company and many of the same shenanigans occurred there as well. The CEO was highly compensated but outsourced all the IT infrastructure. The contracting company provided locally sourced infrastructure staff onsite, but relocated the majority of the software development to India. I was impressed by how inefficiently these IT systems were maintained. Not only that, but every single insurance company communicating with the parent BCBS organization had to duplicate so many different systems. Centrally managing all this at the federal level would have made so much more sense. Single payer anyone?
mkruger
“…Centrally managing all this at the federal level would have made so much more sense. …”
Uh huh. Look how well the ObamaCare rollout was managed (NOT!). How many decades behind the technology curve is the IRS? Uncle Sam can’t attract talented IT staff, and the technology funding process is a nightmare.
Just getting my 91-year-old father (WWII USMC vet) registered at the VA required multiple interventions from our congressman.
But keep thinking those good thoughts about “centrally managed”.
Your anecdotes do not prove that centrally managing things does not work. It just shows that the federal government has not done so. In many cases, they choose not to. The GOP makes the tax code more complicated, cuts funding for the IRS, then blames the government (that they are a part of) for the problem.
I just want to take a moment to point out something: Don Quijones is an excellent writer. I’m not referring to subject matter, which is always interesting, but to style. His prose rolls off the page as a relaxed conversation with an old friend. In a word: Enjoyable. I’ve followed him for years and I see his comment sections, once barren, are catching on. This is good. So pause and enjoy a bit of click bait every so often so he may continue “in his spare time he likes to watch films, read novels, play poker, listen to music, drink vermouth and loiter around Barcelona with his beautiful wife, La Doña.” Sounds like he has the lifestyle figured out. He certainly has a wonderful grasp of the financial world.
DQ also understands Spain very well indeed, which is quite rare among foreigners, who usually just like the sun and toss off lazy, superficial articles.
It’ll be interesting to see if Marcus[1] catches the disease after it sets up in the UK.
[1] https://www.bbc.co.uk/news/business-45599096. tl:dr, Marcus is part of the Goldman Sachs and is opening up in the UK next week.
My bank’s (SunTrust a super regional US bank) on line banking was down for a few days earlier this week. I had to scramble to get my credit card bills paid on time since I could not use the banks ‘bill pay’ service.
If an internal glitch can take down a large bank’s on line banking service one wonders just how vulnerable our banking system is to a more malign state actor deliberately seeking to crash our banking system?
Taking down Bank America or J.P. Morgans IT systems would cause more chaos than an enemy blowing up the Golden Gate or Verrazona Narrows bridges.
Let me guess…they still scrap terminal screens to converted to gui???
I am IT manager at large US bank. It is no surprise banks are closing branches – as our customers tell us this is NOT they way they prefer to bank. Numerous customer survey as well as actual daily transaction counts SHOW us that customer prefer digital banking over physical. In fact, over 70% of our customers had not been to a branch in > 60 days.
Expanding a banking institution’s size/presence in the market is no longer dependent primarily upon building branches. Branches will continue to play a role – however, a much less important role.
“creaking, accident-prone legacy systems dating back to the primeval age of COBOL and mainframe technology” – actually, those are the most reliable elements of these systems. The “accident-prone” aspect comes with all the “21st century” attempts.