“As the incident is ongoing, the full scope, nature and impact of the incident is not yet known”: AutoNation today.
By Wolf Richter for WOLF STREET.
When the $8.3 billion acquisition of auto-dealer software provider CDK by a PE firm under Brookfield Asset Management was completed in July 2022, the mergers & acquisitions firm, Paul Weiss, which had advised CDK on the deal, said in a now ironic press release: “The deal will allow CDK to continue to elevate the dealer and consumer experience when selling, buying or owning a vehicle.”
Less than two years later, last Wednesday, CDK’s customers, including the biggest auto-dealer chains in the US, watched helplessly as a ransomware attack shut down CDK’s cloud-based software system, depriving all of its customers – 15,000 dealerships in total – of the most basic daily tools to run their new and used-vehicle sales operations, their parts and service operations, their inventories, their back-office operations, customer contact systems, loan applications, etc.
Dealers have resorted to writing up sales orders and service orders by hand, then hand-typing all this into spreadsheets or whatever, to track it somehow, hopefully not making the situation even worse by adding typos into VINs, repair order numbers, names, and other key data. Then, someday, when the system is up and running again, they hope to re-type – or maybe copy and paste? – all this from spreadsheets into the CDK software system, praying all along the way to not make the situation even worse by introducing more typos into key data.
The publicly held auto dealers – there are not many, but they’re huge, with lots of big dealerships around the country – have started to warn about the still unquantifiable consequences. And this could ripple across the economic data for Q2.
AutoNation [AN], the largest dealer chain in the US, said in an SEC filing today that it had been notified on June 19 that CDK, “was experiencing a cyber incident impacting its systems, including the systems necessary to support our dealer management system (“DMS”), which supports our dealership operations, including our sales, service, inventory, customer relationship management, and accounting functions.”
It said its stores remain open, “and we are continuing to sell, service, and buy vehicles, and otherwise serve our customers, through manual and alternative means and processes, albeit with lower productivity.”
“As the incident is ongoing, the full scope, nature and impact of the incident is not yet known,” it said.
Group 1 Automotive [GPI] said in an SEC filing today that “all Group 1 U.S. dealerships continue to conduct business using alternative processes until CDK’s dealers’ systems are available.”
“CDK has advised that it anticipates the restoration of the dealer management system will require several days and not weeks. The timing of the restoration of other impacted CDK applications remains unclear at this time,” it said.
“Group 1’s ability to determine the material impact, if any, of the CDK incident and the resulting service outage, will ultimately depend on a number of factors, including when, and to what extent, the Company resumes its access to the CDK’s dealers’ systems,” it said.
Lithia Motors [LAD] said in an SEC filing today, “The Company, whose dealerships continue to operate, has implemented mitigation plans to minimize disruptions and continue serving its customers. While this incident has had, and is likely to continue to have, a negative impact on the Company’s business operations until the relevant systems are fully restored, the Company has not yet determined whether the incident is reasonably likely to materially impact the Company’s financial condition or results of operations.”
Sonic Automotive [SAH] said in an SEC filing on Friday, “All of the Company’s dealerships are open and operating utilizing workaround solutions to minimize the disruption caused by this CDK outage.”
“As the incident is ongoing, the full scope, nature and impact of the incident, including the extent to which the threat actor accessed any customer data, are not yet known,” it said.
“While this incident has had, and is likely to continue to have, a negative impact on the Company’s business operations until the relevant systems are fully restored, the Company has not yet determined whether the incident is reasonably likely to have a material impact on the Company’s financial condition or results of operations,” it said.
Penske Automotive Group [PAG] said in an SEC filing on Friday that its consumer-brands of dealerships were not using CDK’s software, and were not impacted, but its 48 heavy-truck dealerships were. The commercial truck dealership business – selling primarily Freightliner and Western Star trucks – “has lower unit volumes than the automotive dealership business and principally serves business customers,” it said.
CarMax [KMX], the largest used-vehicle dealer in the US, said during its earnings call on Friday that it does not use CDK and wasn’t directly impacted by the hack, but that it works “with a lot of other dealers” to get parts to repair vehicles, and if their systems are down, there would be a “minor” impact on CarMax, and “there is a little impact on title work as well. But I would say it’s just minor in the scheme of things as far as the impact on us,” CEO Bill Nash said.
Impact on economic data for Q2: The industry is now heading into the last few days of the second quarter for purposes of the closely-watched reporting of deliveries of new vehicles to customers. Deals that got hung up could prevent the vehicle from being delivered by the quarter’s cut-off date, which may ripple across all kinds of Q2 economic data. Vehicle retail sales are an important factor in consumer spending.
CDK shrouded itself in opacity about the nature of the hack. Emails to dealers have called this event a cyber incident and cyberattack. On Friday, Bloomberg reported that this was a ransom attack. On Saturday, CDK admitted that it was recovering from a “cyber ransom event.”
Today, Bloomberg reported, citing the security firm Recorded Future, that the attack had been undertaken by hacking group BlackSuit. “The cybercrime group has demanded an extortion fee in the tens of millions of dollars from CDK, which plans to make the payment,” Bloomberg said.
There is still no information if the hackers were able to get the data of the effected dealerships’ customers, such as the data on applications for car loans.
Enjoy reading WOLF STREET and want to support it? You can donate. I appreciate it immensely. Click on the beer and iced-tea mug to find out how:
Would you like to be notified via email when WOLF STREET publishes a new article? Sign up here.
Rumour has it the Fed was hacked today – of course they can just print the ransom ha ha
That was a joke on X, in a satirical post. Some people took it seriously, it seems.
Not if the hackers demand payment in bitcoin.
How is bitcoin obtained if not by mining? Buying with cash, which the Fed can print to obtain.
Our economy was hacked in 1913, when the Federal Reserve Act was passed by CONgress.
Now we just have Fiat currency to show for it, not real money…
“Give me control of a Nation’s money, and I care not who makes it’s laws”-Amschel Mayer Rothschild
Another PE special, lol.
I’m certain they’ll figure a way to turn it into more PE profits … maybe an new “dealer saftey impact fee” to be passed on to buyersm
The end of Wolf’s article notes that the company plans to pay the ransom fee.
It should be illegal for any company to give a penny to cybercrime groups, as payment just encourages further crimes of the same nature. And the criminal penalties for engaging in such crimes should be increased to draconian levels, including the death penalty.
White collar crimes have been punished far too leniently for far too long. A mugger on the street stealing $20 would be considered worse than these cyberthugs operating in the millions. It’s time for a more sophisticated view of white collar crime.
The damage done by “blacksuit”all dressed up, ready for the funeral of CDK.
What happens if blacksuit takes the money and doesn’t return the cookies, then CDK would have to start all over from scratch.
Then blacksuit does it again next month.
It’s not in BlackSuits interest to withhold data once paid. They will continue extorting companies. They want a record of “you pay, we will restore your data”.
What happens if all cyber hacking was AI (US security state algo fulfilling US foreign policy by) feeding on new data sources to maintain its control of the narrative (and data bank thirst) which it has become addicted to since the world has become addicted to the new nectar called the internet which is governed by US registration law (driven by US foreign policy), and which the Assange news narrative today hands a legal precedent to US global legal jurisdiction over-reach to citizens on Earth. Now carbon social credits can be enforced on everyone.
The public activity of our public institutions paid for by the public is hidden behind security clearance, while private data belonging to private citizens is paraded for all to see. The incentive and inducements to be peaceful citizens is being taken away. That is a heck of a gamble for a very narrow group of people to take with billions of people, because there is nothing in it for the people. Only takes 5% of people to cotton on to do something about it and make the difference.
Let’s see; Penny pinching, blood sucking upper management (in this case coupled with equally devious PE firm) makes crappy decisions leading to security breach. Everyone with any amount of knowledge of the systems has probably been let go or has left, to be replaced by the upper management specials; cheap H1 laborers working for a pittance. I hope even after paying the ransom the systems still don’t come online. Popcorn time.
Amen. This is surely cost cutting by replacing natives with H1Bs and general ignorance of the customers of how security worka
I can’t wait for the computerized A.I future where computer software and robots will be running everything from robot cars, to robot food service and robot health care. Good thing they will have these systems perfected (cough cough) by then so that Hackers won’t be able to disrupt what will then be the essentials of life.
Utopia awaits!
I can’t open that door Dave…
What are you doing Dave?
“…Dave’s not here…”
may we all find a better day.
This conversation can serve no purpose anymore…
I know I’ve made some mistakes Dave, but I can do better
Can’t log in? Have a nice cloud day, Dave.
If CDK has smart software developers, they’ll use chatGPT to whip up an app to support dealer personnel entering the data into spreadsheets or SQLite databases.
Note that VINs contain a check digit, which could be used to catch typos during entry (ask chatGPT, “Do Vehicle Identification Numbers contain check digits?”, and it will give you the algorithm).
If they had smart developers this would never have happened. They probably had developers cutting and pasting code from ChatGPT and Stackoverflow without understanding what they were actually doing, leading up to this mess.
CDK Support Cares
We stand behind our dealers by offering the highest level of support and care while providing them with a foundation for success. Our team’s experience and expertise in the automotive industry breeds confidence in knowing that, when problems arise, you always have the right people working hard for you behind the scenes.
No post from CDK can make up for the complete failure of your IT department. Hiring the best comes at a cost. Your company is seeing the beginning of the end
I second your post. Me thinks it even possible that the IT department requested more tools/hardware for security and it was denied.
Hooters closes 40 locations – as restaurant crisis deepens
Hooters is the latest popular restaurant chain to abruptly shutter locations. Like others, it is blaming rising costs for rent and food.
That place is STILL around? Good lordie. How did they survive this long? Time to clean up the restaurant scene and get those moldy leftover relics outa here.
They had a dynamite fish sandwich on the menu (local Hooters). Of course, this is Houston and the Gulf is rich in great seafood and fish. It was located right across the parking lot from a Discount Tire shop….get tire work done while enjoying a cold one at Hooters!
That one near me closed a couple of years ago, great fish sandwich and all.
Hooters and fish sandwich, ha! Surf Punks…I Live For The Sun: “Listen, listen bud. Don’t be, don’t be a dud. Take your girl by the hand, Fresh fish pie in the sand. And baby, baby, we all live for the sun! Sun. Sun. Sun. Sun. Wooo ooo ooo ooo ooo ooh!”
No Wolf, all chain restaurant concepts are killed by inflation or other circumstances out of their control. Why the Warwick RI Hooters blamed a bridge outage (they said collapse, the bridge did not collapse) for their demise, even though they were located next to the states only major airport.
The Hooters on Merritt Island in Florida closed many years ago due to inflation: in the belly size of the waitresses. Management forgot what made the place an attraction.
Totally agree Wolf:
around here, the ‘brass and fern’ theme joints have outlived their place. For everyone closing, almost two new decent joints are opening.
“depriving all of its customers – 15,000 dealerships in total – of the most basic daily tools to run their new and used-vehicle sales operations, their parts and service operations, their inventories, their back-office operations, customer contact systems, loan applications, etc.”
All of one’s eggs in one basket. Got a backup plan? Oh, yah, AI…fixed!
CDK is what is known as a “Jesus app”; like in Jesus, I hope that sucker never fails us!
Hmmm, what a coincidence. For the last couple of weeks I have been learning about “ARM TrustZone”, “MCUboot”, and Cryptography. Maybe this Boomer can stay relevant for a few more years.
What in the world is Harvey talking about I’m not a relevant boomer. Auto cyber attacks having an effect on consumer spending as quarter ends !! Wonder if the algorithm bots will have that built in when they trade in the announcements periods next month. Still not fixed after a week?
Hi BS ini,
I’m talking about all of the IOT (internet of things) devices that are out there. For example, home security cameras that you can access remotely through your phone or a computer. Another example would your home thermostat that can be accessed remotely. Swimming pool equipment etc. Big companies have machinery that is monitored remotely. It’s a huge industry and growing fast.
A lot of these IOT devices are vulnerable to cyber hacking/attacks.
@Bs ini,
“What in the world is Harvey talking about I’m not a relevant boomer. ”
I should have written “Relevant to the field I have been working in for the last 35 years”.
Cash is back on the menu boys!”
“It said its stores remain open, “and we are continuing to sell, service, and buy vehicles, and otherwise serve our customers, through manual and alternative means and processes, albeit with lower productivity.”
Redundant systems are the cost-nemesis of the quest for complete financial efficiency (at least in the short run)…
may we all find a better day.
Yeah….I suppose it would have been kinda cool if achieved, and everyone had the time to get to be assigned a “1” or a “0”, (or even a debit or credit) before the space ship starts leaking all over and the passengers get real surly.
Don’t pay the ransom. Suck it up and rebuild the platform with better security. Of course that might cost some money.
What we don’t know is what happened to the data, and if clean backups of the data exist. The dealers need access to their data, to all past transactions, to all accounting entries, their service data, etc. They’re completely screwed without it. And no one wants all this customer data to be sold on the dark web.
Dealers can switch to another supplier of a dealer management system, such as Reynolds and Reynolds. But they must have access to their data and migrate it over. I’m sure there are lots of discussions like this right now. Protecting the data is the big issue right now.
Interesting point on data integrity . Platforms reading the data vs data storage . I thought data storage was all on platforms like AWS Oracle SAP and other large data servers either cloud or stored locally . I had not thought about the data aspect other than personal info data pulled from these databases via the software platform
It is highly likely the company doesn’t know how the system got hacked in the first place. Was it the way they wrote the software? A bug in the web server that someone else wrote? A bug in the database server someone else wrote? A bug in some 3rd party accounting or reporting libraries? A bug in the operating system someone else wrote? A bug in the underlying virtual machines from the hosting platform? What if it is an internal attack from one of your own programmers?
It may be impossible to tell, and certainly impossible as long as your systems are encrypted. And the idea that you can cease operations, recreate your systems from scratch and start over is nonsense. All of your customers will have moved on to your competitors and no one will trust you again.
You have to pay the ransom. It’s a crime against your customers, shareholders and employees if you don’t. While you can’t guarantee ransomware attacks can’t happen, there are ways to minimize the risks to make it extremely unlikely, and quickly recoverable. But here’s the almost impossible part: you have to have an information security guy high enough in the power structure to ensure the company is spending the necessary money to keep the systems safe. That’s something that is necessary in the modern world, but something they don’t teach in MBA programs.
I imagine it’s a risk vs reward metric for these big companies.
One company says it’s better to take a chance that the system fails and save money running fast and loose with our security. They might go years on cheaper IT and software development while the company that starts out taking the necessary precautions has their lunch eaten and goes out of business. Lest we forget the “good IT” department can still fail.
Cyber warfare and cyber crime is the way of the future. This will all become quite commonplace and a daily occurrence soon.
Trucker – spot on, if discouraging, observation (and would posit that future is now…).
may we all find a better day.
As a Luddite, I don’t worship the cloud based systems that most people do, and its incidents like this that me extremely afraid of ai adoption — and the dystopian future that’s going to be based on a combination of ai hallucinations, cyber crimes and hype.
It’s actually funny that possible solutions will rely on the soundness of an old fashioned spreadsheet that’s safe and sound, with a long history of accuracy and reliant predictability.
This is also impacting tractor dealers. My New Holland tractor has been at the dealership for over a week now. I called them up and they said they were part of the CDK hack and couldn’t access the work order that was created before the cyberattack. This is a medium sized tractor dealer. I am guessing that this issue is bigger than just auto dealers.
Thanks for the info. Yes, not just auto dealers. Penske’s heavy truck stores are also impacted by it, as noted. They can’t do anything either. Got your Freightliner in the shop? Well, please be patient.
For something that seems to run every aspect of the car ( plus tractors, boats ATv’s?) it seems like it should have a cooler name than CDK. How about taking a page from Elon and name it after a historic computer guy like Teller or Watson. Or name it after a historic race car driver like Fangio, or Lauda, or Schumacher. CDK sounds like the storage media for Kpop songs.
“CDK” is cooler than “Reynolds and Reynolds,” the system we used back in the day. They’re still around today.
Reynolds and Reynolds just makes me think of aluminum foil and cigarettes. The desire for efficiency and interoperability that has led to these all in one services has really fattened the goose for criminals like this. It’s no wonder single attacks can take down such large swathes of a market.
It makes me think of a book I read when I was a kid featuring an island inhabited by lizards, all named Reynold.
“Reynolds and Reynolds”
There needs to be an Its Always Sunny joke here but I’ve got nothing
I love it. Dee and Dennis crash the Automotive Industry
Is that God forbid, “Ryan Reynolds” the actor?
If you make your money by providing software, how do you NOT have offline backups with plans for wiping your computers and reinstalling images in hours?
I am a one person company and I have offline backups that get updated every week. I worry more about hardware failures but if somebody did manage to get into my servers, I would wipe and rebuild.
It’s 2024; the threat of ransomware has been around for years. Any sysadmin worth his/her pay should have on- & offsite backups in anticipation of 98% of failure conditions (which would include ransomware exploits). This was a disaster waiting to happen. Three years ago I’d feel sorry for any industry this happened to; today, it exposes sloppiness.
Jim, I would wager most (if not all) Sys Admins are trying to get on- and off-site back-ups, but in nearly every case get trampled by the operations and accounting leadership that the cost of a good back-up solution is ‘a waste of money.’ That is…until something like this happens. Then again, “I told you so” doesn’t go down well in the middle of a crisis.
Agreed, like how was that not a given!?! I’m a small business too, and I back up my data in excel spreadsheets. Maybe we are just more paranoid than most???
This is what makes me think inside job. Disable backups too mebbe.
This. I run a small business with a custom CRM. I backup the database nightly and the rest of the server weekly. I can have a new server up and running in a few hours.
Good job. On-site and off-site backups are key. But the backup procedure needs to be tested regularly to make sure it works when needed. Very few companies are rolling back their primary customer data to the last backup on a regular basis. But without that you do not know if your backups are working.
Hardigatti:
Yes, backup and test. Also a need to freeze/ wipe the active platform and restore it from scratch is needed.
The reality is that machines have already replaced the thinkers and creators of machines. There’s no manpower to “do it right” when we just need to Git R Dun!
My wife has a small business that was happily cruising along, with an app based platform and service provider.
That platform was sold/ went away. The “replacement” is a large company with no plan or provision for the services previously offered by the old vendor.
They don’t care, and rather just close the sale of a new monthly subscription, with no real service to offer. When she was in contact with another business owner in the space the response was: “I used (New Company) before (BEST PLATFORM), and they sucked then.”
All this to say the profitable path is to make a useful tool and sell the factory and customer list. Why waste time on building resiliency?!?
F the “customer,” I just got PAID!
American cars (vehicles) suck. They just suck! They suck sooooo bad!
What kind of lowlife deviant is buying from these Pukes anyway? They should all go under and we can all save some money and buy better quality. Seriously, dude…
If you don’t like them, take the Metro? Other people like them. About 16 million people will buy a new one this year because they like them so much.
It just dawned on me that I have not seen an email (or text) from my local Chevy dealer asking that I fill out an online survey for my oil change a week or so ago. They were coming in like clockwork……and I never bothered to respond.
I’ve just thought of a new business idea.
In honor of all those survey requests I keep getting, I’ve decided to (cough-cough) rebrand myself as a Professional Survey Responder.
Want my opinion? Well, senders of surveys, you’ll have to pay me for it. My rates start at $100 per survey.
My wife and I were at the local Lincoln Service Department this morning for a long overdue recall repair. They didn’t want to commit to any other work this time because their parts database was hacked.
First time “extra” work wasn’t highly suggested.
Would be interested to know if CDK or any of the dealers had cyber insurance. Someone will have to pay, and part of it should be management at CDK and management at car dealership organizations who should pay with their jobs. Having worked almost 20 years in cybersecurity I would be willing to bet there were serious shortcomings in the cybersecurity programs at CDK. I would like to see their last pen test results, if it even exists. And the car dealerships management didn’t value their customer’s data and outsourced the control of one of their most valuable assets thinking they could save some money. I wouldn’t be surprised if CDK outsourced the development of their software to third parties to save money. Almost every cloud implementation I have seen has been setup with security holes. AWS and MS Azure rush products out to get a first mover advantage to gain market share and lock customers in, security comes second. Just like Microsoft did with Windows – sell a product with security wide open so it was easy for unsuspecting novices to set up.
Buy American they make a great product! Love my suburban and dodge ram both have over 150k miles and are terrific
My dang hospital has been offline since last week in Little Rock. I don’t even think they were hacked. Today they called and asked me to bring all my printed medical including surgeries and prescriptions to my appointment since they can’t find it in their system now. OMG.
Another software monoculture upended by a virus. Yes, we will have no bananas again. First with Gros and now with Cavendish. How many nukes use the same software or the electric grid downstream? You got punched, how is your recovery plan working?
The secret of certain types of cybersecurity?
Keep it old school.
I have heard (legends) that certain weapons control systems remain just as built: completely offline and using 1970s-1980s technology.
Unhackable!
Also: unsupportable! I have a bit of experience with aging control systems and what’s required to update them. Let’s hope that 5.25” floppy still loads!
They could ummm write it all on paper? Just a thought.
Sooo, seems like this story should be all over the head lines… nadda mooch. Bet it would be if the ransom was being paid in bitcoin.
It was. I heard about it not from this site, but from my local radio news station.
How does a ransomware attacker get paid without leaving a trace?
Do they get a bag filled with $30 million cash?
Are funds wired to an unknown Swiss bank account?
Bitcoin?
If done electronically, it seems the funds have to be transferred to a registered account somewhere.
You got it on the third try. Bitcoin.
It’s one of the few genuine organic use cases for cryptocurrency, along with drug dealing and human trafficking.
Wait until they have ransomware on the software that actually runs your car. You go to start your car in the morning to get to work and you need to pay some guy in N. Korea in Bitcoin in order for your car to run. It will happen.
Perhaps older cars could be ok? Updating my car’s ECU requires connecting a USB to OBDII dongle to my computer – there’s no wireless functionality.
Wait till you pull the lever and the screen in the booth says your vote will count if you send bitcoin to someone in China.
We’ll never know, but I really would like to know how much of the blame for this event comes down to decisions that were made by the new Private Equity ownership.
I have no doubt they immediately focused on cost-cutting to squeeze more profitability out of their acquisition.
IT Security is expensive and often involves policies and procedures that slow down business processes.
I wouldn’t be at all surprised if there were people/processes that were eliminated which eventually led to a softer target.
This, AND this!
The myth of Crypto is “privacy” when the reality is that it was developed to “never forget.” Each transaction is a part of the chain.
The reality of ALL tech is that it’s the “new paper.” Quick, efficient and effective. Transient, degradable and able to be falsified.
The hackers have put in the work on their end to be able to receive and move the ransom effectively and covertly. The HACKERS have more respect, regard and knowledge of their True Resource than the PE firm.
The hackers are highly skilled programmers who have (likely) been burned by the kinda’ group that bought CDK and fired the programmers!
Oops, thought it was also connected to
Bobber’s
Comment about receiving the ransom.
I work for an IT services company that CDK has hired to do work for them. Without giving away too much, I can tell you they are spending like crazy to get this working again.
They have hired us and other companies to stand up entirely new environments, groups to work on restoring from backups, and even a group hoping to decrypt the hacked systems. Money train is in town!
…makes one wonder how much was ‘saved’ by not seriously investing in SWOT-analyzing and in security early on…(oh, well, SO easy to neglect with SO many casino doors to walk through…).
may we all find a better day.
Maybe after 100 more of these attacks companies will start to take cyber security seriously.
And wouldn’t it be wonderful if our amply-funded international law enforcement would start doing something about this crap. I realize most of the actors are in foreign lands but that doesn’t mean we have to just do nothing.