Global IT Outage reported. (AKA 'Old lags reminiscing)!

gramps

gramps

Suspended / Banned
Messages
44,805
Name
'Gramps'
Edit My Images
No
I wonder if this will turn out to be yet another example of "There's no need to test that". :whistle:
 
Thankfully so far today I've not noticed any impact (company systems are hosted on Azure platforms so I was early checking they are stiill up). My home computers are all ok at present (mix of Win11 and Mac).
 
iMac here is fine so far.
Sky News offline on the TV but otherwise seem unaffected. Harbinger of things to come I guess!
 
It seems like it is software that only goes on critical systems.
 
It seems to be one possible cause is, if I heard the name correctly, due the roll out of a Cloud strike security update.....rather than a malicious 3rd party 'actor'.

Affecting companies & corporate structures who rely on both MS and Cloudstrike for secure services :thinking:

On the face of it, it does not and will affect end users computers!
 
Last edited:
None of our local shops can accept credit card payments.
 
JonathanRyan said:
More likely to be a case of "we sacked all the testers"
Click to expand...
Or, as I've experienced many times, senior management with no IT experience said they wanted it out NOW.
 
Craikeybaby said:
It seems like it is software that only goes on critical systems.
Click to expand...

Large enterprises use various products that get deployed out to their end user systems to lock them down and enforce network security and access - user laptops and desktops, kiosk systems.

Most of these organisations use Windows rather than MacOS. My experience of working with several large organisations - and only one used Apple - and even the Apple network had third party security software installed (their enterprise network was just as clunky and irritating as the Windows ones in my opinion).
 
AndrewFlannigan said:
I wonder if this will turn out to be yet another example of "There's no need to test that". :whistle:
Click to expand...
In my experience of software development, possibly a case of a senior guy not knowing how to write a unit test case and then says 'My code will always run so I won't write unit test case on it.'
 
Dryce said:
Large enterprises use various products that get deployed out to their end user systems to lock them down and enforce network security and access - user laptops and desktops, kiosk systems.

Most of these organisations use Windows rather than MacOS. My experience of working with several large organisations - and only one used Apple - and even the Apple network had third party security software installed (their enterprise network was just as clunky and irritating as the Windows ones in my opinion).
Click to expand...
In my day we used Novell Netware and didn't really care what the user had on the other end, apart from the obvious security concerns. These days, now I am retired, I am firmly Mac, although for my volunteer job I have to use a Widows machine. Thank goodness I no longer have to administer them.
 
Faldrax said:
Or Sales Managers "I promised the customer they'd have it today, so you must release it now".
(and it's never the sales managers fault if there are then faults reported, they still get their bonus...)
Click to expand...
On one project, we were four or five days away from release when we found a problem with a piece of third party software. The team leader went off to a meeting with the managing director and the financial director. Half an hour later he came back, looking stunned.

"Don't tell us, they won't let us hold back the release", came a voice.

Our team leader shook his head slowly. "No. They said we could take another month - they'd rather have it working than not". Took us all a quite a while to recover from that sudden attack of commonsense! :wideyed:
 
40 years ago, I was involved (in a VERY small way!) with a (then) ground-breaking ROM for the BBC Micro and its release was held up for about a year while extensive debugging went on. These days, it seems that any product is rushed to the market and Joe Public is used to find any bugs.
 
Nod said:
40 years ago, I was involved (in a VERY small way!) with a (then) ground-breaking ROM for the BBC Micro and its release was held up for about a year while extensive debugging went on. These days, it seems that any product is rushed to the market and Joe Public is used to find any bugs.
Click to expand...
I guess the bean-counters are very much running the show today.

Back in BBC Micro days, maybe the techies had a bit more authority?

That said, at that time (mid 80s) I was working on a large and very complex automated post-assembly calibration and audit system for solenoid fuel injector production and we got a bit bogged down in the debugging stage. In the end for financial reasons, we had to ship it and finish the debugging on-site during the commissioning phase of the programme.
 
May not be related to the current outage, but yesterday I attempted to use a NatWest cash machine. Soon after it took my card, it started to reboot itself. Took a few minutes to come back up but still wouldn’t give me my card back, so had to cancel my card. While it was rebooting I could see it was running on Windows 7 .. for which support is no longer available from Microsoft. I was a bit shocked !
 
Kevin Allan said:
May not be related to the current outage, but yesterday I attempted to use a NatWest cash machine. Soon after it took my card, it started to reboot itself. Took a few minutes to come back up but still wouldn’t give me my card back, so had to cancel my card. While it was rebooting I could see it was running on Windows 7 .. for which support is no longer available from Microsoft. I was a bit shocked !
Click to expand...
If it's on Win 7 it's one of the newer ones, XP was the OS behind the majority of UK ATM's for a long time.
 
Ed Sutton said:
Cash is still King. :D
Click to expand...
Unrelated to the current problems, but when on holiday in Australia last year, there was a near total power failure in the small town we were in - none of the tills worked, as they were all electronic, but the shops could still use card payments (after adding up the total via pen and paper) as the card readers were all battery powered wireless devices, and the town WiFi was about the only thing that had a battery backed power supply.
 
Anyone want to buy a fiver or two? £10 each.;)
 
The big winners will be the lawyers
 
An ex-colleague of mine was on the BBC this morning as a talking head. Simple case of an update being rushed out without proper testing. Probably because they were all made redundant or massively overworked due to staff shortages (caused by said redundancies) and quietly quitting.

Everything is run on a shoestring these days. This is the thin end of the wedge I tell you!
 
Harlequin565 said:
An ex-colleague of mine was on the BBC this morning as a talking head. Simple case of an update being rushed out without proper testing. Probably because they were all made redundant or massively overworked due to staff shortages (caused by said redundancies) and quietly quitting.

Everything is run on a shoestring these days. This is the thin end of the wedge I tell you!
Click to expand...
Ah well, AI will sort it all out...:rolleyes:
 
I didn't get up until 1pm
 
Harlequin565 said:
An ex-colleague of mine was on the BBC this morning as a talking head. Simple case of an update being rushed out without proper testing. Probably because they were all made redundant or massively overworked due to staff shortages (caused by said redundancies) and quietly quitting.

Everything is run on a shoestring these days. This is the thin end of the wedge I tell you!
Click to expand...
It's capitalism. It is what makes the Western world go round.

Keeping the costs down so we are all to keep more of our money!

What's not to like?
 
Kevin Allan said:
May not be related to the current outage, but yesterday I attempted to use a NatWest cash machine. Soon after it took my card, it started to reboot itself. Took a few minutes to come back up but still wouldn’t give me my card back, so had to cancel my card. While it was rebooting I could see it was running on Windows 7 .. for which support is no longer available from Microsoft. I was a bit shocked !
Click to expand...
Not related. At least not directly. Win 7 and Server 2008r2 weren't impacted.
 
Nod said:
These days, it seems that any product is rushed to the market and Joe Public is used to find any bugs.
Click to expand...
That's Agile isn't it? Invented in the mid-90's, the methodology that you deliver to live whatever has been developed by a certain date, and test in production. Obviously it doesn't say that on the tin, but it's what happens. It's actually HMG policy now "Agile First". I am very familiar with a certain govt dept that does a software release EVERY THURSDAY on a critical system, and at least one in every three releases introduces new, often critical, bugs to the service offered, despite having a large test team in place. The developers decide what tests to do because they write the "user stories" instead of actual users or real business analysts.

Outsourcing QA to India? Yes, I've worked on two projects where the client did that. In both cases I had to build a team to do the real testing after the outsource company had run the automated tests that they had created. We always found major bugs that needed fixing, but often the software was already in production.
 
I guess that there will be some litigation around this, at least in the USA.
If it is the case then it will probably bankrupt the responsible party.
 
lindsay said:
Obviously it doesn't say that on the tin, but it's what happens. It's actually HMG policy now "Agile First". I am very familiar with a certain govt dept that does a software release EVERY THURSDAY on a critical system, and at least one in every three releases introduces new, often critical, bugs to the service offered, despite having a large test team in place. The developers decide what tests to do because they write the "user stories" instead of actual users or real business analysts.
Click to expand...
I once found myself doing error logging and statistics reporting on an Agile project, along with another chap.

He greeted me on my first day with "welcome to the box". He'd been on site from the previous Monday and had already worked out that the developers wanted nothing from us but green lights and reports of Unicorn sightings. We soon put them right on that!
 
jknights said:
I guess that there will be some litigation around this, at least in the USA.
If it is the case then it will probably bankrupt the responsible party.
Click to expand...
...which will just cause further chaos.

I think the whole thing could be summed up in three words: "Irish potato famine". That was caused by creating a monoculture and when a fungus-like microorganism (Phytophthora infestans) took hold, the country's entire food system collapsed, crashing the economy.

Allowing a single organisation to define the majority of our IT infrastructure is an equally foolish mistake. Governments should long since have legislated to prevent this form of monopoly. The question now is: will they start rolling back the Microsoft monster and encourage, or if necessary force, major organisations to diversify their IT, so as to provide some resillience against such problems?
 
AndrewFlannigan said:
...which will just cause further chaos.

I think the whole thing could be summed up in three words: "Irish potato famine". That was caused by creating a monoculture and when a fungus-like microorganism (Phytophthora infestans) took hold, the country's entire food system collapsed, crashing the economy.

Allowing a single organisation to define the majority of our IT infrastructure is an equally foolish mistake. Governments should long since have legislated to prevent this form of monopoly. The question now is: will they start rolling back the Microsoft monster and encourage, or if necessary force, major organisations to diversify their IT, so as to provide some resillience against such problems?
Click to expand...

The tax dept in SA switched to Linux quite a few years ago.
It had a few problems but looked like it was going to work.

Then MS came along and offered to switch it back to NS for no charge and minimal licence fees.

At the time google would find quite a lot on it, I bet it won't find a trace of it now !

MS licensing for education ang government departments is about 25% of the retail price.

Then you have a whole industry of support companies, who are becoming more and more vital as MS makes it harder and harder for the end user to do anything, so they are kept in a job and of course support MS.

The product has never really been much good, but the marketing keeps it going.
 
AndrewFlannigan said:
...which will just cause further chaos.

I think the whole thing could be summed up in three words: "Irish potato famine". That was caused by creating a monoculture and when a fungus-like microorganism (Phytophthora infestans) took hold, the country's entire food system collapsed, crashing the economy.

Allowing a single organisation to define the majority of our IT infrastructure is an equally foolish mistake. Governments should long since have legislated to prevent this form of monopoly. The question now is: will they start rolling back the Microsoft monster and encourage, or if necessary force, major organisations to diversify their IT, so as to provide some resillience against such problems?
Click to expand...


The problem is that alternatives for some key elements are either garbage, don’t have mainstream support, or simply don’t exist.

If you’re on Windows you’re not forced to use Microsoft’s Office suite, its endpoint protection, management platform or its cloud services, in fact over the years these have become increasingly separate due to the push into the Mac and Mobile spaces.

Forcing organisations to diversify their IT is simply a bad idea, yes it should be encouraged and tbh any IT guy worth their money would always look for “best of breed” solutions and not put all the eggs in one basket, and let’s not overlook that this is fundamentally a Crowdstrike kernel driver screw-up, and not directly Microsoft at fault (the public are unlikely to see it this way)
 
Neilc28 said:
The problem is that alternatives for some key elements are either garbage, don’t have mainstream support, or simply don’t exist.

If you’re on Windows you’re not forced to use Microsoft’s Office suite, its endpoint protection, management platform or its cloud services, in fact over the years these have become increasingly separate due to the push into the Mac and Mobile spaces.

Forcing organisations to diversify their IT is simply a bad idea, yes it should be encouraged and tbh any IT guy worth their money would always look for “best of breed” solutions and not put all the eggs in one basket, and let’s not overlook that this is fundamentally a Crowdstrike kernel driver screw-up, and not directly Microsoft at fault (the public are unlikely to see it this way)
Click to expand...
I'm wondering if Crowdstrike managed to screw up their own systems...?
 
Neilc28 said:
The problem is that alternatives for some key elements are either garbage, don’t have mainstream support, or simply don’t exist.

If you’re on Windows you’re not forced to use Microsoft’s Office suite, its endpoint protection, management platform or its cloud services, in fact over the years these have become increasingly separate due to the push into the Mac and Mobile spaces.

Forcing organisations to diversify their IT is simply a bad idea, yes it should be encouraged and tbh any IT guy worth their money would always look for “best of breed” solutions and not put all the eggs in one basket, and let’s not overlook that this is fundamentally a Crowdstrike kernel driver screw-up, and not directly Microsoft at fault (the public are unlikely to see it this way)
Click to expand...

No, it is not MSs fault directly, but it was delivered through their update system, and that has caused many headaches in the past.

It also allowed FTDI to issue a driver update through microsoft updates that bricked and clone FTDI interfaces.
They attacked the innocent buyers, a move that was frowned on by MS and many people considered unethical.

However, it is possible for organisations to control updates to client machines with their servers, but I get the impressions this was a server update. Even then, they can be delayed, but that would involve extra time staff and money.

I'm sure there will be action behind the scenes though :)
 
You must log in or register to reply here.
Back
Top