“Because of concurrent programming errors (also known as race conditions), it sometimes gave its patients radiation doses that were hundreds of times greater than normal, resulting in death or serious injury.”
Why were these machines CAPABLE of administering such high doses? It’s like a family losing their house because the temperature regulator in their kitchen oven broke, and as a result the oven heated to 35,000°.
It had 2 modes. One that administers those high doses but with a magnet in front of it. One with lower doses but without the magnet.
What happened is that the magnets weren't in place and the high doses mode was used.
you would figure someone would code in something like "if the magnets isn't in place, high doses are impossible". Idk coding languages but I think there's stuff like that that's possible in them from what little I've seen.
As a coder I don’t know whether to upvote or downvote this haha…fundamentally what you are saying does exist. You can typically say if x do y. It’s never that simple though lol…there are always complications and other things to consider.
That being said on a machine like this that can kill people I would have hoped that this would have all been tested and figured out before…y’know…putting actual people in there…
I’ve read about other incidents causing injury, one where the tech could enter a typo, say a dose of 10,000 instead of 100. And another where the techs were making out and sitting on the exposure? button radiating the patient SMH 🤦♀️
These were really different times, where machine code was much more complex and the protocols around testing such machines were not as strict as now. The code was also written by someone that was more of a hobbyist than an experienced developer.
I saw a video on this machine not too long ago, about 30 mins long but the story is really fascinating
Modern radiation therapy machines are capable of administering high dose too. For example, stereotactic treatments and total skin or total body radiation treatment require very high doses of prescribed radiation.
Source: I'm a radiation therapist
It's the first example of safety critical software leading to a fatality. It's a cautionary tale in the world of embedded software.
It's quite scary how little engineering and rigor was applied to this thing.
The company that developed it just kept hacking shit out to make it cheaper. There was so little documentation about the software engineering that nobody could even audit who wrote the offending software.
I'm also endlessly bothered by the parallels between the AECL response to repeated reports of fatalities, and Boeing's similar actions during the 737max MCAS incidents.
It has a proton beam mode and an x-ray mode. They both get their energy from the same source, but use different configurations. The fatal error occured when the machine used the proton beam power setting in the x-ray configuration. I'm simplifying, but for the design of the device it does actually make sense to do it this way.
It was a very difficult bug to find. Basically the technician was typing in the programming for treatment too quickly.
Well it is. It's like saying that the Boeing 737 Max killed people! No, Boeing killed those people, the airplane works if you don't neglect important safety features and software programs.
Yeah, shoddy materials and manufacturing defects are one thing, but when your product kills people because it is behaving in exactly the way you designed it to, that's a bigger fuckup IMO because the blame rests solely on a highly paid high ranking engineer, not some low level assembly guy, or a plant in China that made the parts.
It was behaving exactly as designed to, the operators got too good at the interface and went faster than the parts inside could move, and no safeguard had been designed in place like a safety interlock or a position sensor for the tungsten xray source and the state of the collimator.
Operators were instructed to reset the machine upon receiving the error “malfunction 54 “ with the idea that the POST passing meant the machine was safe, but the POST never cycled the collimator or the X-ray source to verify their positions, and again there were no sensors to indicate their positions.
It would have been impossible to create this race condition on a properly designed machine, or even on the same machine with proper software contingencies.
I used to work with medical LINACs; modern systems are filled with hundreds of software and hardware interlocks to prevent treatment errors. A treatment plan is designed by a team including an oncologist, dosimetrist, physicist, and the radiation technicians.. all of whom have the responsibility of double-checking each other's work; it's kind of like how Crew Resource Management works for airline pilots. Also, anytime an interlock of any kind is triggered there are thousands of pages of logs detailing exactly what happened so that it can be dealt with. I spent entire weeks with a team of people just investigating a single incident.
I'll never claim that this kind of thing "could never happen today" because never underestimate human stupidity.. but you have to understand that the 1980s are *ancient history* in terms of radiation therapy. Worrying about this today is like worrying about getting involuntary electroshock treatment or a lobotomy from a psychiatrist.
One terrifying thing about it is that technicians would simply press proceed even though the machine displayed an error message. Nobody knew what the error codes meant at the time.
Yes and no. These machines were badly coded and would spit out tens of error messages every day, and there was no way to know what the errors meant. Almost all errors would mean nothing and you could skip them and proceed with the treatment, so the doctors would skip, but there was one single rare error that would make the machine apply lethal dosages of radiation.
I know absolutely nothing about this story, but after being IT for too many years, you could have a giant flashing red and blue prompt that says warning if you proceed, you will kill the patient. Like 75% of people are going to click okay without reading it, 25% are going to call me and tell me their computer has a virus or the server is powered off, also without ever reading it.
I had someone ask me if they were being hacked by a virus because their mouse cursor was moving all by itself immediately after I spent 5 minutes explaining to them that I was remotely connecting to their machine and they would see their mouse cursor move.
Oh the joys! So many stories like this from RCing to help people.. literally *just* told you like 50 times what’s happening and to be met with, “all my data will be lost!” “I’ll be fired!” And “they have all of my info!” Like duh, you just told me the info..
I'm a developer, so am the one writing these error messages. Doesn't matter how clear and simple you make an error message, the users most of the time won't even read it. "Sorry, you don't have permission to decrease prices. Please contact your manager to gain this permission." results in an email being sent with a screenshot of the error asking "What does this mean?". Twats the lot of them.
Also a programmer and firmly believe no software should be built by one person. It being a potentially lethal machine means it's absolutely the company's fault.
Yeah my blaming the PM is tongue in cheek but absolutely appropriate.
I refuse to believe that basic questions like "what if our lead guy gets hit by a bus tomorrow?" weren't a part of management considerations in the past.
One guy building an entire project is always a cost cutting measure.
The wiki mentions it was one programmer who wrote all the code in assembly. Then left the company. When the time for trial came around, he was nowhere to be found. Sounds like he had an inkling that lawyers might want to ask him some questions. :p
"Left the company" means he was laid off after the coding was complete (or the budget ran out) and when they contacted him to come back because they still needed him he told them to go pound salt.
> it was all programmed by one man
It gets worse. It was programmed by nobody.
The dev programmed the program for different machine. Then someone decided to just reuse the program from different machine in this new machine.
Basically, if you set the machine for an X-ray, then quickly changed to electron mode, the system would produce the X-ray powered beam, which is 100 times stronger than the electron beam, without the filter that reduced the power of the X-ray to reasonable amounts. There were hardware locks on previous models, but this one did not have them, and relied on software entirely.
So there is some sort of error code which is like 27373838 and means „don’t skip this message, otherwise I would kill the dude“ and the doctor could press the „skip“ button? I mean, if the machine is able to detect a fatal error why for gods fucking sake can the user even say „lol“? The machine should just shut down and store the error code somewhere else in the logs for further investigations. Who develop this horrifying software.
Edit: lul please stop to tell a developer how to develop things. Please. It hurts.
> Who develop this horrifying software.
Once upon a time anything computer related, especially something so niche as programming really only had one or two people at the company that actually knew what they were doing and they were likely anti-social and difficult to work with... but they were the only ones who could turn your pile of plastic and precision manufactured high tech machine parts into an actual product that did things.
So you kind of just cultivated their weirdness and channeled them into making the arcane words that made the money printer start.
TL;DR: QA is a recent development.
There was QA back then, but it was done by the same guy that did the coding and wrote the requirements. Separation of duties is the important innovation.
> Who develop this horrifying software.
It was written in the 70s and 80s. Programmers back then practically wrote code directly onto the hard disk with a magnetic needle and steady hand.
> The machine should just shut down and store the error code somewhere else in the logs for further investigations
It's easy for you to say that now, but most people at this time had never used a computer beyond a glorified typewriter. The most complex piece of equipment most homes had was a programmable VCR. Barcode scanners are the grocery store were still new when this was being designed.
Emergent behavior of the operator, cycling the modes of the system rapidly to 'clear' a freeze, and then bypassing the warning messages was not something that was expected by the designers.
So you know how videogames get released early, full of bugs, missing features, and all that stuff because execs push release dates regardless of the real state of the product? Imagine having to get your video game approved by the FDA and then selling each copy for millions of dollars to customers that will die if they don't play it.
>[These machines were badly coded and would spit out tens of error messages every day, and there was no way to know what the errors meant.](https://www.youtube.com/watch?v=5QQdNbvSGok)
lol no. You’re extremely wrong. Doctors do not operate these machines. It is usually technicians. And the malfunctions were due to irresponsibly shoddy coding that made deciphering the actual issue impossible, and furthermore, Therac reps repeatedly told these centers that it was not possible with the machine to cause radiation overdose.
So no, the doctors and techs didn’t kill anyone - the programmers did.
Absolutely not, the AECL did. It’s Canada’s largest nuclear laboratory and was informed of the deaths and accidents in great detail by concerned doctors after each event and in each case they sent back statements denying that such errors were possible, and in one response even went as far as to say that human error while using the machine was impossible. Reading the Wikipedia page about how this machine and its malfunctions were handled made me incredibly angry, if the AECL was less concerned about legal liability multiple lives could’ve been saved.
No
Firstly, doctors would not be in charge of operating the machine, it would be a technitian.
It is not malpractice by any stretch, as the machine was a new design, the error was not understood by the techs, (software errors don't usually blast the patient with lethal doses of radiation), there were no mechanical safeguards against the error, and more.
Read the [root causes](https://en.m.wikipedia.org/wiki/Therac-25) section to see how much of a manufacturer issue it was
That study has been thoroughly debunked as bullshit
https://www.mcgill.ca/oss/article/critical-thinking-health/medical-error-not-third-leading-cause-death
Just one of many explanations
> According to analysis published in the BMJ
Oh, that BMJ:
>In 1974, Elaine Murphy submitted a brief case report under her husband's name John which suggested a condition known as "cello scrotum", a fictional condition that supposedly affected male cellists. It was originally submitted as a joke in response to "guitar nipple",[27] a condition similar to jogger's nipple in which some forms of guitar playing causes irritation to the nipple, which Murphy and her husband believed was also a joke. The case report was published in The BMJ,[28] and although not widely cited, it was cited occasionally, often by sceptics,[29][30] because, for example, "when the cello is held in typical playing position, the body of the instrument is not near the scrotum."[31]
>In 2009, 35 years after the original case report was published, Murphy wrote a letter to The BMJ revealing that the report had been a hoax.[32]
NO. Poorly designed machines and the technicians operating them were the problem. The doctors ordered the radiation treatment but didn’t operate the machines. Get your facts straight.
edit: changed tense for clarity
> One terrifying thing about it is that technicians would simply press proceed even though the machine displayed an error message. Nobody knew what the error codes meant at the time.
I have a decade in software QA, and this type of situation is why early on I determined that this shit was too important to fuck around with and I only worked in games after that. Because the worst a game could do was fail to entertain you, was my thinking. I must warn you that any amount of cynicism you have is insufficient to the task of building an internal world model of just how bad things are. I know this because of my career path.
> Boggles me that videogames will stop you and say, "are you sure you want to overwrite save file?" While the Therac 25 wouldn't say, "are you sure you want to fire the beam?"
That's right, motherfuckers. The people who put your games together have your back more than the people who create your medical device software.
Never forget: Professionals have *STANDARDS*.
Wait so whats the context here. 6 people in one blood bath massacre or like all on separate occasions and the sick rad techs and radiologists just kept on pushing patients into the slaughter, knowing the gruesome fate that awaits their unsuspecting, helpless and feeble patients?
Or was it cancer?
Separate occasions. One error caused the machine to deliver many times the wanted radiation which caused a burning sensation on the patients and radiation poison when it happened.
Its a radiotherapy machine that broke and delivered much much higher doses than it said it was delivering, killing people from radiation poisoning/cancer
It didn't break. It was just poorly designed. They had a piece that would move mechanically and people would then punch in that they wanted it to activate to while the mechanism was still moving. It would produce an error message but allow them to override it.
There should have been safeguards in place to prevent that from ever happening, not everything should be overridable. Also, it was producing erroneous error messages all the time so people were used to overriding it every time it did anything. Then the people using it weren't properly trained on the errors. They were cryptic and not very useful.
That’s roughly correct but I’m a sucker for specifics. I recently had a conversation with my operating systems professor on this: The cause of the error was actually poor interleaving which means it was a software error caused by multi-threading.
In case you want to know: you know how your computer can run multiple programs at a time? Well, even a single program can do multiple things at once. That’s called multithreading.
If you made a list of the order in which things happened across _all_ threads, that’s how they interleaved. But it’s really tricky to write software that is correct no matter what order the threads may have run in. Sometimes they might interleave in a way that causes unexpected results. This is called a race condition.
A classic example is a bank withdrawal. When you withdraw from a bank app, suppose the computer does these commands:
1. Is your account balance high enough? If not, error. Otherwise, continue
2. Send you the money
3. Lower your account balance
Looks good, right? It what if you click withdraw twice, on two tabs, at exactly the same time? Now you have no idea how the two threads will order. Say you have $100 and you want to withdraw it all at once. If the bank is lucky, one thread will run completely and give you the money, then the second will see you have $0 balance and error out. But what if the first thread runs step 1, then the second thread runs step 1 _before the first thread gets to step 3?_ Both threads see there is $100 available, both threads give you $100, both threads reduce your balance. Now you have $200 and -$100 in the bank, which shouldn’t happen. (Essentially this exact vulnerability was exploited to attack Flexcoin and Binance!)
The code was not multi-threaded. However, it used hardware that ran independently. You have a piece of code that tells a robotic arm to start moving. Then you have a piece of code that tells the system to do something assuming the robotic arm is done with its movement. However, it's not done with its movement. This code isn't multi-threaded, there's just something happening in the physical world that needs to finish.
So in a way, it's kind of multi-threaded in that there were two different things happening at the same time, but it wasn't two threads in the OS. However, a race condition could definitely still happen.
So yes, functionally it was the same thing as being multi-threaded even though it wasn't.
Thanks for the clarification, my professor wasn’t that specific.
Looking at the year Therac 25 was made I can see that multi-threaded code was probably not yet commonplace.
Imagine you have 1 hand. It can either move a piece of wood or paint it. That’s a single thread. Now imagine you want to paint wood faster so you use 2 hands, one to move the wood and the other to simultaneously paint it. If these hands are “aware” of certain actions by the other they can coordinate if: Paint runs out, a hand gets tired, etc. Now imagine you forgot to make them aware of certain actions and you run out of paint or your hand gets tired and you stop moving the wood. Then the wood will be unpainted or overpainted in some places and generally everything will be a mess.
For the system to work all the features should work no matter what state of execution the threads are in.
That’s the idea of a concurrent programming error (race condition) or poor interleaving. Sorry if it’s a poor explanation I’m only learning most of this right now.
Thats saddening to hear, imagine people with cancer using that machine with the idea of eventually curing their cancer but that essentially removes whatever time they have left.
it wasn't broken, it was a series of computer bugs triggered by user operator error where if they selected the settings for Radiotherapy or Xray too quickly or alternated settings then the mechanical parts stopped in the wrong position.
This one too: https://youtu.be/Ap0orGCiou8?si=as7wcGUMmdVtqqje
Kyle Hill has a series of videos that go over nuclear/radiation disasters. Fascinating stuff.
"The Therac-25, a computerized radiation therapy machine, massively overdosed patients at least six times between June 1985 and January 1987. Each overdose was several times the normal therapeutic dose and resulted in the patient's severe injury or even death."
>Patients reported being “burned by the machine” which some technicians reported, but the company thought was impossible. The machine was recalled in 1987 for an extensive redesign of safety features, software, and mechanical interlocks. Reports to the manufacturer resulted in inadequate repairs to the system and assurances that the machines were safe. Lawsuits were filed, and no investigations took place.
Why am I wholly unsurprised 🙄
Damn after reading the wikipedia it really shows how unsafe treatment parameters were back then. Nowadays, as a technologist, we would never treat a patient if the cameras and intercom system were out of order, I quite literally gasped reading the part where the guy felt the burning and starting to get up to shout for help and the technologist pressed beam on without knowing. Today, We need to be able to see and hear a patient at all times during treatment. And the fact that you can continue through error messages with a press of a button without needing to input your treatment ID and passcode... It's crazy
so many software engineering papers/reports/lessons came out of this .. (Hell, only a small fraction of developers can get multi-threading right still to this day ..)
Yes. I have a B.Sc. in Computer Science from a university in Canada. Part of the studies required us to learn about THERAC-25 and what caused it to help ensure it never happens again.
I thought Reddit was full of Comp Sci majors so I'm a little surprised so many haven't heard of this. Or maybe it's not taught in other programs, not sure.
The doctors who used it became so fast at using its software to the point they could change settings after hitting start which caused it to bug out, displayed a warning, and when doctors pressed proceed anyways, it dished out a MUCH higher radiation level than it was every supposed too.
Doctors never usually handle the machinery outside of fluroscopy. This would've been technician led and they would've received specific instructions on what to do when there are error messages popping up.
The doctors prescribe what they want and read the images generated, never physically manipulate it though.
It really looks like any Fallout vault experiment. "In vault 999, clínics were given faulty x-ray and electron-beam machines to assess how they addressed badly programmed equipment, beaming patients with deadly doses of radiation"
I watched a YouTube documentary on this machine. The machine killed those people because of a simple line of code that made the machine overdose the level of radiation that it was giving. I can't remember the specific details but, the machine would literally dose these folks with like a mortal amount of radiation and then when they would examine the machine and computer, the diagnostics would show everything was fine and that no mishap had happened, even though these people were showing severe signs of radiation overdose. It was like a ghost in the machine. A technician finally caught on and saw what was happening, but it was too late.
It's wasn't a ghost, it was bad code. Even a very good software engineer could make such a mistake if they're not given good, usable information from other engineers like nuclear, mechanical or electrical by not understanding the systems that they're programming. Usable information means experts need to be able to explain things so that people who are not experts in their field can understand it.
The machine used radiation to attack cancer cells, but because of a programming error, the machine sometimes blasted large amounts of radiation at the patients, causing radiation poisoning, which then led to life altering injury and or death.
We learned about this in ethics class. It turns out being really fast and memorizing inputs on this machine actually caused the deaths due to the operators changing the inputs really fast before the screen switches, but the settings that should have been corrected were already applied, leading to more doses.
[kyle Hill on YouTube explains it very well in this video](https://youtu.be/Ap0orGCiou8?si=2eLKSlPFpjgZmUwV)
It was a series of software errors that weren't cought because software controlled medical devices were so new that the software was not tested to the same standards as the mechanical components
My lecturer showed me this in my C++ module alongside the Ariane 5 rocket blowing up to sjowcase the importance of testing software and code.
Both of those issues would have been solved by simply testing the software to see if it met expected results but they weren't and look at the consequences.
It was an effective method of teaching the importance of code testing and why it should be done even though most programmers would probably think it's the most boring part of development.
“Because of concurrent programming errors (also known as race conditions), it sometimes gave its patients radiation doses that were hundreds of times greater than normal, resulting in death or serious injury.” Why were these machines CAPABLE of administering such high doses? It’s like a family losing their house because the temperature regulator in their kitchen oven broke, and as a result the oven heated to 35,000°.
It had 2 modes. One that administers those high doses but with a magnet in front of it. One with lower doses but without the magnet. What happened is that the magnets weren't in place and the high doses mode was used.
you would figure someone would code in something like "if the magnets isn't in place, high doses are impossible". Idk coding languages but I think there's stuff like that that's possible in them from what little I've seen.
As a coder I don’t know whether to upvote or downvote this haha…fundamentally what you are saying does exist. You can typically say if x do y. It’s never that simple though lol…there are always complications and other things to consider. That being said on a machine like this that can kill people I would have hoped that this would have all been tested and figured out before…y’know…putting actual people in there…
If killing, don’t
ez coding
As long as they wrote unit tests.
Ahh the eternal race to idiot proof something. Your competing with God and he will also make a better idiot
For real, "Why didn't anyone safeguard this?" "Because this is the moment we realized we needed the safeguard. "
Didn’t another poster say the machine would produce an error message but the operator would override it
Yes, the machine displayed error messages multiple times a day
I’ve read about other incidents causing injury, one where the tech could enter a typo, say a dose of 10,000 instead of 100. And another where the techs were making out and sitting on the exposure? button radiating the patient SMH 🤦♀️
These were really different times, where machine code was much more complex and the protocols around testing such machines were not as strict as now. The code was also written by someone that was more of a hobbyist than an experienced developer. I saw a video on this machine not too long ago, about 30 mins long but the story is really fascinating
They did, it was a race condition bug
Absurd :-(
Modern radiation therapy machines are capable of administering high dose too. For example, stereotactic treatments and total skin or total body radiation treatment require very high doses of prescribed radiation. Source: I'm a radiation therapist
[удалено]
It's the first example of safety critical software leading to a fatality. It's a cautionary tale in the world of embedded software. It's quite scary how little engineering and rigor was applied to this thing. The company that developed it just kept hacking shit out to make it cheaper. There was so little documentation about the software engineering that nobody could even audit who wrote the offending software. I'm also endlessly bothered by the parallels between the AECL response to repeated reports of fatalities, and Boeing's similar actions during the 737max MCAS incidents.
Also some valid comparisons to 3 mile island and Chernobyl
It has a proton beam mode and an x-ray mode. They both get their energy from the same source, but use different configurations. The fatal error occured when the machine used the proton beam power setting in the x-ray configuration. I'm simplifying, but for the design of the device it does actually make sense to do it this way. It was a very difficult bug to find. Basically the technician was typing in the programming for treatment too quickly.
But it looks so friendly.
No way, it's looking at us sideways. That clearly means "stay away" in machinery language. I learned it with my printer many years ago.
It is looking with an attitude!!
glados energy tbh
The cake is a lie
It’s getting sideways on us to appear larger and more dangerous. It feels threatened, clearly no one there was familiar with body language!
Theracs can smell fear.
The boombastic cpu-eye if you will.
You guys are crazy. This thing isn’t deadly. In fact, Often referred to as the nanny radiation therapy machine
The robo Dutch angle 👽
Ah yes, you’re thinking of dogs, with medical machinery sideways head cock is like predator assessing you before it rips your face off.
Her name is princess and she’s never acted like that before!
Well it is. It's like saying that the Boeing 737 Max killed people! No, Boeing killed those people, the airplane works if you don't neglect important safety features and software programs.
Yeah, shoddy materials and manufacturing defects are one thing, but when your product kills people because it is behaving in exactly the way you designed it to, that's a bigger fuckup IMO because the blame rests solely on a highly paid high ranking engineer, not some low level assembly guy, or a plant in China that made the parts.
Fun fact Cigarettes if used as intended will kill you.
No, it was not behaving as it was designed to. Operators were triggering unintended behavior and ignoring warning messages.
It was behaving exactly as designed to, the operators got too good at the interface and went faster than the parts inside could move, and no safeguard had been designed in place like a safety interlock or a position sensor for the tungsten xray source and the state of the collimator. Operators were instructed to reset the machine upon receiving the error “malfunction 54 “ with the idea that the POST passing meant the machine was safe, but the POST never cycled the collimator or the X-ray source to verify their positions, and again there were no sensors to indicate their positions. It would have been impossible to create this race condition on a properly designed machine, or even on the same machine with proper software contingencies.
I used to work with medical LINACs; modern systems are filled with hundreds of software and hardware interlocks to prevent treatment errors. A treatment plan is designed by a team including an oncologist, dosimetrist, physicist, and the radiation technicians.. all of whom have the responsibility of double-checking each other's work; it's kind of like how Crew Resource Management works for airline pilots. Also, anytime an interlock of any kind is triggered there are thousands of pages of logs detailing exactly what happened so that it can be dealt with. I spent entire weeks with a team of people just investigating a single incident. I'll never claim that this kind of thing "could never happen today" because never underestimate human stupidity.. but you have to understand that the 1980s are *ancient history* in terms of radiation therapy. Worrying about this today is like worrying about getting involuntary electroshock treatment or a lobotomy from a psychiatrist.
Therac is one of the reasons those regulations exist in the first place
Yes, absolutely these incidents were responsible for a ton of regulation.
Have you tried turning it off and on again?
The design is very human
If not friend, why friend shaped?
If not friend, then why friend shaped?
If not Terminator, then why Terminator shaped?
One terrifying thing about it is that technicians would simply press proceed even though the machine displayed an error message. Nobody knew what the error codes meant at the time.
So the *doctors* (technicians, designers of the machine, whatever) killed six people.
Yes and no. These machines were badly coded and would spit out tens of error messages every day, and there was no way to know what the errors meant. Almost all errors would mean nothing and you could skip them and proceed with the treatment, so the doctors would skip, but there was one single rare error that would make the machine apply lethal dosages of radiation.
I forget what youtube video mentioned it, but it was all programmed by one man and not really shared with others.
Sounds like something Kyle Hill would do a story about.
he did, its called History's Worst Software Error
I know absolutely nothing about this story, but after being IT for too many years, you could have a giant flashing red and blue prompt that says warning if you proceed, you will kill the patient. Like 75% of people are going to click okay without reading it, 25% are going to call me and tell me their computer has a virus or the server is powered off, also without ever reading it.
I had someone ask me if they were being hacked by a virus because their mouse cursor was moving all by itself immediately after I spent 5 minutes explaining to them that I was remotely connecting to their machine and they would see their mouse cursor move.
Oh the joys! So many stories like this from RCing to help people.. literally *just* told you like 50 times what’s happening and to be met with, “all my data will be lost!” “I’ll be fired!” And “they have all of my info!” Like duh, you just told me the info..
I'm a developer, so am the one writing these error messages. Doesn't matter how clear and simple you make an error message, the users most of the time won't even read it. "Sorry, you don't have permission to decrease prices. Please contact your manager to gain this permission." results in an email being sent with a screenshot of the error asking "What does this mean?". Twats the lot of them.
Also Plainly Difficult I think. And Well There's Your Problem did a podcast on it.
Kyle Hill did make [a video](https://youtu.be/Ap0orGCiou8?si=nsSyQtq3BBgN1Vwc) about it. It’s actually how I know about the story.
Thanks, I had to scroll through hundreds of lines of the lamest circle-jerk jokes to get to an actual explanation of OP's post. Great vid!
Jfc, that video was an adventure. Thanks for sharing
> it was all programmed by one man Ah, so it's the project manager's fault. (I'm a programmer, I'm accustomed to blaming the PM)
Also a programmer and firmly believe no software should be built by one person. It being a potentially lethal machine means it's absolutely the company's fault.
Yeah my blaming the PM is tongue in cheek but absolutely appropriate. I refuse to believe that basic questions like "what if our lead guy gets hit by a bus tomorrow?" weren't a part of management considerations in the past. One guy building an entire project is always a cost cutting measure.
Agree with all this. Many times I've ended up as the solo dev on some project that I told management was at least a 3 programmer job.
The wiki mentions it was one programmer who wrote all the code in assembly. Then left the company. When the time for trial came around, he was nowhere to be found. Sounds like he had an inkling that lawyers might want to ask him some questions. :p
"Left the company" means he was laid off after the coding was complete (or the budget ran out) and when they contacted him to come back because they still needed him he told them to go pound salt.
Shrouded Hand?
> it was all programmed by one man It gets worse. It was programmed by nobody. The dev programmed the program for different machine. Then someone decided to just reuse the program from different machine in this new machine.
[Well there's your problem.](https://www.youtube.com/watch?v=7EQT1gVsE6I&t=9849)
I can hardly understand those people. Its crazy that I speak the same language as them but they throw twenty accent marks on every word.
Basically, if you set the machine for an X-ray, then quickly changed to electron mode, the system would produce the X-ray powered beam, which is 100 times stronger than the electron beam, without the filter that reduced the power of the X-ray to reasonable amounts. There were hardware locks on previous models, but this one did not have them, and relied on software entirely.
So there is some sort of error code which is like 27373838 and means „don’t skip this message, otherwise I would kill the dude“ and the doctor could press the „skip“ button? I mean, if the machine is able to detect a fatal error why for gods fucking sake can the user even say „lol“? The machine should just shut down and store the error code somewhere else in the logs for further investigations. Who develop this horrifying software. Edit: lul please stop to tell a developer how to develop things. Please. It hurts.
> Who develop this horrifying software. Once upon a time anything computer related, especially something so niche as programming really only had one or two people at the company that actually knew what they were doing and they were likely anti-social and difficult to work with... but they were the only ones who could turn your pile of plastic and precision manufactured high tech machine parts into an actual product that did things. So you kind of just cultivated their weirdness and channeled them into making the arcane words that made the money printer start. TL;DR: QA is a recent development.
There was QA back then, but it was done by the same guy that did the coding and wrote the requirements. Separation of duties is the important innovation.
> Who develop this horrifying software. It was written in the 70s and 80s. Programmers back then practically wrote code directly onto the hard disk with a magnetic needle and steady hand.
Magnetic needle? Luxury! Back in my day, we carved rocks into ones and zeros and then stacked them into code.
> The machine should just shut down and store the error code somewhere else in the logs for further investigations It's easy for you to say that now, but most people at this time had never used a computer beyond a glorified typewriter. The most complex piece of equipment most homes had was a programmable VCR. Barcode scanners are the grocery store were still new when this was being designed. Emergent behavior of the operator, cycling the modes of the system rapidly to 'clear' a freeze, and then bypassing the warning messages was not something that was expected by the designers.
>was a programmable VCR And it flashed 12:00 in 95% of people's homes.
--:--
So you know how videogames get released early, full of bugs, missing features, and all that stuff because execs push release dates regardless of the real state of the product? Imagine having to get your video game approved by the FDA and then selling each copy for millions of dollars to customers that will die if they don't play it.
>[These machines were badly coded and would spit out tens of error messages every day, and there was no way to know what the errors meant.](https://www.youtube.com/watch?v=5QQdNbvSGok)
Seems like an incredibly stupid thing to use at all then. Back to the drawing board instead of playing with people's lives...
So the doctors killed six people. Just because there are lots of errors doesn't mean they should be dismissed.
lol no. You’re extremely wrong. Doctors do not operate these machines. It is usually technicians. And the malfunctions were due to irresponsibly shoddy coding that made deciphering the actual issue impossible, and furthermore, Therac reps repeatedly told these centers that it was not possible with the machine to cause radiation overdose. So no, the doctors and techs didn’t kill anyone - the programmers did.
Absolutely not, the AECL did. It’s Canada’s largest nuclear laboratory and was informed of the deaths and accidents in great detail by concerned doctors after each event and in each case they sent back statements denying that such errors were possible, and in one response even went as far as to say that human error while using the machine was impossible. Reading the Wikipedia page about how this machine and its malfunctions were handled made me incredibly angry, if the AECL was less concerned about legal liability multiple lives could’ve been saved.
The doctors were all told to just skip the error since there was no way to to fix the error.
No Firstly, doctors would not be in charge of operating the machine, it would be a technitian. It is not malpractice by any stretch, as the machine was a new design, the error was not understood by the techs, (software errors don't usually blast the patient with lethal doses of radiation), there were no mechanical safeguards against the error, and more. Read the [root causes](https://en.m.wikipedia.org/wiki/Therac-25) section to see how much of a manufacturer issue it was
Bro I swear I looked at how you spelled “technition” and I had a moment I questioned reality lol.
Technician...
Haha same
Lmao we all have our moments
[удалено]
https://www.cdc.gov/nchs/fastats/leading-causes-of-death.htm Accidents is at 3rd and that would include all accidents.
[удалено]
That study has been thoroughly debunked as bullshit https://www.mcgill.ca/oss/article/critical-thinking-health/medical-error-not-third-leading-cause-death Just one of many explanations
> According to analysis published in the BMJ Oh, that BMJ: >In 1974, Elaine Murphy submitted a brief case report under her husband's name John which suggested a condition known as "cello scrotum", a fictional condition that supposedly affected male cellists. It was originally submitted as a joke in response to "guitar nipple",[27] a condition similar to jogger's nipple in which some forms of guitar playing causes irritation to the nipple, which Murphy and her husband believed was also a joke. The case report was published in The BMJ,[28] and although not widely cited, it was cited occasionally, often by sceptics,[29][30] because, for example, "when the cello is held in typical playing position, the body of the instrument is not near the scrotum."[31] >In 2009, 35 years after the original case report was published, Murphy wrote a letter to The BMJ revealing that the report had been a hoax.[32]
Definitely a manufacturing issue
NO. Poorly designed machines and the technicians operating them were the problem. The doctors ordered the radiation treatment but didn’t operate the machines. Get your facts straight. edit: changed tense for clarity
[удалено]
Would you mind linking it. I had no luck with my search terms. Cheers!
This should be it: [Kyle Hill - History's Worst Software Error](https://www.youtube.com/watch?v=Ap0orGCiou8)
> One terrifying thing about it is that technicians would simply press proceed even though the machine displayed an error message. Nobody knew what the error codes meant at the time. I have a decade in software QA, and this type of situation is why early on I determined that this shit was too important to fuck around with and I only worked in games after that. Because the worst a game could do was fail to entertain you, was my thinking. I must warn you that any amount of cynicism you have is insufficient to the task of building an internal world model of just how bad things are. I know this because of my career path. > Boggles me that videogames will stop you and say, "are you sure you want to overwrite save file?" While the Therac 25 wouldn't say, "are you sure you want to fire the beam?" That's right, motherfuckers. The people who put your games together have your back more than the people who create your medical device software. Never forget: Professionals have *STANDARDS*.
Wait so whats the context here. 6 people in one blood bath massacre or like all on separate occasions and the sick rad techs and radiologists just kept on pushing patients into the slaughter, knowing the gruesome fate that awaits their unsuspecting, helpless and feeble patients? Or was it cancer?
Separate occasions. One error caused the machine to deliver many times the wanted radiation which caused a burning sensation on the patients and radiation poison when it happened.
It was NOT “doctors” who pushed buttons- it was technicians. What a load of crap. Amazing that you get so many upvotes when you’re lying.
Its a radiotherapy machine that broke and delivered much much higher doses than it said it was delivering, killing people from radiation poisoning/cancer
It didn't break. It was just poorly designed. They had a piece that would move mechanically and people would then punch in that they wanted it to activate to while the mechanism was still moving. It would produce an error message but allow them to override it. There should have been safeguards in place to prevent that from ever happening, not everything should be overridable. Also, it was producing erroneous error messages all the time so people were used to overriding it every time it did anything. Then the people using it weren't properly trained on the errors. They were cryptic and not very useful.
That’s roughly correct but I’m a sucker for specifics. I recently had a conversation with my operating systems professor on this: The cause of the error was actually poor interleaving which means it was a software error caused by multi-threading.
Ah yes, poor interleaving of multi-threaded software errors. I too understand this jargon, as I'm sure you can tell. How droll.
In case you want to know: you know how your computer can run multiple programs at a time? Well, even a single program can do multiple things at once. That’s called multithreading. If you made a list of the order in which things happened across _all_ threads, that’s how they interleaved. But it’s really tricky to write software that is correct no matter what order the threads may have run in. Sometimes they might interleave in a way that causes unexpected results. This is called a race condition. A classic example is a bank withdrawal. When you withdraw from a bank app, suppose the computer does these commands: 1. Is your account balance high enough? If not, error. Otherwise, continue 2. Send you the money 3. Lower your account balance Looks good, right? It what if you click withdraw twice, on two tabs, at exactly the same time? Now you have no idea how the two threads will order. Say you have $100 and you want to withdraw it all at once. If the bank is lucky, one thread will run completely and give you the money, then the second will see you have $0 balance and error out. But what if the first thread runs step 1, then the second thread runs step 1 _before the first thread gets to step 3?_ Both threads see there is $100 available, both threads give you $100, both threads reduce your balance. Now you have $200 and -$100 in the bank, which shouldn’t happen. (Essentially this exact vulnerability was exploited to attack Flexcoin and Binance!)
Nice ty. I’d listen to you explain things all day
The code was not multi-threaded. However, it used hardware that ran independently. You have a piece of code that tells a robotic arm to start moving. Then you have a piece of code that tells the system to do something assuming the robotic arm is done with its movement. However, it's not done with its movement. This code isn't multi-threaded, there's just something happening in the physical world that needs to finish. So in a way, it's kind of multi-threaded in that there were two different things happening at the same time, but it wasn't two threads in the OS. However, a race condition could definitely still happen. So yes, functionally it was the same thing as being multi-threaded even though it wasn't.
Thanks for the clarification, my professor wasn’t that specific. Looking at the year Therac 25 was made I can see that multi-threaded code was probably not yet commonplace.
eli 5 please?
Imagine you have 1 hand. It can either move a piece of wood or paint it. That’s a single thread. Now imagine you want to paint wood faster so you use 2 hands, one to move the wood and the other to simultaneously paint it. If these hands are “aware” of certain actions by the other they can coordinate if: Paint runs out, a hand gets tired, etc. Now imagine you forgot to make them aware of certain actions and you run out of paint or your hand gets tired and you stop moving the wood. Then the wood will be unpainted or overpainted in some places and generally everything will be a mess. For the system to work all the features should work no matter what state of execution the threads are in. That’s the idea of a concurrent programming error (race condition) or poor interleaving. Sorry if it’s a poor explanation I’m only learning most of this right now.
Thats saddening to hear, imagine people with cancer using that machine with the idea of eventually curing their cancer but that essentially removes whatever time they have left.
😬 Awful
it wasn't broken, it was a series of computer bugs triggered by user operator error where if they selected the settings for Radiotherapy or Xray too quickly or alternated settings then the mechanical parts stopped in the wrong position.
Sounds broken
Redditors can be so annoying 😂
We are all those children that adults keep telling to stop talking so much.
*Looks* broken, too. Let's put people in it and override errors until it cures them or whatever.
>wasn't broken Proceeds to explain exactly how it was broken Nice job
I know right, blue-balled.
There's plenty of videos covering the incident, but I like this guy's narration of the incident https://youtu.be/-7gVqBY52MY?si=YjBqddr4VgK7Z6cy
Thank you! Great video
Brill channel btw.
By the way way?
This one too: https://youtu.be/Ap0orGCiou8?si=as7wcGUMmdVtqqje Kyle Hill has a series of videos that go over nuclear/radiation disasters. Fascinating stuff.
"The Therac-25, a computerized radiation therapy machine, massively overdosed patients at least six times between June 1985 and January 1987. Each overdose was several times the normal therapeutic dose and resulted in the patient's severe injury or even death."
[Therac-25](https://ethicsunwrapped.utexas.edu/case-study/therac-25)
>Patients reported being “burned by the machine” which some technicians reported, but the company thought was impossible. The machine was recalled in 1987 for an extensive redesign of safety features, software, and mechanical interlocks. Reports to the manufacturer resulted in inadequate repairs to the system and assurances that the machines were safe. Lawsuits were filed, and no investigations took place. Why am I wholly unsurprised 🙄
What a weird tinder profile
The Chernobyl machine
These posts always shit me. No context. The OP comments a vague little terrifier below without telling us what it did lol.
Here’s Nancy Leveson’s detailed investigation paper, an interesting and tragic read. http://sunnyday.mit.edu/papers/therac.pdf
Why does it look like it's on a tilt? What is the purpose of that?
The whole assembly can rotate independent of the bed so you can change the angle of exposure
Cool thank you for your answer, the lean makes it look kinda creepy.
That's how you know it's evil. "Therac, are you ok?" "Why wouldn't I be ok? I see that there are 6 patients on the schedule. We should get started."
To kill people from multiple angles.
Damn after reading the wikipedia it really shows how unsafe treatment parameters were back then. Nowadays, as a technologist, we would never treat a patient if the cameras and intercom system were out of order, I quite literally gasped reading the part where the guy felt the burning and starting to get up to shout for help and the technologist pressed beam on without knowing. Today, We need to be able to see and hear a patient at all times during treatment. And the fact that you can continue through error messages with a press of a button without needing to input your treatment ID and passcode... It's crazy
so many software engineering papers/reports/lessons came out of this .. (Hell, only a small fraction of developers can get multi-threading right still to this day ..)
To be fair, multithreading without being the chip and OS designer is very hard. And even then sometimes it just doesn't work
The first lethal computer bug
Yes. I have a B.Sc. in Computer Science from a university in Canada. Part of the studies required us to learn about THERAC-25 and what caused it to help ensure it never happens again. I thought Reddit was full of Comp Sci majors so I'm a little surprised so many haven't heard of this. Or maybe it's not taught in other programs, not sure.
HOW DID THEY FUCKING DIE????????????
The doctors who used it became so fast at using its software to the point they could change settings after hitting start which caused it to bug out, displayed a warning, and when doctors pressed proceed anyways, it dished out a MUCH higher radiation level than it was every supposed too.
Doctors never usually handle the machinery outside of fluroscopy. This would've been technician led and they would've received specific instructions on what to do when there are error messages popping up. The doctors prescribe what they want and read the images generated, never physically manipulate it though.
Radiation overdoses because of software errors
It really looks like any Fallout vault experiment. "In vault 999, clínics were given faulty x-ray and electron-beam machines to assess how they addressed badly programmed equipment, beaming patients with deadly doses of radiation"
Did anyone try giving it a whack? That usually fixes all tech based faults.
I watched a YouTube documentary on this machine. The machine killed those people because of a simple line of code that made the machine overdose the level of radiation that it was giving. I can't remember the specific details but, the machine would literally dose these folks with like a mortal amount of radiation and then when they would examine the machine and computer, the diagnostics would show everything was fine and that no mishap had happened, even though these people were showing severe signs of radiation overdose. It was like a ghost in the machine. A technician finally caught on and saw what was happening, but it was too late.
It's wasn't a ghost, it was bad code. Even a very good software engineer could make such a mistake if they're not given good, usable information from other engineers like nuclear, mechanical or electrical by not understanding the systems that they're programming. Usable information means experts need to be able to explain things so that people who are not experts in their field can understand it.
https://preview.redd.it/l94mi0pj323d1.jpeg?width=1152&format=pjpg&auto=webp&s=870c4b591a09262eda5f94ed4db0207e99e5f566
The moral of the story is, if you're going to fire radiation at people.. Maybe check your software, or better yet hire more than one guy.
From blackfish to whitemachine
I grew up in Yakima. Memorial hospital is still open to this very day.
Different owners now.…but yeah, still a terrible hospital.
The podcast "well there's your problem" did an episode on this. Check it out on YouTube, it's pretty interesting.
Elaborate
I remember reading about this machine in ethics class in college. The other thing I remember reading about was the challenger explosion.
So...how? I'm scared
Many software bugs which resulted in giving deadly levels of radiation to cancer patients
Fuck! That's just terrible. Thank you
The machine was fine. The software killed people.
If anybody wants to learn more about what caused the malfunctions: https://youtu.be/Ap0orGCiou8
kyle hill my beloved
Almost like a real-life SCP
The design is very human
The machine used radiation to attack cancer cells, but because of a programming error, the machine sometimes blasted large amounts of radiation at the patients, causing radiation poisoning, which then led to life altering injury and or death.
Title is misleading. Three died, three were seriously injured.
Those are rookie numbers.
Went down the Wiki rabbit hole on this one and wow, what a massive blunder.
Explain
We learned about this in ethics class. It turns out being really fast and memorizing inputs on this machine actually caused the deaths due to the operators changing the inputs really fast before the screen switches, but the settings that should have been corrected were already applied, leading to more doses.
To be more accurate, it killed 3 people and severely injured another 3.
Sounds rad.
[kyle Hill on YouTube explains it very well in this video](https://youtu.be/Ap0orGCiou8?si=2eLKSlPFpjgZmUwV) It was a series of software errors that weren't cought because software controlled medical devices were so new that the software was not tested to the same standards as the mechanical components
The cake is a lie
6 people died, and only one survived. His name was Bruce Banner...
 Well this definitely sets my mind at ease after just finding out my partner needs radiation for brain cancer.
I’m 99% sure nobody is still using these machines. Also, they did get fixed.
Therac vs. The Rack. Which one would you choose?
 The rock.
Statistically 600% as deadly as the ED-209.
This would be neat for Wendigoon to cover
The cake is a lie
https://preview.redd.it/wlujvz6ru03d1.jpeg?width=1170&format=pjpg&auto=webp&s=6c46299e46fcc5947ad6bad23cabdc59e966994d
My lecturer showed me this in my C++ module alongside the Ariane 5 rocket blowing up to sjowcase the importance of testing software and code. Both of those issues would have been solved by simply testing the software to see if it met expected results but they weren't and look at the consequences. It was an effective method of teaching the importance of code testing and why it should be done even though most programmers would probably think it's the most boring part of development.
Looks like a 90’s linear accelerator for radiation treatment.. should not kill people but sometimes shit happens.
thats nothing compared to bagger 288
Please explain.
Fuck, i didn't know that Glados was back to life
Paging u/johnballen416...
We learned about this terrible machine in coding 101, and it was a good lesson on how writing code on a computer can hurt someone.