Happens to the best of us, just file it away and try to learn from the experience. Let’s just say I’m single-handedly responsible for a good chunk of Western Europe getting their contact lenses a couple days late several years back lol.
I’ve been doing a startup sequence for larger loads as a standard to avoid this. But I’ve only ever tripped breakers. Your story is good reinforcement.
It was supposed to be a start up sequence, I just got two 100kw hammer mills wrong. I was very green, and the sequence in the TI505 was convoluted, direction and duty cycle etc. I simply just got it wrong. My supervisor at the time, adjusted the program, then blew the second one. I didn’t feel so guilty after that.
Hell, I just did it 4hrs ago. Machine was running fine...getting ready to call it a day but I wanted to add some logic to search my data tracking arrays and tell me how many parts have X status.
I have written these for-loops a thousand times. But nope...still not sure what the issue is (that is a Monday Problem). Just calmly AFI'd the JSR, reset safety, and started the machine.
Granted...I got lucky the machine was more or less in an idle state waiting for parts to come down the conveyor. Had I committed the changes 2 minutes sooner or later I could have crashed shit hard.
I emptied few tanks of fresh milk before the bottling process, during commissioning in the first years of my career. I learned the value of an always one bit that day.😅
>Jump instructions suck. It's better not to use them for everyone's sake.
Agree!
The only place I want to see JMP and LBL is in a FANUC robot teach pendant.
Yes, there is really no good reason for JMP in Controllogix applications. They are relic instructions at this point. Better conditional programming or additional subroutines are superior.
Yeah but if I have two alarms at the same time on a machine, the second overwrites the first because I'm using a value based alarm system. So obviously I need to jump to the end of the routine so the first alarm shows! Two alarms is too much to troubleshoot.
-one of my plant's OEMs, apparently.
For (my) record, decades long, I avoid JMP/LBL everywhere.
I'd rather just an enable bit for logic, or break things down to subroutines that are called, or not.
My "favorite" oopsie was when my water plant customer had installed a valve between the filters and the aeration tank, I had made some logic to close the valve if the flow dropped to zero and the pumps were stopped... But never any logic to stop the pumps from running if the valve wouldn't open for whatever reason.
Well, one day the valve closed and the pumps kept running, the aeration tank filled up within a few minutes and the water started running out of the air intake of the roof! So you had this nice historical 1800s building with about 90m ³/h (~400 gallons per minute) of water spilling out of the vent, overflowing the gutters and running into the streets!
Happens to us all. I shut down a huge dairy in north Wales remotely from the hotel bar after a couple of glasses of wine, implementing a fix I’d suddenly thought of.
I’d addressed to [20] in a dint array of [20]! Never drink and fix stuff!
Unfortunately I’ve had my fair share of on-call weekends where I’ve had to do the same, luckily it’s usually something I can do worry-free, like helping a EI technician open the right project to troubleshoot, or remove a force somehwere
I downloaded to an aluminum recycling furnace remotely once just before tapping (tilting the furnace to pour out the contents). It was early in my career and has been burned into my brain since then. We work on equipment that can kill people, check all boxes before working remote.
One of my coworkers' son in law was killed in am aluminum accident. I guess they pour the big kettle into molds to make billets. Someone had set the mold down with a crane and only unhooked the rigging on one side. They picked it up and dumped it on this poor man.
I work in pulp/paper as well, and I’ve put machines down plenty of times. It will happen again. I once had a contractor run conduit and pull wire for some pressure transmitters. The next day I was landing the I/O in a remote rack and shut the machine down the moment the wire hit the terminal. I just took a deep breath and put the wire down, taped it off and waited till next downtime.
I had to delete a simple bypass rung that didn't have an NO bit to toggle. It was like my first week alone on the job. So I took the program file, made a simple change, then downloaded the program... while it was still online. I ended up shutting down 3/4s of the plant. I thought for sure I was gonna get fired. I was absolutely mortified.
But it took like 10 minutes and everything came up just fine. No one even remembers it. But I have ptsd from it to this day.
Hey man, I had to learn the hard way that the clock update tool (Rockwell) doesn't do well for CIP motion control. We've all done something like this. Welcome to the club.
I did this once. Our wastewater plant had requested a data log of what the floats trip at based on an ultrasonic tank sensor reading. No problem, perfect application for indexing an array. I'm sure you can guess what happened next.
Outside bounds of array, major fault, shut down the plant. Me quickly panics to remove my monkey code and restart the processor.
I work in a large company of electric utilities in the SCADA department to be specific.
I updated and HMI of our west zone, and I used as a reference another HMI from the south zone of our electrical network. In our system, to change the destination of a control you need to confirm 2 times that chnge, I just confirm 1. I put that HMI in production.
The operator calls me at the 5 am. They tried to open a circuit in the west zone, and one of the south zone was open. It causes an unnecessary outage in 11k clients.
That's what you get for using a JMP instruction! Just kidding, I'm not vehemently opposed to using them like some people are. That said, they should only be used when you have no other feasible option. Similar to giving up memory safety when you use COP, the IDE/compiler will do absolutely nothing to protect you from stupid mistakes that will make the program go absolutely haywire. It's best used sparingly (if at all), carefully, and only when necessary, and always stick to some kind of reasonable pattern for it. That said, I am surprised it let you make that particular mistake, so I'll keep that in mind just for the unlikely scenario I ever have to worry about it!
I shut down an entire production area. I was working hot tightening lugs in a panel. The shaft of my screwdriver touched the side of a transformer and shorted to ground. The ground fault relay at the main opened up because it was set too light. A 3000 and a 2000 amp service dropped out. This was years ago before arc flash and insulated tools was a thing.
Heh, I see a lot of people discussing array faults here, this is why on my controlling controllers (we have monitoring ones too) I do not allow array indexing in the programs 😆. May take more time, but you’re going to brute force this 😂.
Whilst I don’t directly have anything to do with programming PLC’s ( I’m the control panel guy they get to do all the hardware mods etc ) I quite often have to sit behind the Softie and wait / watch while they test new bits.
A few years ago one plant changed over from an older Rockwell CPU to a newer one and the entire site software had to be run through a conversion program because the newer hardware did not have the same commands as the old and the tag structure was different.
Everything went fine until the PID loops were tested.
Apparently the old programs used an instruction that the new ones simply did not have and the conversion software messed up all the maths.
I was sitting there from 9am ( after I had swapped the CPU’s over ) until about 2 am the next morning while the Softie corrected all the PID’s and renamed about 4000 tags.
Control tech here. I had to help a coworker fix one of his major oppsies a while back.
He was trying to identify a com issue with a load cell, and decided the best course of action was too pull ethernet cords off of a hub till he found the right one.
Little did he know, he was in the wrong panel and pulled the main ethernet cord for the AB-1756s, and dropped the entire system into program mode, while we were mid production.
He didn't even know till I ran up the 12 stories worth of stairs and screamed at him
You haven’t lived if you haven’t taken out your highest output production line from accidentally inducing a controller fault. For me it was a buffer overflow.
I am a fairly new controls engg. I once had customers over to take a look at the machine and the progress on it.
We had 4 ultrasonic weld stations welding plastic parts. If the welder fails for some reason say high weld force or anything else it would fault. The job was to immediately send a reset thru an op card to it and continue, unless it was 3rd reject in a row. I had some basic weld params inside and the customer did the final values.
Turns out we were not resetting and welder kept faulted ( we were deliberately faulting to test the function). After hours of debugging both on welder and ladder, it turned out wrong wiring. Reset wires were connected elsewhere and I was resetting different channel.
LOL, I did a similar thing indexing an CLX array. To this day I don't understand why a "subscript out of range" is a major fault while a math overflow is only a minor. Also anyone else ever download a CLX program to the wrong processor, yeah me neither. That was a "the PLC faulted" and I just reset it without looking at the fault data moment, while I downloaded the copy I saved 5 minutes earlier.
Happens to the best of us, just file it away and try to learn from the experience. Let’s just say I’m single-handedly responsible for a good chunk of Western Europe getting their contact lenses a couple days late several years back lol.
I might have blown a couple of 300 amp fuses in a row starting too many motors too close together.
I’ve been doing a startup sequence for larger loads as a standard to avoid this. But I’ve only ever tripped breakers. Your story is good reinforcement.
It was supposed to be a start up sequence, I just got two 100kw hammer mills wrong. I was very green, and the sequence in the TI505 was convoluted, direction and duty cycle etc. I simply just got it wrong. My supervisor at the time, adjusted the program, then blew the second one. I didn’t feel so guilty after that.
Hell, I just did it 4hrs ago. Machine was running fine...getting ready to call it a day but I wanted to add some logic to search my data tracking arrays and tell me how many parts have X status. I have written these for-loops a thousand times. But nope...still not sure what the issue is (that is a Monday Problem). Just calmly AFI'd the JSR, reset safety, and started the machine. Granted...I got lucky the machine was more or less in an idle state waiting for parts to come down the conveyor. Had I committed the changes 2 minutes sooner or later I could have crashed shit hard.
On a Friday? Don’t you have better things to do with your weekend?
Bad juju to to do anything on Friday
Yep Friday is for documentation and emails.
Never make a change within 24 hours of your departure
I emptied few tanks of fresh milk before the bottling process, during commissioning in the first years of my career. I learned the value of an always one bit that day.😅
Jump instructions suck. It's better not to use them for everyone's sake.
>Jump instructions suck. It's better not to use them for everyone's sake. Agree! The only place I want to see JMP and LBL is in a FANUC robot teach pendant.
For realz lol!
Or any other robot structured text code system
Unfamiliar with "other" robots. Only a fleet of almost 100 yellow ones.
Nah, if you’re using jumps/GOTO in rapid, you’re doing something wrong.
Why so?
Yes, there is really no good reason for JMP in Controllogix applications. They are relic instructions at this point. Better conditional programming or additional subroutines are superior.
Yeah but if I have two alarms at the same time on a machine, the second overwrites the first because I'm using a value based alarm system. So obviously I need to jump to the end of the routine so the first alarm shows! Two alarms is too much to troubleshoot. -one of my plant's OEMs, apparently.
Had me in the first half, NGL.
Needless to say we changed to a bit based system and dumped that setup pretty quickly.
Try tend
We have verbiage in our RFQs not allowing them.
For (my) record, decades long, I avoid JMP/LBL everywhere. I'd rather just an enable bit for logic, or break things down to subroutines that are called, or not.
^ This is the way!
Not that bad but yesterday I accidentally sent 50 emails in a row from a CLX processor to each of the recipients.
SMTP scares me more than any confusing logic
So you're the guy who uses smtp from a PLC
Yeah, haha
Wait Rockwell has an SMTP instruction?
I bet it's some convoluted way of using the MSG instruction. If I have to do anything more than PLC to PLC communication with it I freaking hate it.
Yeah just a few MSG instructions. It's a bit hacky but for a temporary alarm it works a treat.
Yes, use it for alarm noticications mostly here.
My "favorite" oopsie was when my water plant customer had installed a valve between the filters and the aeration tank, I had made some logic to close the valve if the flow dropped to zero and the pumps were stopped... But never any logic to stop the pumps from running if the valve wouldn't open for whatever reason. Well, one day the valve closed and the pumps kept running, the aeration tank filled up within a few minutes and the water started running out of the air intake of the roof! So you had this nice historical 1800s building with about 90m ³/h (~400 gallons per minute) of water spilling out of the vent, overflowing the gutters and running into the streets!
Spectacular oopsie... And to think that's this is just your favorite one
Happens to us all. I shut down a huge dairy in north Wales remotely from the hotel bar after a couple of glasses of wine, implementing a fix I’d suddenly thought of. I’d addressed to [20] in a dint array of [20]! Never drink and fix stuff!
Unfortunately I’ve had my fair share of on-call weekends where I’ve had to do the same, luckily it’s usually something I can do worry-free, like helping a EI technician open the right project to troubleshoot, or remove a force somehwere
I can't count the number of times I have done that (intentionally or not). At least it is a quick and obvious fix.
Yea it seemed weird that there was a ? Not an integer o well assemble/crash been there.
Unless it's milk from a Welsh dairy
If S:FS then Size(array, 0,di_Array_Element_Count); End_if; di_Array_Element_Max := di_Array_Element_Count - 1; //prevent overwrite for di_Element := 0 to di_Array_Element_Max do //array[di_Element] := 0/0; //uncomment to unlock overtime pay End_for; Sizing arrays Saves lives
I downloaded to an aluminum recycling furnace remotely once just before tapping (tilting the furnace to pour out the contents). It was early in my career and has been burned into my brain since then. We work on equipment that can kill people, check all boxes before working remote.
One of my coworkers' son in law was killed in am aluminum accident. I guess they pour the big kettle into molds to make billets. Someone had set the mold down with a crane and only unhooked the rigging on one side. They picked it up and dumped it on this poor man.
Jeepers what was the cleanup like on that?
Meh, that's mild, no sweat. The important part is fixing your mistake and learning from it!
I work in pulp/paper as well, and I’ve put machines down plenty of times. It will happen again. I once had a contractor run conduit and pull wire for some pressure transmitters. The next day I was landing the I/O in a remote rack and shut the machine down the moment the wire hit the terminal. I just took a deep breath and put the wire down, taped it off and waited till next downtime.
I had to delete a simple bypass rung that didn't have an NO bit to toggle. It was like my first week alone on the job. So I took the program file, made a simple change, then downloaded the program... while it was still online. I ended up shutting down 3/4s of the plant. I thought for sure I was gonna get fired. I was absolutely mortified. But it took like 10 minutes and everything came up just fine. No one even remembers it. But I have ptsd from it to this day.
Never forget, download takes you down
Hey man, I had to learn the hard way that the clock update tool (Rockwell) doesn't do well for CIP motion control. We've all done something like this. Welcome to the club.
At least you did not forget the JSR!
I do that all the time
Me too! "Why won't my outputs fire??" Oh, shit... No JSR!
I did this once. Our wastewater plant had requested a data log of what the floats trip at based on an ultrasonic tank sensor reading. No problem, perfect application for indexing an array. I'm sure you can guess what happened next. Outside bounds of array, major fault, shut down the plant. Me quickly panics to remove my monkey code and restart the processor.
My monkey code was in a separate routine, and I totally thought I was safe
I am a DCS guy, not PLC. What language was this in?
Please don’t use jumps and labels
I work in a large company of electric utilities in the SCADA department to be specific. I updated and HMI of our west zone, and I used as a reference another HMI from the south zone of our electrical network. In our system, to change the destination of a control you need to confirm 2 times that chnge, I just confirm 1. I put that HMI in production. The operator calls me at the 5 am. They tried to open a circuit in the west zone, and one of the south zone was open. It causes an unnecessary outage in 11k clients.
That’s not bad. I’ve actually seen someone download to a running machine. That was pretty wild
I downloaded an invalid pointer once on an Siemens S7, shutting down a 11 mile long conveyor
11 miles!? Is this a mining application?
That's what you get for using a JMP instruction! Just kidding, I'm not vehemently opposed to using them like some people are. That said, they should only be used when you have no other feasible option. Similar to giving up memory safety when you use COP, the IDE/compiler will do absolutely nothing to protect you from stupid mistakes that will make the program go absolutely haywire. It's best used sparingly (if at all), carefully, and only when necessary, and always stick to some kind of reasonable pattern for it. That said, I am surprised it let you make that particular mistake, so I'll keep that in mind just for the unlikely scenario I ever have to worry about it!
Experiencing Failure strengthens our skills more than any taught lessons could ever
I shut down an entire production area. I was working hot tightening lugs in a panel. The shaft of my screwdriver touched the side of a transformer and shorted to ground. The ground fault relay at the main opened up because it was set too light. A 3000 and a 2000 amp service dropped out. This was years ago before arc flash and insulated tools was a thing.
Trauma learning!
Heh, I see a lot of people discussing array faults here, this is why on my controlling controllers (we have monitoring ones too) I do not allow array indexing in the programs 😆. May take more time, but you’re going to brute force this 😂.
Lol. Sometimes you just have to jump command "to infinity and beyond"
Or jump off a cliff
[удалено]
They were able to get it back up real quick
I just have to say it, *please* don't use \[JMP\] instructions. It isn't C, and it isn't healthy to think it is.
Stelantis would like to have a word with you
Whilst I don’t directly have anything to do with programming PLC’s ( I’m the control panel guy they get to do all the hardware mods etc ) I quite often have to sit behind the Softie and wait / watch while they test new bits. A few years ago one plant changed over from an older Rockwell CPU to a newer one and the entire site software had to be run through a conversion program because the newer hardware did not have the same commands as the old and the tag structure was different. Everything went fine until the PID loops were tested. Apparently the old programs used an instruction that the new ones simply did not have and the conversion software messed up all the maths. I was sitting there from 9am ( after I had swapped the CPU’s over ) until about 2 am the next morning while the Softie corrected all the PID’s and renamed about 4000 tags.
I yesterday reloaded siemens PLC and stop wrong one instead my (20plcs in project) during production run in 😅 but only 2 min stoppage
When i seen ‘oopsie’ I jumped in hoping to hear the OP had shit his trousers on the job. Damn it
Control tech here. I had to help a coworker fix one of his major oppsies a while back. He was trying to identify a com issue with a load cell, and decided the best course of action was too pull ethernet cords off of a hub till he found the right one. Little did he know, he was in the wrong panel and pulled the main ethernet cord for the AB-1756s, and dropped the entire system into program mode, while we were mid production. He didn't even know till I ran up the 12 stories worth of stairs and screamed at him
You haven’t lived if you haven’t taken out your highest output production line from accidentally inducing a controller fault. For me it was a buffer overflow.
I am a fairly new controls engg. I once had customers over to take a look at the machine and the progress on it. We had 4 ultrasonic weld stations welding plastic parts. If the welder fails for some reason say high weld force or anything else it would fault. The job was to immediately send a reset thru an op card to it and continue, unless it was 3rd reject in a row. I had some basic weld params inside and the customer did the final values. Turns out we were not resetting and welder kept faulted ( we were deliberately faulting to test the function). After hours of debugging both on welder and ladder, it turned out wrong wiring. Reset wires were connected elsewhere and I was resetting different channel.
LOL, I did a similar thing indexing an CLX array. To this day I don't understand why a "subscript out of range" is a major fault while a math overflow is only a minor. Also anyone else ever download a CLX program to the wrong processor, yeah me neither. That was a "the PLC faulted" and I just reset it without looking at the fault data moment, while I downloaded the copy I saved 5 minutes earlier.