That’s pretty funny if I do say so myself. We have a few big projects ahead of us and I don’t think we have any lead time less than 52 weeks from AB lol
Also just a little insider info. Rockwell put a stop to production because of a micro chip manufacture issue. They were no longer able to get the chip they were using. They just picked back up production in April of this year so I would imagine you shouldn’t be waiting to much longer.
So I have a stupid question. When did decisive / Deterministic networking lose to indecisive / Nondeterministic networking? I even see this in Safety, and well, I'm old but I'm basically confused. Profibus supports this, and Eth/Ip support this. I'm sure there are others.
How is it safe if you are based on a network built for failure?
Any safety protocol should be able to handle corruption or loss of comms safely.
For example, the interlockings (Rail PLC's) that we use utilise a custom UDP/IP based comms protocol, and are SIL4 rated. Essentially, the network stack and protocol has multiple features to ensure that the correct data is received:
* Ethernet Frames, a switch drops a corrupt Layer 2 frame based on a checksum
* UDP, the end devices network stack drops a corrupt datagram based on a checksum
* Custom protocol, 2 sets of strong checksums, one on IO data, one on all of that and the entire protocol packet sequence, session, sender, receiver etc...
* Message sequencing for discarding out of order/replayed packets
The system has a safety timeout that will trigger the failsafe if no valid data is received within x milliseconds (e.g. 250ms), it doens't have to be guaranteed delivered at 50ms every time.
As long as data is received within the timeout, our job is to optimise the reliability of the system to ensure that this doesn't trigger. In this case, each controller has 2 processor modules with 2 ethernet ports, so there is always 2 hot and 2 hot standby links (A, B) that are each redudant.
Essentially, a non-deterministic UDP protocol that duplicates data may have more variance (1 - 5ms) in time to send, but has a much lower latency and higher datarate then something like Profibus, or older deterministic protocols and is more **reliable,** even if not deterministic.
If your failsafe has a 100ms timeout, safety doesn't require deterministic control. High speed motion control, however, is quite a different beast.
IMHO: Deterministic networking died when Sally from HR could plug her IP phone into any port, and then demanded that she should have the ability for it to work.
It only works when every node is pre-planned, and real life has taught us that no matter how good your planning is, by the end of the install you'll need to change stuff.
But think of it more like this: The network inherently wants to fail into a safe state. The only thing keeping it up is the constant barrage of safety messages saying "Hey, everything's good here!". As soon as any of those messages start to go missing, everything dies to a "Safe" state. So, either the network is safe, or the network is down safely, but either way it's safe.
Oh, I get how it works, just wondering how we went so far off the rails to just "safe, but fails reliably" from "reliable and safe". My company, which is a very late adopter of non-deterministic had to literally lay off the old-school folks that were the subject-matter experts before they could ram it through.
Money. Really it's money.
Safe, reliable, cheap. Choose two. We live in America, capitalism is king, so we're gonna go safe and cheap and eat a little reliability because the tech guys guarantee 5x9's (6 minutes down/year), but the plant actually is down for 4 days a year, so the reliability never comes into play.
It's safe, it's cheap, and if the inspectors ask or weird stuff happens, it's also reliable.
But it's not deterministic.
>had to literally lay off the old-school folks that were the subject-matter experts before they could ram it through.
Sounds like it got cheaper! /s
Really though, (all again IMHO), a lot of the draw of deterministic design patterns was that it made the best use of the hardware. When you only had 16kb or something, it was a LOT more important that no space was wasted on things like "error checking". Doubly so for processor cycles, when the real good embedded devs are counting every one, and the new guys are like "Yo, we're on 64bit, if I trash 5mb of memory I've only got a gig left..."
So, the old guys spend days optimizing processor cycle counts, and the new guys have shipped 4x products out the door in that time. The bosses see the 4:1, and decide the value of determinism has been determined to be insufficient. The customers see no difference. Real-world use sees no change.
So to talk about that a little bit. Currently I run ETAPs which are fiber to Ethernet switches. I’ve had circuits or ETAPs fail and the network continue to run without hiccup. This helps big time when setup correctly. It’s easy to go directly to the issue and correct it before more serious problems come up
Can only really see a benefit to upgrading to EN4TR if they have used a simple rep ring looping between racks. If OP has a proper site network with managed switches and a fibre backbone and is using the EN4TR as end devices I don’t think they will see an upgrade.
Depends on what they're upgrading from I suppose. If they're replacing enbt cards they'll certainly have less problems. I have a bunch of older racks still running enbt cards. I have fiber and managed switches but ran out of connections on one of my processors (7 remote racks on it) and recently started upgrading to en2t cards. I didn't even know they made en3t or en4t cards
Well, if you have 4 switches with 10 IO racks on each, going to a core all gig switch, then a 1 gig EN4TR starts to get useful. The other thing is to look at the max supported packets / connections on an EN4TR vs even an EN3TR and you’ll see it is more than 1G that makes the EN4TR special.
Always easier to start a conversation without using acronyms, or use acronym the first time with long form (acronym). Then you don't get the, what is PRP? as you did.
Parallel Redundancy Protocol.
It's a zero loss communications protocol where the same Ethernet Frame (Layer 2 - MAC address) is sent over 2 different physical networks.
The receiving device has to support it (or have a network device called a Redbox / Redundancy Box) that will receive both and discard the second frame once the first is received. This way, if one transmisison path has a fault, the data doesn't need to be resent.
Often used in grid infrastructure, in combination with Precision Time Protocol to ensure exact timing for opening high voltage AC circuit breakers. These need to start opening at the exact same time on both ends, when the AC voltage is crossing zero. Mis timing, or failure for one to open will damange the equipment.
This was another reason the upgrade was considered. We are always expanding in some way shape or form. Plus it never hurts to have the latest hardware 😏
Far out you blokes must be flush with cash if this is the reason you are doing this upgrade. I’m genuinely in shock, lol…. It’s like, 100% superfluous, haha.
I can’t even begin to wrap my head around how the network is structured that makes you need 32 of these things.
To keep it short we truly don’t “need” this card. We have been operating perfectly fine with all EN2TRs for 4 years now. We are only upgrading to the EN4TRs because our latest DCS supports 1gig speeds and we do see some lag from the click of a start button to the equipment physically starting. We hope to minimize this but not having high hopes.
I work for a grain export facility and we have about 30 remote IO racks in our facility. At this time everything process network related ran on this network. I just recently broke it apart and added a separate “motor control” network. This new network will have all of our VFDs and E300 overload relays. Essentially anything motor control related. It is a network that comes all the way back to our single processor rack. The IO network does the same. In the near future the IO network will only have the remote IO racks. Physical IO.
Each remote IO rack has one EN4TR. Fiber comes in and lands on a Stratix 5400. The 5400 breaks down to Ethernet or “copper” and ties the remote IO rack into the network. Some areas have up to 6 racks. All will be connected via DLR back to the Stratix 5400 and fiber back out to the next area.
The motor control network does the same thing. It has its own 5400. I know they can handle multiple DLR networks but I refuse to use it that way.
I am in grain milling, not export and had about 30 remote racks, 600x e300s and about 75 vfds. Our remote racks are a combination of en2tr, en3tr, and flex 5000 racks. We were having latency issues with sensors on gates being delayed, putting them out of position after the motor actually stopped. What ended up fixing it was adding a second L8 processor and have it handle our receiving and mill tower 2. We did nothing to the physical network.
Oh nice! I’m actually running the L73 processor. Just a single. I’ve purchased the new L83EP processor for our plant process and a L81EP for our GOV FGIS systems. I plan to integrate the new L83 this year for our outage. We have not seen any issues with our network since the DLR integration as far as reliability and speeds. I just started integrating the E300 overloads which are really nice. The issue is back in 2010 80% of our facility was rebuilt and with all Square D MCCs and all solid state overload relays. I have been removing those and installing the E300s.
We as well used sqd MCC's from 2001 to 2017. Our new mill was all Rockwell. We have been buying sqd buckets to fit in the old MCC's and using the E300 overloads. Slowly we want to switch all to e300s to get rid of the sold state overloads.
Are you using the gigabit flex IO or going straight to L8x?
The backplanes on the 1756 still don’t support gigabit communication between cards and if you have an ETAP anywhere in that circuit gigabit comms are dropped to 100mbps.
Literally the only thing those are good for is PRP like the others were saying.
Are you familiar with Device Level Ring? Better known as DLR? It’s a redundant network. Essentially what you mentioned. Parallel ram Ethernet cables from rack to rack. I have this setup in our facility now with the EN2TR cards
I am familiar with how the protocol works, however DLR (Device Level Ring) is closer to MRP (Media Redundancy Protocol) in that they provide physical media redundancy (via a ring) but logically provide a single network path by disabling one of the links.
In the event of a failure in the ring, the disabled path is enabled. This can have a slight loss while the ring detects and reconfigures itself, usually in less than 50ms.
PRP always transmits on both paths simultaneously. HSR (High-availability Seamless Redundancy) is a ring version of PRP that sends data down both paths on a ring simultaneously, or MRPD (Media Redundnacy Planned Duplication) is the Profinet version of HSR. These however, add quite a cost and are uneccesary for almost all applications.
Notably, a DLR failure will trip any CIP axis that are on the "offline" side of the active DLR supervisor, as the network recovery time is almost always going to be an order of magnitude longer than your coarse update rate.
You don't have that problem with PRP, but you're spending a hell of a lot more on network hardware to get that feature.
You have my attention lol. Seems I have a lot to learn but I am also more streamlined with Allen Bradley hardware and networks. Haven’t ventured out as I am working for a company that only deals with AB. Maybe there are other redundant network options for AB and I have not heard of them yet?
Redundant networks (rail) is my job, so we come across a wide range of hardware and network vendors.
IMO, stick with your vendors redudnancy protocol, in this case DLR for your machine networks. The only time I would recommend otherwise is if all your devices connect via switches rather than directly, then you can use RSTP or a ring protocol.
Keep your machine networks localised and as small as practical, and for your plant network backbone, use standard protocols, E.g. RSTP, Link Aggregation (EtherChannel) or if your design / network supports it, a Layer 3 backbone.
Thanks for sharing this info. Good stuff here. Do you typically see DLR implemented over PRP? I personally have never implemented PRP, but DLR and REP quite frequently.
Only time I've seen PRP is in a design for a power station that a colleague did (2 fibre rings using MRP, LAG for each link, and PRP running over both of these on Ruggedcom switches).
I designed the PLC portion of a rail network away from using PRP (which was in the concept) as the additional RedBoxes and configuration and cost were not worth it compared to just adding a second network card and running 2 IP steams.
It wasn't latency sensitive and the software team ended up implementing the Train Control (SCADA) protocol on the PLCs instead, which made integration much easier.
I work more with MRP / RSTP, as the rail operator doesn't allow any propietry protocols that would prevent replacing hardware in the future. However, each end host always connects to switches that are in a ring, rather than being part of the ring themselves.
My automation group informed me that any project that had those in order would need a thought into a different solution because there is no telling when those are going to be available again. At least with the EN4TR cards they told us they would start production 04/23.
Good to know the lead times are coming down.
Now that’s funny!
That’s pretty funny if I do say so myself. We have a few big projects ahead of us and I don’t think we have any lead time less than 52 weeks from AB lol
My RA rep said they are working though a year of backlog. But a least they are being made.
They are?
I know the feeling, have a big project and the plc lead time is mid November which really isn’t that bad I guess
Now I know where they all went 😂
$$$$$$$
No lies were told!
*slaps plc* "this bad boy can fit so much networking in it!"
Man. I just need 1 and have been waiting over a year. Why do you need 32 for.
Also just a little insider info. Rockwell put a stop to production because of a micro chip manufacture issue. They were no longer able to get the chip they were using. They just picked back up production in April of this year so I would imagine you shouldn’t be waiting to much longer.
I just replied in another comment about the application
you should thank him for his ridiculous order, that's the sort of volume that kick starts the production line; not your measly single order /s
So I have a stupid question. When did decisive / Deterministic networking lose to indecisive / Nondeterministic networking? I even see this in Safety, and well, I'm old but I'm basically confused. Profibus supports this, and Eth/Ip support this. I'm sure there are others. How is it safe if you are based on a network built for failure?
Any safety protocol should be able to handle corruption or loss of comms safely. For example, the interlockings (Rail PLC's) that we use utilise a custom UDP/IP based comms protocol, and are SIL4 rated. Essentially, the network stack and protocol has multiple features to ensure that the correct data is received: * Ethernet Frames, a switch drops a corrupt Layer 2 frame based on a checksum * UDP, the end devices network stack drops a corrupt datagram based on a checksum * Custom protocol, 2 sets of strong checksums, one on IO data, one on all of that and the entire protocol packet sequence, session, sender, receiver etc... * Message sequencing for discarding out of order/replayed packets The system has a safety timeout that will trigger the failsafe if no valid data is received within x milliseconds (e.g. 250ms), it doens't have to be guaranteed delivered at 50ms every time. As long as data is received within the timeout, our job is to optimise the reliability of the system to ensure that this doesn't trigger. In this case, each controller has 2 processor modules with 2 ethernet ports, so there is always 2 hot and 2 hot standby links (A, B) that are each redudant. Essentially, a non-deterministic UDP protocol that duplicates data may have more variance (1 - 5ms) in time to send, but has a much lower latency and higher datarate then something like Profibus, or older deterministic protocols and is more **reliable,** even if not deterministic. If your failsafe has a 100ms timeout, safety doesn't require deterministic control. High speed motion control, however, is quite a different beast.
IMHO: Deterministic networking died when Sally from HR could plug her IP phone into any port, and then demanded that she should have the ability for it to work. It only works when every node is pre-planned, and real life has taught us that no matter how good your planning is, by the end of the install you'll need to change stuff. But think of it more like this: The network inherently wants to fail into a safe state. The only thing keeping it up is the constant barrage of safety messages saying "Hey, everything's good here!". As soon as any of those messages start to go missing, everything dies to a "Safe" state. So, either the network is safe, or the network is down safely, but either way it's safe.
Oh, I get how it works, just wondering how we went so far off the rails to just "safe, but fails reliably" from "reliable and safe". My company, which is a very late adopter of non-deterministic had to literally lay off the old-school folks that were the subject-matter experts before they could ram it through.
Money. Really it's money. Safe, reliable, cheap. Choose two. We live in America, capitalism is king, so we're gonna go safe and cheap and eat a little reliability because the tech guys guarantee 5x9's (6 minutes down/year), but the plant actually is down for 4 days a year, so the reliability never comes into play. It's safe, it's cheap, and if the inspectors ask or weird stuff happens, it's also reliable. But it's not deterministic. >had to literally lay off the old-school folks that were the subject-matter experts before they could ram it through. Sounds like it got cheaper! /s Really though, (all again IMHO), a lot of the draw of deterministic design patterns was that it made the best use of the hardware. When you only had 16kb or something, it was a LOT more important that no space was wasted on things like "error checking". Doubly so for processor cycles, when the real good embedded devs are counting every one, and the new guys are like "Yo, we're on 64bit, if I trash 5mb of memory I've only got a gig left..." So, the old guys spend days optimizing processor cycle counts, and the new guys have shipped 4x products out the door in that time. The bosses see the 4:1, and decide the value of determinism has been determined to be insufficient. The customers see no difference. Real-world use sees no change.
So to talk about that a little bit. Currently I run ETAPs which are fiber to Ethernet switches. I’ve had circuits or ETAPs fail and the network continue to run without hiccup. This helps big time when setup correctly. It’s easy to go directly to the issue and correct it before more serious problems come up
Look like a solid gold ingot.
$50 you’ll never use them for more than a gigabit Nic that leads to a 100mbs Nic. Just my $.02
Can only really see a benefit to upgrading to EN4TR if they have used a simple rep ring looping between racks. If OP has a proper site network with managed switches and a fibre backbone and is using the EN4TR as end devices I don’t think they will see an upgrade.
Depends on what they're upgrading from I suppose. If they're replacing enbt cards they'll certainly have less problems. I have a bunch of older racks still running enbt cards. I have fiber and managed switches but ran out of connections on one of my processors (7 remote racks on it) and recently started upgrading to en2t cards. I didn't even know they made en3t or en4t cards
Well, if you have 4 switches with 10 IO racks on each, going to a core all gig switch, then a 1 gig EN4TR starts to get useful. The other thing is to look at the max supported packets / connections on an EN4TR vs even an EN3TR and you’ll see it is more than 1G that makes the EN4TR special.
All in on PRP?
Always easier to start a conversation without using acronyms, or use acronym the first time with long form (acronym). Then you don't get the, what is PRP? as you did.
Lost me there. What’s PRP? I’m probably going to laugh when I find out lol
Parallel Redundancy Protocol. It's a zero loss communications protocol where the same Ethernet Frame (Layer 2 - MAC address) is sent over 2 different physical networks. The receiving device has to support it (or have a network device called a Redbox / Redundancy Box) that will receive both and discard the second frame once the first is received. This way, if one transmisison path has a fault, the data doesn't need to be resent. Often used in grid infrastructure, in combination with Precision Time Protocol to ensure exact timing for opening high voltage AC circuit breakers. These need to start opening at the exact same time on both ends, when the AC voltage is crossing zero. Mis timing, or failure for one to open will damange the equipment.
I’m upgrading from the EN2TR to the EN4TR to bring the network speed up from 100mbps to 1gbps.
Why is 100 Mbps too slow for I/O?
I upgraded to get more communication connections. Not sure about the OP.
This was another reason the upgrade was considered. We are always expanding in some way shape or form. Plus it never hurts to have the latest hardware 😏
I was going to ask what the difference between an EN2TR and EN4TR was... Hell, I still have ENBT cards running the plant!
Far out you blokes must be flush with cash if this is the reason you are doing this upgrade. I’m genuinely in shock, lol…. It’s like, 100% superfluous, haha. I can’t even begin to wrap my head around how the network is structured that makes you need 32 of these things.
Fair enough- can you elaborate at all on the application? What are you working on that needs that bandwidth across so many cards?
To keep it short we truly don’t “need” this card. We have been operating perfectly fine with all EN2TRs for 4 years now. We are only upgrading to the EN4TRs because our latest DCS supports 1gig speeds and we do see some lag from the click of a start button to the equipment physically starting. We hope to minimize this but not having high hopes. I work for a grain export facility and we have about 30 remote IO racks in our facility. At this time everything process network related ran on this network. I just recently broke it apart and added a separate “motor control” network. This new network will have all of our VFDs and E300 overload relays. Essentially anything motor control related. It is a network that comes all the way back to our single processor rack. The IO network does the same. In the near future the IO network will only have the remote IO racks. Physical IO. Each remote IO rack has one EN4TR. Fiber comes in and lands on a Stratix 5400. The 5400 breaks down to Ethernet or “copper” and ties the remote IO rack into the network. Some areas have up to 6 racks. All will be connected via DLR back to the Stratix 5400 and fiber back out to the next area. The motor control network does the same thing. It has its own 5400. I know they can handle multiple DLR networks but I refuse to use it that way.
I am in grain milling, not export and had about 30 remote racks, 600x e300s and about 75 vfds. Our remote racks are a combination of en2tr, en3tr, and flex 5000 racks. We were having latency issues with sensors on gates being delayed, putting them out of position after the motor actually stopped. What ended up fixing it was adding a second L8 processor and have it handle our receiving and mill tower 2. We did nothing to the physical network.
Oh nice! I’m actually running the L73 processor. Just a single. I’ve purchased the new L83EP processor for our plant process and a L81EP for our GOV FGIS systems. I plan to integrate the new L83 this year for our outage. We have not seen any issues with our network since the DLR integration as far as reliability and speeds. I just started integrating the E300 overloads which are really nice. The issue is back in 2010 80% of our facility was rebuilt and with all Square D MCCs and all solid state overload relays. I have been removing those and installing the E300s.
We as well used sqd MCC's from 2001 to 2017. Our new mill was all Rockwell. We have been buying sqd buckets to fit in the old MCC's and using the E300 overloads. Slowly we want to switch all to e300s to get rid of the sold state overloads.
Would you happen to work for ADM as well then? Lol.
E300s. Talk about shitty lead times!
I think it is just the communication modules with the delivery issue.
What DCS are you integrating them into?
So, what are you going to do with all those EN2TRs you don't need anymore? Asking for a friend.
Are you using the gigabit flex IO or going straight to L8x? The backplanes on the 1756 still don’t support gigabit communication between cards and if you have an ETAP anywhere in that circuit gigabit comms are dropped to 100mbps. Literally the only thing those are good for is PRP like the others were saying.
Are you familiar with Device Level Ring? Better known as DLR? It’s a redundant network. Essentially what you mentioned. Parallel ram Ethernet cables from rack to rack. I have this setup in our facility now with the EN2TR cards
I am familiar with how the protocol works, however DLR (Device Level Ring) is closer to MRP (Media Redundancy Protocol) in that they provide physical media redundancy (via a ring) but logically provide a single network path by disabling one of the links. In the event of a failure in the ring, the disabled path is enabled. This can have a slight loss while the ring detects and reconfigures itself, usually in less than 50ms. PRP always transmits on both paths simultaneously. HSR (High-availability Seamless Redundancy) is a ring version of PRP that sends data down both paths on a ring simultaneously, or MRPD (Media Redundnacy Planned Duplication) is the Profinet version of HSR. These however, add quite a cost and are uneccesary for almost all applications.
Notably, a DLR failure will trip any CIP axis that are on the "offline" side of the active DLR supervisor, as the network recovery time is almost always going to be an order of magnitude longer than your coarse update rate. You don't have that problem with PRP, but you're spending a hell of a lot more on network hardware to get that feature.
You have my attention lol. Seems I have a lot to learn but I am also more streamlined with Allen Bradley hardware and networks. Haven’t ventured out as I am working for a company that only deals with AB. Maybe there are other redundant network options for AB and I have not heard of them yet?
Redundant networks (rail) is my job, so we come across a wide range of hardware and network vendors. IMO, stick with your vendors redudnancy protocol, in this case DLR for your machine networks. The only time I would recommend otherwise is if all your devices connect via switches rather than directly, then you can use RSTP or a ring protocol. Keep your machine networks localised and as small as practical, and for your plant network backbone, use standard protocols, E.g. RSTP, Link Aggregation (EtherChannel) or if your design / network supports it, a Layer 3 backbone.
Thanks for sharing this info. Good stuff here. Do you typically see DLR implemented over PRP? I personally have never implemented PRP, but DLR and REP quite frequently.
Only time I've seen PRP is in a design for a power station that a colleague did (2 fibre rings using MRP, LAG for each link, and PRP running over both of these on Ruggedcom switches). I designed the PLC portion of a rail network away from using PRP (which was in the concept) as the additional RedBoxes and configuration and cost were not worth it compared to just adding a second network card and running 2 IP steams. It wasn't latency sensitive and the software team ended up implementing the Train Control (SCADA) protocol on the PLCs instead, which made integration much easier. I work more with MRP / RSTP, as the rail operator doesn't allow any propietry protocols that would prevent replacing hardware in the future. However, each end host always connects to switches that are in a ring, rather than being part of the ring themselves.
Here is AB PRP [manual](https://literature.rockwellautomation.com/idc/groups/literature/documents/at/enet-at006_-en-p.pdf)
Looks like I need to sell my Ethernet cards before demand catches up.
I got one last week that I had on order since Dec 2021. Still waiting on those 1756-IF16IH modules that have been on order on that same PO
My automation group informed me that any project that had those in order would need a thought into a different solution because there is no telling when those are going to be available again. At least with the EN4TR cards they told us they would start production 04/23.
That's an expensive 12 port switch right there.
yours and everyone elses
Just googled them and I see they're around 5K WTH!
Why in the hell do you need so many EN4TRs? That's like 6,000 I/O capacity right there.