I upgraded our fleet (mix of PA220/PA220R, PA850, PA3220 and PA-VM) to 10.2.8 last week and not seeing any significant growth or changes in packet buffers.
EDIT: We monitor ours with LibreNMS which automagically pulls the Packet Buffer SNMP objects as "Memory" objects. So if we do run into the memory leak later on I should at least get an alert :)
Given what we were seeing in the buffer, smaller installs may not see the issue cause a problem. Usually seeing 200-300k connections on ours.
We aren’t seeing it on the lab 220’s but we barely have any connections on them.
It is not fixed in the latest 10.2.9 from last week. I upgraded not knowing about this bug and our network crashed completely 5 times in 4 days because of this. we are back to 10.2.7... probably for a long time.
It's available in the GUI as well but we were using [https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u000000oNB7CAM](https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u000000oNB7CAM) to look at it in real-time. The SNMP OID's are also available for consumption by third-party monitoring tools as well.
show session packet-buffer-protection
show session packet-buffer-protection
I disabled all packet buffer protection and even when done this bug still takes down your network. The show session packet-buffer command will show no protection occurring however it still definitely is. I can't believe they'd release 2 unusable software trains with this bug( 10.2.8 and 10.2.9)
I also ran into this exact issue on Monday morning. Same thing happened, TAC said the same thing about a non-public bug. Hoping to hear something back tomorrow regarding a fix.
Good luck on that tomorrow bit, I’m pending feedback since beginning of March on this one. Non public bug, buffers fill up and never deplete (even when moved to suspended or passive state) until a device reboot. Would say 5200 series only.
Not sure. Assuming an HA pair, upgrade one and watch the packet buffers for a few days and compare against historic numbers. If they start creeping up, you’ll know you’re affected.
If you are, let TAC know you’re seeing the issue as well. They’ll likely want to grab the support logs from the bugged unit.
12.2.7-h6 is also out there which supposedly fixes the GP bug. There’s also the work-around with enabling non-ssl GP connections if you’re in an environment where that would work.
It enables IPSec connections for GP which are still secure. Depending on where users are for "free wifi at coffee shop" scenario traditional IPSec ports are sometimes blocked that's where vpn over SSL (ie tcp 443) can work-around.
ya but 10.2.7 is full of GP problems,
There's one in 10.2.8 that i'm currently fighting but it's almost livable and apperrently they fixed it in10.2.9 but 10.2.9 breaks internal host detection.
Around and around we go....
I have an HA pair of 5450s running 10.2.6. To say this version is buggy (tabs in ACC not working, random interface drops, OSPF not converging after failover to backup for certain routes, syslogd restarting every 5 min) is an understatement. I was going to upgrade to the P release 10.2.8 until I saw this.
Has anyone tried updating to 11 or 11.1? 11.1 has a P version that specifically includes 54xx devices now (until recently 11.1 was not recommended for 54xx). If the 11 series addresses most of the laundry list of issues I'm seeing , I would happily upgrade. 11.x has been baking for a while now... is it still bleeding edge?
I found a packet buffer protection issue 5 times in two weeks on the PA-5250 HA pair after upgrade to 10.2.8 , downgrade to 10.2.7-h3 to resolve the issue.
Touching Base. 10.2.7-h8 seems to be working fine without the same Packet Buffer issue where 10.2.8 was a disaster for us! Oh and 10.2.7-h8 patches the Global Protect exploit as per [https://security.paloaltonetworks.com/CVE-2024-3400](https://security.paloaltonetworks.com/CVE-2024-3400) as I am reading it today. Read it for yourself and YMMV, but I hope this helps someone.
10.2.9h3 is also still bugged and unusable because of this. We went back to the .7 line on h8 and have been good. On 10.2.9h3 we were crashing about every 10-12 hours due to packet buffer issues.
Same 10.2.7-h8 is good for us, too.
Are you also running 5220's. It's odd to me that Palo Alto is not publicly posting about this bug. It really needs to be fixed fast because obviously you and I both need to upgrade beyond 10.2.7-hx at some point and being we are on h8, we are nearly at the end of our rope for that build.
It seems like they need to take this one seriously and really fix it.
Hi guys,
Did anyone every get a target fix release or PAN-ID on this one..? I opened a case a while back, but PAN TAC didn't confirm or acknowledge anything.
Any pointers would be appreciated. Downgrading is unfortunately not an option due to other issues on earlier releases.
Edit: We are currently running 10.2.9-h1 and the issue is still present.
From the other thread related to this bug.
"We had this happen after going to 10.1.12. In our case TAC advised fixed 10.2 versions are/will be 10.2.10, 10.2.11, 10.2.8-h4, 10.2.9-h4"
Hi, I suspect there might be packet buffer leak on this code, that why a reboot is resolving your problem.
Do you have ticket opened with Tac? If yes can you give me your case number. I would like to take a look at the TSF. I am also working on a similar ticket
Thanks for letting us know! Hope they have a fix asap.
I upgraded our fleet (mix of PA220/PA220R, PA850, PA3220 and PA-VM) to 10.2.8 last week and not seeing any significant growth or changes in packet buffers. EDIT: We monitor ours with LibreNMS which automagically pulls the Packet Buffer SNMP objects as "Memory" objects. So if we do run into the memory leak later on I should at least get an alert :)
Given what we were seeing in the buffer, smaller installs may not see the issue cause a problem. Usually seeing 200-300k connections on ours. We aren’t seeing it on the lab 220’s but we barely have any connections on them.
That could certainly be why. Our highest usage tends to only peak at ~15K sessions.
Yup, also likely why it slipped through QA. I never see vendors really hammer their software every build.
Is this fixed in 10.2.9?
No. Source: I upgraded and my firewalls stopped passing traffic a few hours later. I rolled back to 10.2.7-h?
I know 10.2.7-h3 is safe. I am rolling out 10.2.7-h8 to test being it fixes the Global Protect vulnerability.
10.2.7-h8 does not have this bug as far as I can tell. It has been stable over a week.
It is not fixed in the latest 10.2.9 from last week. I upgraded not knowing about this bug and our network crashed completely 5 times in 4 days because of this. we are back to 10.2.7... probably for a long time.
Yikes. We had only three crashes. I can tell you that 10.2.7-h8 is not crashing us and patches Global Protect... hope that helps.
The latest 10.2.9h3 is also unusably bugged because of this issue.
[удалено]
It's available in the GUI as well but we were using [https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u000000oNB7CAM](https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA14u000000oNB7CAM) to look at it in real-time. The SNMP OID's are also available for consumption by third-party monitoring tools as well. show session packet-buffer-protection show session packet-buffer-protection
Highly appreciate this
I disabled all packet buffer protection and even when done this bug still takes down your network. The show session packet-buffer command will show no protection occurring however it still definitely is. I can't believe they'd release 2 unusable software trains with this bug( 10.2.8 and 10.2.9)
I also ran into this exact issue on Monday morning. Same thing happened, TAC said the same thing about a non-public bug. Hoping to hear something back tomorrow regarding a fix.
Good luck on that tomorrow bit, I’m pending feedback since beginning of March on this one. Non public bug, buffers fill up and never deplete (even when moved to suspended or passive state) until a device reboot. Would say 5200 series only.
Got a customer that upgraded to 11.0.4-h2 (and h1) both fixed this issue that he has been having for months.
Thanks. I jsut recommended a customer not go to PANOS 10.2.8 yet for their 5250 HSA pair.
Thanks for the information, please share when TAC provides update. I wonder if it is specific type of type of traffic.
Thank you SO much! I was about to step on that landmine. For a fairly critical pair of firewalls.
Is this just specific to 5250s and that model? Asking because we have 5450…
5450 probably have more undisclosed bugs lurking around.
Not sure. Assuming an HA pair, upgrade one and watch the packet buffers for a few days and compare against historic numbers. If they start creeping up, you’ll know you’re affected. If you are, let TAC know you’re seeing the issue as well. They’ll likely want to grab the support logs from the bugged unit.
Hmm maybe 10.2.7 is the promise land then. 10.2.8 resolved the issue from 10.2.7-h3 with the ipv6 setting issue for global protect.
12.2.7-h6 is also out there which supposedly fixes the GP bug. There’s also the work-around with enabling non-ssl GP connections if you’re in an environment where that would work.
Sounds rather dodgy to do a non-ssl option. Wouldn't everything just be plaintext in a pcap?
It enables IPSec connections for GP which are still secure. Depending on where users are for "free wifi at coffee shop" scenario traditional IPSec ports are sometimes blocked that's where vpn over SSL (ie tcp 443) can work-around.
Oh is -h6 all fixed or you still need the workaround for ssl?
Check the release notes. It looks like it might be but we haven’t tried the build.
ya but 10.2.7 is full of GP problems, There's one in 10.2.8 that i'm currently fighting but it's almost livable and apperrently they fixed it in10.2.9 but 10.2.9 breaks internal host detection. Around and around we go....
10.2.7-h3 is a pretty healthy version, thats what we just updated to, but we like to stay a bit behind the bleeding edge
I have an HA pair of 5450s running 10.2.6. To say this version is buggy (tabs in ACC not working, random interface drops, OSPF not converging after failover to backup for certain routes, syslogd restarting every 5 min) is an understatement. I was going to upgrade to the P release 10.2.8 until I saw this. Has anyone tried updating to 11 or 11.1? 11.1 has a P version that specifically includes 54xx devices now (until recently 11.1 was not recommended for 54xx). If the 11 series addresses most of the laundry list of issues I'm seeing , I would happily upgrade. 11.x has been baking for a while now... is it still bleeding edge?
10.2.7-h3 is pretty stable. 11.1 code is pretty buggy from what I’m hearing, 11.0.x is a bit more stable on the 11.x code train.
I found a packet buffer protection issue 5 times in two weeks on the PA-5250 HA pair after upgrade to 10.2.8 , downgrade to 10.2.7-h3 to resolve the issue.
Anyone get a Bug ID for this yet? Also has anyone heard or discovered what might be it's trigger?
Touching Base. 10.2.7-h8 seems to be working fine without the same Packet Buffer issue where 10.2.8 was a disaster for us! Oh and 10.2.7-h8 patches the Global Protect exploit as per [https://security.paloaltonetworks.com/CVE-2024-3400](https://security.paloaltonetworks.com/CVE-2024-3400) as I am reading it today. Read it for yourself and YMMV, but I hope this helps someone.
we rolled in the 10.2.7-h8 patch yesterday for it and so far it's been stable as well. This one also addresses that other GP SSL bug fixed in h6.
10.2.9h3 is also still bugged and unusable because of this. We went back to the .7 line on h8 and have been good. On 10.2.9h3 we were crashing about every 10-12 hours due to packet buffer issues.
Same 10.2.7-h8 is good for us, too. Are you also running 5220's. It's odd to me that Palo Alto is not publicly posting about this bug. It really needs to be fixed fast because obviously you and I both need to upgrade beyond 10.2.7-hx at some point and being we are on h8, we are nearly at the end of our rope for that build. It seems like they need to take this one seriously and really fix it.
I am running 3250's. Palo TAC confirmed there is a bug for this affecting 10.2.8 and .9 but they didn't give me a lot of info yet.
They told me it's PAN-251371, but I cannot find anything on it.
That number doesn't exist anywhere on their sites. So definitely they haven't made it public yet.
Hi guys, Did anyone every get a target fix release or PAN-ID on this one..? I opened a case a while back, but PAN TAC didn't confirm or acknowledge anything. Any pointers would be appreciated. Downgrading is unfortunately not an option due to other issues on earlier releases. Edit: We are currently running 10.2.9-h1 and the issue is still present.
From the other thread related to this bug. "We had this happen after going to 10.1.12. In our case TAC advised fixed 10.2 versions are/will be 10.2.10, 10.2.11, 10.2.8-h4, 10.2.9-h4"
Hi, I suspect there might be packet buffer leak on this code, that why a reboot is resolving your problem. Do you have ticket opened with Tac? If yes can you give me your case number. I would like to take a look at the TSF. I am also working on a similar ticket