Michael McNamara

technology, networking, virtualization and IP telephony

PanOS 9.1.12 breaks GlobalProtect VPN

by

When possible it’s always a good idea to test any software upgrades, because you just never know what your going to get. That was the case recently when I upgraded our test PA-220 from 9.1.7 to 9.1.12-h3 and seemingly breaks all GlobalProtect VPN functionality. The portal doesn’t respond on TCP/443 at all, so it looks like the firewall itself is dropping the traffic.

The issue turned out to be Strict IP Address Check which was just “resolved” or enabled in 9.1.12.

AN-175934 Fixed an issue where packed-based zone protectio settings (such as
Strict IP Address Check) were not applied to return traffic.

When I disabled Strict IP Address Check on the zp_untrusted zone protection profile GlobalProtect started working again.

What is Strict IP Address Check?
Check that both of the following conditions are true:

  • The source IP address is not the subnet broadcast IP address of the ingress interface.
  • The source IP address is routable over the exact ingress interface.

If either condition is not true, discard the packet.

Looks like a bug to me.

Cheers!

COVID-19 The War Waged by Information Technology Professionals

by

The past few weeks have been extremely exhausting both professionally and personally. Coronavirus (COVID-19) has taken the world by storm and is literally upending people’s daily lives and ruining businesses large and small. Let’s not forget the large number of people that have lost their lives to this virus. My thoughts and prayers for all those who have lost love ones. My thanks and admiration to all those medical professionals on the front lines treating the sick.

While very few of us have planned and organized days these past few weeks have been unlike anything I’ve ever experienced, running from one fire to another, one disaster to another. Whether it’s a power failure in a data center or someone deciding to water the potted plant that they hung over the network switch, there’s always some new emergency or problem that requires IT to jump in and save the day. This event was no different but the scale and duration was a whole new experience for everyone.

We started mobilizing our disaster preparedness plan around the middle of February. The initial request from the leadership team was pretty straight forward, “How do we prepare to have our home office employees and call centers agents work remotely?”. Like most large-medium sized enterprises we have a couple of hundred people working remotely every day, however we were talking about going from 200-300 daily remote users to potentially 3,000-4,000 daily remote users in a very short time span. And a significant portion of those users still had desktop devices.

In the span of a week we had ordered, imaged, configured and deployed (shipped or handed out) over 400 laptops to over 400 employees and call center agents. We also spun up a new Virtual Private Network (VPN) solution using Palo Alto Network’s GlobalProtect to help supplement our existing Pulse Secure and Microsoft Direct Access solutions.

I should note that I reached out to Pulse Secure and they offered us a temporary 60 day license to help us cope with the additional users – kudos to Pulse Secure.

Like everyone we’re in the middle of our second week and the Internet itself is starting to show it’s cracks. This past Monday and Tuesday we experienced connectivity issues across 30 stores in and around London, UK for ~ 45-60 minutes at a time. We later learned that Monday was the first day in the UK with all schools closed and British Telecom (BT) wasn’t handling the strain well. I’m sure it’s not helping BT that Disney+ just launched in the UK and Ireland on Wednesday.

We’ve had a number of issues with Microsoft, Slack and Zoom over the past two weeks and expect those issues will likely continue as more and more people around the nation and globe transition to working remotely.

Nobody’s really sure what the future holds… hopefully things will start to improve as we work to flatten the curve.

Thanks to all the IT folks that are continuing to carry on the struggle, be it onsite or from the confines of your own home…we know what what your’re going through and we appreciate your efforts!

If you have story to share, let us know below.

Stay safe! Cheers!

Pulse Secure Client – Invalid or Missing Certificate

by

I ran into an interesting problem recently on my Windows 10 laptop running the Pulse Secure VPN client where I started recieving an “Invalid or Missing Certificate” warning when trying to connect to the Pulse VPN appliance (formerly Juniper Secure Access appliance). I pulled the log file from the client (C:\ProgramData\Pulse Secure\Logging\debuglog.log) and found that the Pulse Secure client was serving up an old certificate that didn’t appear to be installed on my laptop. I searched for it using the Microsoft Management Console and the Certificate snap-in but had no luck.

I eventually realized that the Pulse Secure client was “caching” the certificate selection because I had chosen to “Save Settings” in the client when originally prompted to choose which SSL certificate the client should provide while authenticating. 

The solution.. go into the client and select “Forget Saved Settings“.

The next time I attempted to connect I was prompted to choose which certificate to use for authentication purposes.

Cheers!

Verizon FiOS Internet – Juniper Private VLANs

by

I recently stumbled over an interesting problem with Verizon’s FiOS Internet service while doing some consulting. In an effort to protect the innocent and prevent and ass hattery, I’ve changed the IP addressing to use something from RFC5737.

A client had two physical sites about 1 mile apart which were connected to the Internet by separate Verizon FiOS broadband connections and which were assigned the following static IP addresses;

Site A:

IP Network: 198.51.100.226/28
Subnet Mask: 255.255.255.0
Default Gateway: 198.51.100.1
Usable IP Addresses: 198.51.100.226 – 198.254.100.238

Site B:

IP Network: 198.51.100.50/28
Subnet Mask: 255.255.255.0
Default Gateway: 198.51.100.1
Usable IP Addresses: 198.51.100.50 – 198.51.100.63

Let me be the first to admit that the information above isn’t quite right… there is no IP address block 198.51.100.226/28, it should be 198.51.100.224/28. I believe that’s Verizon trying to avoid having customers accidentally use the network address or the first address in the IP address block which is likely reserved for the actual Verizon Actiontec router.

The client was trying to establish a VPN tunnel between the two sites and was running into difficulties. The issue was with the IP addressing provided by Verizon and it’s likely implementation of private VLANs on the Juniper hardware. I’m assuming that Verizon is likely using PVLANs to isolate traffic between individual customers to minimize the number of IP subnets they need to create. Instead of creating 16 /28 IP networks they are using a single /24 network and then isolating the traffic between customers using PVLANs. The issue in the example above is pretty obvious – the individual client devices are attempting to communicate with each other on the local subnet. Believing that there’s no need to signal the upstream router because the netmask indicates that the remote site should be in the same IP network. While the remote site is actually in the same IP network, the implementation of PVLANs is blocking communication between the client devices.

Anyone have any experience with Verizon FiOS using PVLANs?

I believe I heard years ago that Verizon chose Juniper for their FiOS implementation.

Cheers!

Reference: Juniper – Understanding Private VLANs on EX Series Switches

Recent Posts

Categories

SUBSCRIBE BY EMAIL