Spock, Uhura, Chapel, heck even M’Benga don’t make it a prequel, but a lieutenant Kirk does?

Nice job. Packet loss will definitely cause these issues. Now, you just need to find the source of the packet loss.

In your situation, I’d first try to figure out if it is ISP/Internet before looking inside either network. I wouldn’t expect it to be internal at these speeds. Though, did you get CPU/RAM readings on the network equipment during these tests? Maxing out either can result in packet loss.

I’d start with two pairs of packet captures when the issue happened: endpoint to endpoint and edge router to edge router. Figure out if the packet loss is only happening in one direction or not. That is, are all the UK packets reaching DE but not all the DE making it back? You should clearly be able to narrow into a TCP conversation with dropped packets. Dropped packets aren’t ones that a system never sent, they’re ones that a system never received. Find some of those and start figuring out where the drop happened.

Just curious if you’ve had the chance to dig into this and can report anything back?

If the bandwidth numbers you’ve described are accurate, I’d start looking at CPU and RAM usage on the network device. The Fortigates are going to be doing extra work to handle the VPN. I wouldn’t expect an IPSEC VPN on a Fortigate to top out at 10mbps, but if it’s doing a lot of other work, it’s possible. ACL’s on the Cisco devices? You run the potential of CPU/RAM exhaustion on those. Hopefully, you have remote monitoring on all network devices and you can just look at the history when these transfers are happening.

If nothing obvious there, then I’d try packet captures when this is happening, perhaps to start on the system doing the ssh and on one or two others experiencing issues. What are you seeing? Evidence of dropped packets? High latency? If dropped packets, start capturing the same traffic on the network devices it’s flowing through.

Kids these days with their containers and their pipelines and their devops. Back in my day…

Don’t get me started about the internal devs at work. You’ve already got me triggered.

And, I can just imagine the posts they’re making about how the internal IT slows them down and causes issues with the development cycle.

Yep. I’ve hosted my own mail server since the early oughts. One additional hurdle I’d add to you list is rDNS. If you can’t get that set up, you’ll have a hard time reaching many mail servers. Besides port blocking, that’s one of the many reason it’s a non-starter on consumer ISP.

I actually started on a static ISDN line when rDNS wasn’t an issue for running a mail server. Moved to business class dsl, and Ameritech actually delegated rDNS to me for my /29. When I moved to Comcast business, they wouldn’t delegate the rDNS for the IPv4. They did create rDNS entries for me, and they did delegate the rDNS for the IPv6 block. Though the way they deal with the /56 IPv6 block means only the first /64 is useable for rDNS.

But, everything you list has been things I’ve needed to deal with over the years.