Server Response Time > 10 seconds for admin panel

Most of the time (not every time but the majority) the admin panel takes between 3-20 seconds to initially connect/load - once I'm "inside" I can navigate it just fine with minimal delays, but that first connection is extremely slow. Using the Network Waterfall tool in Chrome Devtools I can see its the "Server Response Time" that matches this waiting period for anything to start loading on the page. This slowdown only happens on the initial connection to the admin panel, the rest of the admin panel is pretty responsive and so is the public side of the website. What is causing this?

@treminaor, The tasks are trying to fetch data from Grav's servers. Maybe there's a delay over there?

But there is 9 seconds of delay before the tasks fetch even begins, right?

@treminaor, You're right, my attention was drawn to the red tasks in the image, and not the first line...