Page Speed should be prioritized above all else, here is why:
Have you ever clicked on a link on a mobile phone and then ... ... and then ... ... ?
As of 2015, the number of mobile-only internet users exceeds desktop-only in the U.S. So, if you're not designing for mobile, you're not designing for most of the population. And, as we've already established, mobile connections are often slow.
Now, you may be saying, "yeah, but I've got the latest iPhone with the iXX chip, it's super fast". Right, well, unless your target audience is exactly like you, many are probably still using an Android phone from 3 years ago that's not quite as fast.
We believe in delighting users and nothing is a worse buzz-kill than a page not loading or taking several seconds to do so. We target < 1 second to page load.
My guess is you may have run some ads on Facebook or a similar platform to drive traffic to your site. You're also likely using a service to track visitors and how they interact with your product.
So, tell me, how much information do you get from users who wait for a page load, only to bounce before it loads succesfully. You get nothing useful. If you're unlucky, you may trigger a page view and the bounce could lead you to believe something is wrong with the content of your landing page.
If users never make it to the content, you'll never know if they like it.
As of 2010, Google has been using site speed in its ranking algorithms. Recently, Google has endorsed projects like AMP which focuses on extremely fast sites.
It goes without saying that Google rankings are very important to SEO. So, if you care about SEO, make sure your site is fast.