Serving static files with Nginx also helped a lot. That was a "doh" moment.