The server is the first thing people blame for a slow site, yet on most sites it accounts for a small share of the total time. The real weight is in the files you send to the browser.
Images
On nearly every slow site this is the first cause. Serving a 4000px photo into a 400px slot makes the user download tens of megabytes for nothing. Producing images at the size they are actually used, in a modern format, makes a real difference on its own.
JavaScript
Every kilobyte you ship has to be downloaded, parsed and executed. Keeping code the page doesn't need out of the main bundle is the biggest win after images.
Fonts
If text is invisible until a webfont loads, the user is looking at an empty page. A font-display setting and preloading the font close that gap.
Third-party scripts
Analytics, a chat widget, a map, an ad pixel. Each looks small alone; together they are often larger than your own code. The only useful question here is what you would actually lose by dropping one.
Don't start without measuring
Optimizing on a hunch is the easiest way to miss where the time is really going. Measure first: once you can see how long each file takes, the priority order writes itself.

