Reduce response times

I'm running a VPS with 1GB with Debian 7. Nginx's response time for static HTML files are around 300-500ms, which seems relatively high. These are not spikes, it's the average response time of single requests. No load tests or anything...

Is that normal? Shouldn't all static content be delivered in under 100ms?

This is my config: http://pastebin.com/BME76CdJ

Thanks a lot,
Simon

I'm only referring to the "Waiting" time. I'm ignoring the DNS Lookup, Connection time and Receiving time, not included.