Disable NGINX caching 304 Responses from Origin Server

We have a pretty simple setup with NGINX sitting on the front and a backend server (on a separate physical server) that provides the content.

Nginx then caches content based on the EXPIRES and Cache-Control headers set by the origin server.

We noticed that NGINX was not issuing 304 headers to images that were not in the local NGINX cache when the If-Modified-Since header was sent by the client. Instead, it would issue a 200 with the full data file.

To fix this, we applied:
proxy_set_header If-Modified-Since $http_if_modified_since

So then the If-Modified-Since header was passed to the backend and of course, it returned correctly with the 304 header - great.

But what we noticed was that NGINX would cache this 304 response and deliver future responses as 304 to clients even without the If-Modified-Since header.

How can we disable caching of 304 responses and fix this issue?

Thank you for your help, suggestions, and tips in advance.