Submit Query to a particular server behind NGINX

I am querying an NGINX server for static file, I believe there are more than one upstream servers behind the NGINX at :80. I say this by the two different 'Server' headers I receive in the response. One server is significantly faster than the other one, is there any way I could direct my request to a particular server from the client side ?
Or is there any way to re-set the proxy_cache_key associated with my requests from the client side ? so that my requests will go to the good server consecutively ?

Any guidance is greatly appreciated.

Thank you