Pinning traffic between proxy and server

I have a relatively simple load balancer setup, here's the setup

192.168.1.0 is the "outside" client facing network
192.168.10.0 is the "inside" server facing network

eth0: 192.168.1.10
eth0:1 192.168.1.11
eth1: 192.168.10.10
eth1:1 192.168.10.11

http {
upstream server1.domain.com {
server 192.168.10.20:8080;
keepalive 16;
}
upstream server2.domain.com {
server 192.168.10.21:8080;
keepalive 16;
}

server {
listen 192.168.1.10:80;
server_name server1.domain.com;
location = / {
proxy_pass http://server1.domain.com;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
}
location / {
proxy_pass http://server1.domain.com;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
}
}
server {
listen 192.168.1.11:80;
server_name server2.domain.com;
location = / {
proxy_pass http://server2.domain.com;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
}
location / {
proxy_pass http://server2.domain.com;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
}
}
}

As you can see I just want to listen for traffic on the 192.168.1 network and reverse proxy it to backend servers on the 192.168.10 network. What I can't figure out how to do is pin traffic for communication between the proxy and the backend server to a particular IP. I would like to have traffic for server1 use 192.168.10.10 when sending requests to 192.168.10.20 and to use 192.168.10.11 when sending traffic to 192.168.10.21. Is it possible to do this with nginx? I'd rather not have to do any odd iptables or routing to make this work.

Thanks!