default_server and access by IP

I have many sites on different IPs. I need:
1. if user open site with domain name - open site;
2. if user open site with IP - open default html page.

What is the best solution for this task? I tried:
~~~
# default.conf
# default settings for all unknown server names, including requests by IP
server
{
listen 80 default_server;
listen 443 default_server;
server_name _;
root /path/to/dir/
index index.html
}

# my_site_name.conf
server
{
listen 123.123.123.123:80
server_name my_site_name.com
...
}
~~~
But with these settings, if access with IP(lynx 123.123.123.123), nginx use server-settings where I mention this IP (my_site_name.conf). But this request doesn't has server name, only IP. Is it correct?

thank you.