Block file access in specific directory

I need to block access to log files that are stored in a publicly accessible directory from being viewed publicly. These are stored as CGI files and as things are right now, anyone can view the directory directly and download the .CGI file which is a huge security concern as it showcases IP address of visitors and other things.


How can I block access to "/path/to/the/directory/*.cgi" ?

I tried:

location /path/to/the/directory/*.cgi {
deny all;
}

But it did not work.