nginx with set_real_ip_from AND allow/deny proxy only

Hello,

I'm trying to set up nginx to work with CloudFlare.
I want 2 separate things that don't seem to work together:

1. I want to only allow connections from a list of CloudFlare IPs, rejecting any direct access that might bypass it. This can be easily done with an allow list of IPs followed by `deny all`.

2. I also want to get the real visitor IPs. This can be done with `set_real_ip_from` and `real_ip_header CF-Connecting-IP`.

When put together this falls apart, because I no longer have the proxy IP, but only the real one. Even if I put a geo $isCF {x.x.x.x 1;} in the http block and then do an if{$isCF=...}, I $remote_addr is always evaluated to the real ip.

Is there any way to have both things working?

Thanks.