connection timed out (110: Connection timed out) while proxying connection

My config looks likes this:

stream {
server {
listen 9999 ssl;
tcp_nodelay on;
ssl_certificate /etc/nginx/cert.crt;
ssl_certificate_key /etc/nginx/cert.key;
ssl_protocols TLSv1.2;
ssl_session_cache shared:TLS:2m;
proxy_pass xxx.xxx.xxx.xxx:8888;
}
}

Under heavy traffic, things look fine. Under low traffic, I get lots of:

2018/04/10 19:10:47 [info] 8856#8856: *1 connection timed out (110: Connection timed out) while proxying connection, client: yyy.yyy.yyy.yyy, server: 0.0.0.0:9999, upstream: "xxx.xxx.xxx.xxx:8888", bytes from/to client:11417/75978, bytes from/to upstream:75978/11417

Why? How do I fix that?

Site note: I use to use stunnel which had some TCP keepalive related settings. I was wondering whether I need something like this with nginx. Is there a way to send TCP keep alives to xxx.xxx.xxx.xxx:8888 ?