Database session frequently expired (before sess_expiration)

I'm using CI 3.1.10. I'm using database to save the sessions and here is my config:

$config['sess_driver'] = 'database';
$config['sess_cookie_name'] = 'ci_admin_session';
$config['sess_expiration'] = 14400;
$config['sess_save_path'] = 'ci_admin_sessions';
$config['sess_match_ip'] = FALSE;
$config['sess_time_to_update'] = 14400;
$config['sess_regenerate_destroy'] = FALSE;

I have tried setting $config['sess_time_to_update'] to 300 or 14400, also setting sess_expiration to 0 (never expired)

However session still expired way before sess_expiration. I have minimal ajax call and I can't seem to find the answer.