When you say API are you using CI's session library methods to get/set data or did you build a layer on top of that? You could sticky file sessions by setting it on your load balancer until you can figure out the root cause.
For us (using Redis not Memcache) requests went from 1-2s down to the normal 100-200ms adding session_write_close.