Using CI for AJAX (how about performance?)

"Will CI itself be a huge impedance for frequent AJAX requests?"

No

"It's way more convenient to implement it all in CI (because i can reuse models, and helpers and query builder is quite nice.)"

That's why you use a framework

Simpler is always better