CI is a Memory Hog by design

[eluser]Robert Wallis[/eluser]
I've been having problems with CI and EE for a few years. The problem is that any dataset larger than about 2000 records will cause the site to crash like this:

Fatal error: Allowed memory size of 26214400 bytes exhausted (tried to allocate 16 bytes) in /home/example/domains/example.com/public_html/system/database/drivers/mysql/mysql_result.php on line 162

The solution for us is two part:

1) Write a new Database handler in EE, or just the DB_result class in CI to use PHP Iterators instead of pulling every single row into a massive array.
http://codeigniter.com/bug_tracker/bug/7389/

2) Send each result directly to the browser via echo() or print().

Most commonly, I want to keep using a view, but I can't because Views are designed to return into a string. Then that string is sent through a str_replace and a preg_match on Loader.php(673) (which is optional). Then a str_replace on Output.php(273) and then str_replace on Output.php(276) before finally being sent to the browser on 351. Those 4 functions make copies of the string output, and don't free the result, which wastes tons of memory making the app hit the 26mb limit. The total HTML output of my page when bypassing Loader is about 2mb. That's how bad this problem is.

So, I'd like a way to let a view be processed on the fly, instead of stuck in the Output class by Loader.

If someone wants to use the {memory_usage} and {elapsed_time} they can. But I'll pass, to keep my memory usage down.

I also I had to write my own DB_utility::csv_from_result() function. Although mine, besides not saving everything to a massive string, also sets the HTTP Content-Disposition and Content-Type to force a download instead of viewing in the browser which automatically gets picked up by Excel.

What I'd like to see from Ellis Labs is: more importance put on memory management, starting with bug 7389 which is not considered high priority. That would make me pretty happy, and improve the overall quality of the software.