Welcome Guest, Not a member yet? Register   Sign In
CI is a Memory Hog by design
#1

[eluser]Robert Wallis[/eluser]
I've been having problems with CI and EE for a few years. The problem is that any dataset larger than about 2000 records will cause the site to crash like this:

Fatal error: Allowed memory size of 26214400 bytes exhausted (tried to allocate 16 bytes) in /home/example/domains/example.com/public_html/system/database/drivers/mysql/mysql_result.php on line 162

The solution for us is two part:

1) Write a new Database handler in EE, or just the DB_result class in CI to use PHP Iterators instead of pulling every single row into a massive array.
http://codeigniter.com/bug_tracker/bug/7389/

2) Send each result directly to the browser via echo() or print().

Most commonly, I want to keep using a view, but I can't because Views are designed to return into a string. Then that string is sent through a str_replace and a preg_match on Loader.php(673) (which is optional). Then a str_replace on Output.php(273) and then str_replace on Output.php(276) before finally being sent to the browser on 351. Those 4 functions make copies of the string output, and don't free the result, which wastes tons of memory making the app hit the 26mb limit. The total HTML output of my page when bypassing Loader is about 2mb. That's how bad this problem is.

So, I'd like a way to let a view be processed on the fly, instead of stuck in the Output class by Loader.

If someone wants to use the {memory_usage} and {elapsed_time} they can. But I'll pass, to keep my memory usage down.

I also I had to write my own DB_utility::csv_from_result() function. Although mine, besides not saving everything to a massive string, also sets the HTTP Content-Disposition and Content-Type to force a download instead of viewing in the browser which automatically gets picked up by Excel.

What I'd like to see from Ellis Labs is: more importance put on memory management, starting with bug 7389 which is not considered high priority. That would make me pretty happy, and improve the overall quality of the software.
#2

[eluser]tomcode[/eluser]
Aren't PHP Iterators PHP 5 only ? So this would break a CI essential (ok for me, though).
#3

[eluser]Robert Wallis[/eluser]
The PHP docs don't have a version for the Iterator class. So I'm not sure when it was introduced.
#4

[eluser]tomcode[/eluser]
I had been looking into the methods of the Iterator Interface Iterator::current, there the manual states (PHP 5 >= 5.1.0).
#5

[eluser]Robert Wallis[/eluser]
Bummer, now I know they won't implement it Smile I don't think there's a PHP 4 workaround other than the C way of checking the PHP version and loading a different result class based on the version. But it would make the site inconsistent.

Sad
#6

[eluser]Robert Wallis[/eluser]
Add this to the DB_result class, at the end of the class (but inside it) is fine if you want to use functionality like this:

Code:
$result = $this->db->query("SELECT * FROM bigtable");
foreach($result as $row) {
    // ...
}

Here's the code to insert into DB_result:
Code:
/**
     * Iterator implemented functions
     * http://us2.php.net/manual/en/class.iterator.php
     */
    
    /**
     * Rewind the database back to the first record
     *
     * @return void
     * @author Robert Wallis
     */
    function rewind()
    {
        if ($this->result_id !== FALSE AND $this->num_rows() != 0) {
            $this->_data_seek(0);
            $this->valid = TRUE;
        }
    }
    
    /**
     * Return the current row record.
     *
     * @return object row_data
     * @author Robert Wallis
     */
    function current()
    {
        if ($this->current_row === -1)
            $this->next();

        return $this->row_data;
    }
    
    /**
     * The current row number from the result
     *
     * @return int current_row
     * @author Robert Wallis
     */
    function key()
    {
        return $this->current_row;
    }
    
    /**
     * Go to the next result.
     *
     * @return bool Is there a next result.
     * @author Robert Wallis
     */
    function next()
    {
        $this->row_data = $this->_fetch_object();
        if ($this->row_data) {
            $this->current_row++;
            if (!$this->valid)
                $this->valid = TRUE;
            return true;
        } else {
            $this->valid = FALSE;
            return FALSE;
        }
    }
    
    /**
     * Is the current_row really a record?
     *
     * @return bool
     * @author Robert Wallis
     */
    function valid()
    {
        return $this->valid;
    }




Theme © iAndrew 2016 - Forum software by © MyBB