Welcome Guest, Not a member yet? Register   Sign In
Improving performance of csv_from_result method
#1

[eluser]qwertyqwerty[/eluser]
I was having a tough time of it today, wondering why on earth it was taking over one minute to convert 20,000 records to a csv string using the $this->dbutil->csv_from_result() method. Whilst 20,000 rows is a fair amount, it shouldn't take that long and after trying to export the query from phpMyAdmin, it only took a second or two, I knew something had to be up.

I'm no PHP expert but it seems that the problematic code is the following:

Code:
foreach ($row as $item)
{
    $out .= $enclosure.str_replace($enclosure, $enclosure.$enclosure, $item).$enclosure.$delim;            
}
$out = rtrim($out);
$out .= $newline;

I don't see why you need to all these string operations when this can be achieved in one line using the implode function.

Code:
$out .= '"' . implode('","', $row) . '"' . $newline;

Using implode, lets look at the difference

Code:
// My version
function csv_from_result($query, $delim = ",", $newline = "\n", $enclosure = '"')
{
    if ( ! is_object($query) OR ! method_exists($query, 'field_names'))
    {
        show_error('You must submit a valid result object');
    }    

    $out = '';
    
    // First generate the headings from the table column names
    $out = '"' . implode('","', $query->list_fields()) . '"' . $newline;

    // Next blast through the result array and build out the rows
    foreach ($query->result_array() as $row)
    {
        $out .= '"' . implode('","', $row) . '"' . $newline;
    }
    return $out;
}

// Original version
function csv_from_result($query, $delim = ",", $newline = "\n", $enclosure = '"')
{
    if ( ! is_object($query) OR ! method_exists($query, 'field_names'))
    {
        show_error('You must submit a valid result object');
    }    

    $out = '';
    
    // First generate the headings from the table column names
    foreach ($query->list_fields() as $name)
    {
        $out .= $enclosure.str_replace($enclosure, $enclosure.$enclosure, $name).$enclosure.$delim;
    }
    
    $out = rtrim($out);
    $out .= $newline;
    
    // Next blast through the result array and build out the rows
    foreach ($query->result_array() as $row)
    {
        foreach ($row as $item)
        {
            $out .= $enclosure.str_replace($enclosure, $enclosure.$enclosure, $item).$enclosure.$delim;            
        }
        $out = rtrim($out);
        $out .= $newline;
    }

    return $out;
}

To say this code is faster is an understatement as I've shaved off roughly 50 seconds of processing time.

Hope this helps someone and please bear in mind I'm not a PHP expert and I may have just misunderstood why the original code was done the way it was.

Cheers,
Ryan


Messages In This Thread
Improving performance of csv_from_result method - by El Forum - 07-31-2008, 08:17 PM
Improving performance of csv_from_result method - by El Forum - 09-10-2008, 04:36 AM
Improving performance of csv_from_result method - by El Forum - 09-10-2008, 06:10 AM
Improving performance of csv_from_result method - by El Forum - 09-10-2008, 04:39 PM
Improving performance of csv_from_result method - by El Forum - 09-11-2008, 09:12 AM
Improving performance of csv_from_result method - by El Forum - 09-11-2008, 09:15 AM
Improving performance of csv_from_result method - by El Forum - 09-11-2008, 10:29 AM
Improving performance of csv_from_result method - by El Forum - 09-11-2008, 03:26 PM
Improving performance of csv_from_result method - by El Forum - 02-07-2009, 05:29 AM



Theme © iAndrew 2016 - Forum software by © MyBB