sending email to 15,000 recipients

[eluser]ldg430[/eluser]
I have a list of 15,000 users that I need to send a html-format email with dynamic content.

I am using a similar approach as Derek's in this thread

In my case, I do it for every 100 users at a time - get 100 users, assemble the data fields, load the view and send email. Then get the next 100, and so on...

My dilemma is while the approach works, the memory usage increases as the emails are processed. I am using the command line interface, CLI, to run the script and then watch the process using top. At some point, the process uses up the max memory for the script and then dies. I have the memory limit set at 256mb and the process dies around 11,500.

So I thought that it had to do with loading the view, since the Loader includes the view file every time, so in my code I replaced

Code:

$html = $this->load->view(view_file, $data, true);

with

Code:

extract($data);
ob_start();
echo eval('?>'.file_get_contents($view).'<?');
$html = ob_get_contents();
@ob_end_clean();

...send email

unset($html);

hoping that I release the memory used up in generating the html.

While this works, the memory usage was still increasing as the emails are sent.

I know I can increase the memory limit but I want to know if there is a way to prevent the process from eating up too much memory.

Appreciate any help...

[eluser]Unknown[/eluser]
I think your problem is the database query, I had a similar problem dealing with a large data set from the command line. I also had a memory limit of 256mb and it tanked about 11,500. :coolsmile:

CodeIgniter stores all the queries it executes in an array (I think), this is why the memory usage keeps increasing. You can use this:

Code:

$this->database->save_queries = false;

Also check out this _fetch_object discussion: http://ellislab.com/forums/viewthread/129194/

These two changes fixed my problems, good luck!

[eluser]ldg430[/eluser]
Thanks for the tip!

Will try the solution shortly....

[eluser]ldg430[/eluser]
tons of fun,

Setting save_queries to false did the trick for me! I was saving $data and $html in the db after sending the email. The sql associated with this update was 20-30k per email - no wonder the available memory was being eaten up!

I did not have to use _fetch_object as I was only doing 100 records at a time - so my record set was relatively small. But it's good information!

Thanks again for your response!