Sending a large number of emails

[eluser]mrh[/eluser]
Hello all,

First I want to say that learning CodeIgniter has been great. Since I was turned on to it I've done 3 small sites and I'm finishing up a fairly large one with a database. Very cool framework. Probably one of the best I've come across in all languages.

Anyway on to my question. For this larger site which has a database our customer has clients which we call "projects" (each has an entry in a database) and these projects are non-profit organizations. Each of these non-profits can have donors to their cause. Our site allows them to enter and maintain their donor data in a small donor database.

Our client who we did the site for wants us to provide a simple way that their projects (clients) can send an email to the donors in their donor database. So far no issue. I have a small form where the project once logged in can type in their email content and then I had planned to simply loop through their donor database and send that content to each donor they have with a valid email address.

This is not like mass email spamming. These donors have contributed money to the project cause and it is one responsibility of the project to notify the contributors of how the money is being spent. They also might like to solicit for more donations too. So while we plan to offer the projects the ability to download their database in CSV format so they can use a tool like MailChimp to send stuff out, our client did want us to provide a simple email interface.

My question is this. If a project has say 50 donors then I don't see much of a problem. I simply loop through the 50 donors and create an email and send it using the CI email stuff. I think our sites email engine will be ok with that. But what if they have 10,000 donors (excessive I'm sure but just thinking)?

Because of this possibility I'm thinking that when a project fills out the form and presses send that what should really happen internally is I could copy the email content and donors over to a "task" table. The purpose of this "task" table would be to allow a cron job to periodically work on things to do.

I'm sort of imagining that the task table would have a code field which told the PHP script run by the cron job what to do with an entry. One code would be to send an email. So the script would pull the email address, subject, and content from the task record and send the email then the task record would be deleted.

Does this sound like a reasonable way to approach this? I'm open to other ideas and like I said we will offer the ability for the projects to make use of MailChimp. We just want to give them something simple which would be used for smaller communications.

Thanks in advance

[eluser]PhilTem[/eluser]
It really sounds like a reasonable approach - I thought of pushing you towards using cronjobs as I was about to reading the 5th paragraph. But then you mentioned it yourself.

Anyway, you don't need to have a snippet of PHP script in your database. I had a similar task to accomplish in a project one year ago and ever since then I'm using the same code for other projects. Let me just tell you my experience and talk a little about my business logic:

First of all you're right, you will want to use cronjobs to send out so many emails since you might end up with exceeding the maximum execution time for a PHP script if you wanted to use either PHP's mail() or CI's email library. The problem however, you need root access to your server in order to install a cronjob that's run under the webserver's username. But that shouldn't be a problem here.

What you would put in your crontab file can be of two different kinds:
- Either put a call to the PHP CLI (command line input) and refer to the CI user's guide on how to use the CLI
- Put a "wget http://www.example.com/controller/method/args" in it
I'm usually going with the second one since I can store the output in a nicer way to a custom log file.

Now, since we will basically be making a simple GET-Request to some controller of our page, we need to create this.

How you create it doesn't matter, it should just grab the recipients from a database, grab the content of the mail from your database and mix it together to make an email. Then you can loop over e.g. 10 recipients per call to your cron-controller and you want put too much load on your system.

If you want to have a look at my controller, have a look here (Right now I'm too lazy to write more in here, but you can hopefully understand a bit from my code).
The libraries and models used for it can be found inside the same module by just browsing to the corresponding folder

I had very good experiences with this setup however it was a little bit difficult setting up the cronjob correctly and without any errors. But that was due to not knowing how to use cronjobs

For any questions just post here or contact me at philipp (at) ignitedco (dot) de

[eluser]mrh[/eluser]
Hi Philipp,

Thanks for a wonderful answer. Ok I've basically got it working. I have no idea yet on what a good "load" would be. I'm going to have to play with this with some demo data.

I used your idea of the wGet. I was having trouble on this host getting a direct call to php to accept the right arguments. The wGet works perfect every time.

So the process I decided to follow was this:

When a project wants to send an email I ended up copying the email subject, content and then looping through the donor table for the project and creating what I called a task record. So each task record holds everything needed to do one task.

My cron job calls a task processor. This looks in the task table and executes 75 tasks at a time. I'm wondering if that is too many. I guess I need try a few things and see.

Anyway your help was amazing. Thank you!

[eluser]PhilTem[/eluser]
I can't tell if 75 is too much or just fine since it will also depend on how often you run your cronjobs. If you run it ever 1 minute but need to process these 75 in two minutes you'll end up with some emails being send twice.
You could avoid this by having a 'processing' flag on all items you're processing. That means, you grab the oldest 75 items which have not yet been processed, set the processing flag to 1, then send them out, and afterwards delete these rows from the DB. With another process running one minute later you will do the same but won't be duplicating tasks unnecessarily

[eluser]mrh[/eluser]
Well I was thinking about running this about every 30 minutes at least initially. I'll need to see what the email load is so I've built in some monitoring to let me know how much is going on. Frankly I don't think my client cares if it takes a day or more to send out a bunch of emails for one of their projects. I think his view would be if you need it faster, use MailChimp.

Thanks again.