A queue system?

[eluser]brainer[/eluser]
This isn't necessarily a CodeIgniter question, but:

Basically I'm looking for a queue where I can add user_ids to, then processes this queue via cron.

I thought Amazon SQS was a potential solution, but it seems really slow + only allows 10 items to be fetched from a queue at a time. So it doesn't really lend itself well to working with large amounts of data.

I was considering creating my own queue system based on MongoDB, but I'm wonding first if anybody knows any other already available solutions? Maybe one with a nice CodeIgniter implementation?

[eluser]jedd[/eluser]
Hi brainer.

I haven't used this directly myself, but the developers I'm working with at the moment just settled on beanstalkd / pheanstalk as a lightweight queue.

We did look at a few others - rabbitmq being the obvious one, as well as Amazon (we have a lot of AWS services already) but the former was just too heavy and the latter was just too expensive for the quantity we were looking at. You have to shard beanstalk, as I understand it, at least to an extent - a bit like how you'd use a memcache cluster I think.

Certainly worth a look, anyway. Don't believe there's a CI library ready to go, but it should be pretty easy to write your own library for it I'd think.

[eluser]brainer[/eluser]
hi,

I actually went ahead and developed a CI library that uses MongoDB to store jobs. It's very simple and the usage is actually very similar to pheanstalk. After a few more tweaks I think I'll share it with the community.

I'm going to look at beanstalkd a little more though, maybe it's a better solution.