Web Services, Queues, polling them... best practices?

[eluser]andjules[/eluser]
Hi, looking for the opinion of more experienced developers (than me)...

I'm building an app that uses web services (for uploading, transcoding, image manipulation, etc.)... Most of the critical calls to the web services either i) are processed immediately, or ii) file a callback to me when they're done processing.

HOWEVER, there are a few instances where processing might take a while, and the service API doesn't fire a callback on completion, so:

It seems if something isn't ready when I first request it, I need to set up a queue, then poll the queue and re-check, until the service is done processing the original request, at which point I delete from the queue.

a) I'm aware of Amazon SQS, I'm aware of Apache MQ (which I keep hearing is a pain), are there any other local PHP practices for creating a queue (memcached?)?
b) what's the best practice for polling the queue? Cron? (that doesn't seem very elegant...)

Thanks in advance

[eluser]Elliot Haughin[/eluser]
The simple way to process a queue of something, is to write a PHP script that handles one item.
So, it just pulls one item from the DB and processes it, then sets the status to 'done'.
And then add it to a cron.

You can use wget for your cron if you're running your script inside CI.

Like this:

Code:

*/5 * * * * /usr/sbin/wget -O /dev/null http://www.mysite.com/process/queue/ > /dev/null

I use something like this with FilePanda to push files to amazon S3 and it seems to work great

However, I will be moving it over to Amazon's Simple Queue System soon.

Elliot

[eluser]andjules[/eluser]
Thanks so much for the insight, Elliot.

By the way, as you may be aware, S3 has recently opened up direct (browser-based) uploads. However, when I was looking into it, there seemed to be a lot of issues with responses, mime validation, etc.

So, for the site I am working on for a client, I've gone with S3-competitor Nirvanix. It allows me to use swfUpload (and get a nice progress bar! my client will get large uploads from its customer base) and upload directly, get a callback, etc. Anyhow, they are an interesting alternative to (the excellent) s3 service.

[eluser]Elliot Haughin[/eluser]
Awesome!

Just for the record... I've just finished putting and upload progress bar on FilePanda.
Check it out (you'll need a reasonable sized file to see it working).

It's working just using PHP, APC and AJAX.

I'd always want to upload to my server first so I can do some work on the file.

Elliot 'hates flash' Haughin

[eluser]Wuushu[/eluser]
[quote author="Elliot Haughin" date="1208749600"]Awesome!

Just for the record... I've just finished putting and upload progress bar on FilePanda.
Check it out (you'll need a reasonable sized file to see it working).

It's working just using PHP, APC and AJAX.

I'd always want to upload to my server first so I can do some work on the file.

Elliot 'hates flash' Haughin[/quote]

Just gave your upload script a swirl - absolutely love the simplicity. Would you care sharing some light with the few rest of us how you pulled the server progress off?

(/upload/progress/FILEID)

It seems to return a speed, percent etc? How?

Thanks and great work!