[eluser]Aea[/eluser]
I'm currently working on an application which is responsible for making a lot of API calls at varying frequencies (mostly once every hour), several calls per user, and I'm designing this to accommodate roughly a hundred users. Each call will have a lot of processing work going behind it, and I believe that if these were to run on top of each other they would cause problems. My initial plan was to have a "static" queue system, where each user would have his data processed sequentially, one user at a time. This could work as a solution for my problem, and it is probably the simplest system to implement. I would start a cron that runs as often as necessary, and staggers each API call so that only one runs at once.
At the same time however, I want my users to be doing complex calculations (think traversing & joining on the order of 10K, to 100K rows). I don't want my users waiting for these to be completed, nor do I want them to tie up the rest of the system.
My question is has anybody developed a dynamic queue system? Where requests can be submitted at any time and arranged in terms of priority (or maybe even a more elaborate system which assigns weights based on priority and time spent in queue). For this I was considering potentially using the following model...
-> Job Submitted, either by user or by a cron
-> Job request added to database, and queue script is run.
-> Queue script checks if it's running already (I don't know how to implement something like this reliably), and if not activates itself and starts running jobs in the queue
So, how would I go about making sure that the queue processor is only running once using just general PHP? I considered "check outs" through MySQL, but if PHP or MySQL throws a wobbly it would remain "checked out," maybe transactions could somehow be implemented, although I see no way of doing it.
Any suggestions would be greatly appreciated, I realize this is quite the advanced topic, but in the interest of performance and greater development as a developer this is something I very much would like to pursue.
