Welcome Guest, Not a member yet? Register   Sign In
PHP and Queue Systems?
#1

[eluser]Aea[/eluser]
I'm currently working on an application which is responsible for making a lot of API calls at varying frequencies (mostly once every hour), several calls per user, and I'm designing this to accommodate roughly a hundred users. Each call will have a lot of processing work going behind it, and I believe that if these were to run on top of each other they would cause problems. My initial plan was to have a "static" queue system, where each user would have his data processed sequentially, one user at a time. This could work as a solution for my problem, and it is probably the simplest system to implement. I would start a cron that runs as often as necessary, and staggers each API call so that only one runs at once.

At the same time however, I want my users to be doing complex calculations (think traversing & joining on the order of 10K, to 100K rows). I don't want my users waiting for these to be completed, nor do I want them to tie up the rest of the system.

My question is has anybody developed a dynamic queue system? Where requests can be submitted at any time and arranged in terms of priority (or maybe even a more elaborate system which assigns weights based on priority and time spent in queue). For this I was considering potentially using the following model...

-> Job Submitted, either by user or by a cron
-> Job request added to database, and queue script is run.
-> Queue script checks if it's running already (I don't know how to implement something like this reliably), and if not activates itself and starts running jobs in the queue

So, how would I go about making sure that the queue processor is only running once using just general PHP? I considered "check outs" through MySQL, but if PHP or MySQL throws a wobbly it would remain "checked out," maybe transactions could somehow be implemented, although I see no way of doing it.

Any suggestions would be greatly appreciated, I realize this is quite the advanced topic, but in the interest of performance and greater development as a developer this is something I very much would like to pursue.
#2

[eluser]Pascal Kriete[/eluser]
I certainly haven't seen such a system in this community, although it would probably be a real pleasure to dissect. The first commercial solution that comes to mind is Zend Platform.

The best approach would probably be to start quite small and make sure that the foundation is rock solid. It will be all the little things that make or break such a system. For example, one thing that is often overlooked in regular old applications is database optimization. Considering the amount of queries you're proposing, you definitely want to set priorities for your queries so as not to bring down the whole application with your backend scripts. [url="http://dev.mysql.com/doc/refman/5.0/en/optimization.html"]This[/url] should provide for some interesting reading.

I believe you're on to something that could benefit everyone here, so please continue to share.
#3

[eluser]Aea[/eluser]
Inparo, I had no idea that the idea would sound appealing to others, half expected to be told to optimize, optimize, and optimize instead of seeking such a load management solution.

On the pure topic of SQL queries, unfortunately in terms of row count optimizations aren't entirely possible, while such queries will be highly uncommon, a user may want to traverse data back a considerable timespan, and apply calculations to the results. Of course, it would be best to have the system automatically run these queries for the users that require them doing off-peak times. So, while I would usually shy away from doing something which seems to be more complex then necessary for 99% of applications out there, this would be highly beneficial to what I am developing.

I'll continue to think about a mechanism for a queue system, will make another post if I have any revelation about ensuring a script "runs once" Smile

Random Idea : Potentially an endless loop which checks the queue database every x seconds, does its job in that way, and it will be executed on PHP server start up? Of course, there'll need to be isolation so that if something that the queue executes results in an error the queue file doesn't go down with it. A bit of an inelegant solution, would prefer to find a way to know if a script is currently running.
#4

[eluser]ChrisMiller[/eluser]
[quote author="Aea" date="1201600393"]
Random Idea : Potentially an endless loop which checks the queue database every x seconds, does its job in that way, and it will be executed on PHP server start up? Of course, there'll need to be isolation so that if something that the queue executes results in an error the queue file doesn't go down with it. A bit of an inelegant solution, would prefer to find a way to know if a script is currently running.[/quote]

Same thing I was thinking about, write a controller and have it setup so it rechecks the queue and have the application go to sleep (http://us2.php.net/sleep) for 180 secs (3mins) or so and then after the sleep have it reload itself thus making a neverending loop. Now I would also add a function so you can throw a manual stop at it so you dont end up with a loop system that is hard to stop and neverending execution. Here is a quick example of a repeating class... I know its not the best but it gets the idea out...
Code:
<?php
class Queue {
    
    var $sleep = 180;
    
    function Queue()
    {

    }
    
    function process()
    {
        //-------------------------------------
        // Check for manual end here
        //-------------------------------------
        
        
        
        //-------------------------------------
        // Your Code (remember to trap errors)
        //-------------------------------------
        // Do your php here code...
        
        
        //-------------------------------------
        // Go to sleep for a bit...
        //-------------------------------------
        @sleep( $this->sleep );
        
        
        //-------------------------------------
        // Lets reload ourself to make a loop
        //-------------------------------------
        $this->process();
    }
    
}
?>

Now with that where you store the pending list you should have it setup with a priority list like said above thus important changes are pushed through first and non-important ones are just added to the end of the list but not forgotten. Now you say you would like to process more requests on off times then if you have php 5 then you might want to be interested in running a second looping script but instead have it run only during say 2am -> 5am then shutdown till 2am the next day using the following idea:
Code:
<?php
class Queue {
    
    var $sleep = 180;
    
    function Queue()
    {

    }
    
    function process()
    {
        //-------------------------------------
        // Check for manual end here
        //-------------------------------------
        
        
        
        //-------------------------------------
        // Your Code (remember to trap errors)
        //-------------------------------------
        // Do your php here code...
        
        
        //-------------------------------------
        // Go to sleep for a bit...
        //-------------------------------------
        if( time() >= mktime(4,0,1) )
        {
            // Before 4am keep running...
            @sleep( $this->sleep );
        }
        else
        {
            // Go to bed till 2am tommorow
            @time_sleep_until( (time() + 86380) );
        }
        
        //-------------------------------------
        // Lets reload ourself to make a loop
        //-------------------------------------
        $this->process();
    }
    
}
?>
Once Again this is just to give you some ideas, I am not saying this is safe to run on your server, you should do some reading up on how to secure it much better and also how much memory/processer this sucker will chew up, I am not gonna do all the work for ya now ;-P
#5

[eluser]adamp1[/eluser]
Hate to say this but, PHP is not the best thing for a task like this. If your wanting to handle huge amounts of data and do it in real time may I suggest maybe writing this in a language like C++/Ada/C#. It depends how good the performance you want out of the system. Anything written and compiled will be magnitudes faster than PHP. Remember PHP is not for calculations its for displaying web pages. I haven't done anything in C# or C++ to do with real time programming but I am in the process of using Ada for a real-time systems.

Really it depends how much your willing to work on it and how fast you want it. I know there are MySQl implementation packages for Ada out there. Also it depends on if you can run the actual files on your server (probably not if its shared).




Theme © iAndrew 2016 - Forum software by © MyBB