Welcome Guest, Not a member yet? Register   Sign In
How multithread my request to a slow API URL in CI?
#1

[eluser]Unknown[/eluser]
I've looked around a little for multithreading, and even found a couple examples ... but I'm a little unsure about to apply them to my need.

I'm building an application (my first time using CI) which works with some of our product data. At one point in the work flow, we need to hit a specific web service API to get some pretty intensive market condition data.

That API cards through a truck load of disparate data, and returns an XML up to four or five hundred kb. It is pretty darn slow ... some responses deem to take as long as 10-20 seconds.
Naturally, that bottleneck is a major concern when I have a SEVERAL dozen request to make at a time.

The author of the API invited me to hammer away at it like crazy with multiple simultaneous requests. (He kinda wants to find it's threshold for pain. I'll then throttle down to just below that point.)

I'd like to stay out of JS for this, and to do it in my controllers.

Now, when I get the XML results back, I have no need for sending them to a view.
All I need is to parse them and save some info to the DB.

Anyone have any good suggestions?
Something I'm missing in the CI manual?
Links to tutorials or some (well documented) examples?
Other??

Any and every suggestion will be well appreciated Smile
Thanks-a-bunch,
~ Mo
#2

[eluser]srpurdy[/eluser]
First thing I would try is to setup benchmarking points in your code to confirm they're isn't some other bottle neck in the code itself that may be the cause. Just a suggestion in-case you haven't tried that already. Smile

Another thing to consider is not so much codeigniter or your code for that matter, but apache. This can just as easily be a server problem. If apache is being thrashed heavily this can really slow down the processing of your requests.
#3

[eluser]Unknown[/eluser]
Thanks.
My code has been confirmed as <b><i>not</i></b> contributing to the slowdown.

The problem I'm trying to address right now is <b><i>not</i></b> the slow response itself, but rather the best way to submit simultaneous requests to his webservice API <i>and</i> be able to capture the results in a way that allows me to process what gets returned.

The approach that I <b><i>expect</i></b> to take is to loop through the list of products, start a thread on each loop by submitting a request to his API, and move on to the next iteration of the loop to start yet another thread.

My goal is to submit as many request as quickly as I can, process all the responses I receive and separate them all into two list; a list of items which returned results, and a list of items which did not.
From there I'll process what results I did get, and submit the others to a different asset so they can be researched.

How can this be done?

NOTE: I initially thought I'd need to track each response to it's appropriate request, but I'm beginning to realize that, since the response might have the search term in it, I <b><i>MIGHT</i></b> be able to just catch and process the responses without knowing/caring which request it's for ... but I'll have to verify that before I can say definitively.




Theme © iAndrew 2016 - Forum software by © MyBB