Web Spider or Search Functionality

[eluser]Sebish[/eluser]
Hi Everyone,

I had a look on Google, and searched the CI Forums and couldn't find anything that would help me give a helping hand. Basically I know that it would be quite simple to create search functionality that could search my database and feed back links, etc.

However the problem that I am having is that in the current project I am working on, half my content is located in the database and the other half is located in static PHP files on the server. The files that I really want to search are the PHP files.

I have been looking around at a few different websites such as Google Custom Search, Atomz, Digital Point, etc to come up with a simple solution. The problem with all of these is all are covered in ads and also redirect the user from my website (instant Fail).

I was wondering how easy or difficult it would be to create a module; controller or something of the type that would act as my own personal spider so then I could feed the information back to my website’s database and use it as a search. I have been reading tutorials on “creating your own web spider” however I have only found information on ASP and Python both of which I have no interest in using or learning at this point in time.

If anyone has come across this problem before and have discovered a solution or know an online solution which I can build my own templates for and have it hosted on my own servers. The information would be priceless; I have spent some time investigating this and have little-to-no luck.

I would prefer to build something in Code Igniter so I can add its settings and statistics into my administration area.

Thanks for taking the time to read this post; I hope it all makes sense and if you have any further questions please reply and I will get back to you as soon as possible!

Cheers.

[eluser]darrenm[/eluser]
Sphider and isearch are both good search engines. They work by spidering your site and then performing searches on the resulting index.

This would cover off your problem of returning results form both static and db content.

As far as integrating them into CodeIgniter, that's more of a challenge! You'd have to port the main search files into a CI format. Do-able, but time consuming. You'll probably find it quicker to include your CI views within the existing search script.