[eluser]Rob Steele[/eluser]
alright, well in my CS class we had to index words with a spider. Basically in strictly theoretical dumbed down version of it, when new documents are added to the db, that document is tokenized into words. Afterwards a pointer to that document is put into an array that is indexed to that word(We used a hash map for speed). To cut down on size, there are certain words you ignore that are tooo common place, ie ('the', 'an') all the articles in grammar. But the easiest way is to just find a 3rd party lib, unless you really want to learn how to do it the nitty gritty way.
Also, someone please tell me if i'm wrong but, to make that index would be a many-to-many pattern
ie
a words table
word_id
word
a document table
document_id
document_location
and a word_doc table
word_id
document_id
that way you can check to see if a word is already associated with a document and it saves space sense you don't have to worry about writing the document path hundreds of times.
-Rob
