How does a search engine find a dynamic URL address that's based on values in a DB?

[eluser]Dandy_andy[/eluser]
I've been using codeigniter for a couple of weeks now and I can see the benefits of the framework. However, I'm a little bit confused as to how search engines find and crawl dynamic URL addresses if they don't actually exist as files?

My current networking site uses a system whereby it creates an index.php file for each member and stores it in a folder. I can understand how search engines crawl that, but I am redesigning my site so that I don't have to have thousands of index.php files for each member. This is all great news, but how does Google for example find each members profile page if the page is dynamically generated?

[eluser]boltsabre[/eluser]
Search engines get to pages by links! If you have a link to a page it can get crawled. I presume you are pulling user data from a DB and using that to generate unique links such as mysite.com/users/profile/1234...?

If so, it doesn't matter if you use "template" view to populate each and every users data, so long as the urls are unique, pointing to unique/dynamic content, you'll be fine.

Or did I misunderstand your question?

[eluser]Dandy_andy[/eluser]
No you understood correctly and I thought the answer would be something like that. I haven't built my dynamic structure yet - I'm just playing around with it and I was just curious to know. So does the dynamic system need to address the fact that search engines will search for the information? Or do I just assume that as long as the data exists and a particular dynamic page extracts and processes information from my DB, a search engine will also find it? Do I work on the basis that if I can type it into the URL address bar and it produces a page, google will find ti? I'm quite amazed at how this all works and I'll try and explain my thinking...

Suppose I have dynamically created pages called

www.domain.com/profile/member1
www.domain.com/profile/member5
www.domain.com/profile/member6
www.domain.com/profile/member7
www.domain.com/profile/member8 etc.

How does google know that member2 doesn't exist for example? Or how does it know that something completely different like abcdefg123 doesn't exist? Does it actually search for every possible dynamic variation or does something tell google which dynamic addresses are possible?

[eluser]CroNiX[/eluser]
As boltsabre correctly stated, search engines only find your content by following LINKS that are on your site. They don't go making things up to see if they exist, like your member2 example. If you want the profiles crawled and cataloged by search engines then you need to have links to them that the public can access, such as on a membership list page or something.

[eluser]boltsabre[/eluser]
To the best of my knowledge there are only two ways that SE bots crawl cyber space is:

1. Via a html link - if page A does not have a link anywhere in cyberspace to it then they cannot find it.
2. Via a site xml sitemap. (you can create a xml document and tell the bots about every page you have on your site. However, this is just a "guide" to them, it's best to have both point 1 and 2 working in tandem to get maximum exposure of your links).

It does get a little more complicated than this(for example, when you register a new domain, google MAY get hold of this information and go have a look at your top level/index page even though there may not yet be a single link to this new domain , but this is it in it's most basic and pure form, no link and no sitemap, don't expect any bots along!

So, somewhere on your site you want links to the pages you want google to index/crawl. How you go about this, well it's your site, you know best!

If you're interested in SEO, this is the google forum (powered by volunteers, not official google employees, much the same as this forum):
http://productforums.google.com/forum/#!...g--ranking

Let me know if you need it explained in any more detail.

[eluser]Dandy_andy[/eluser]
I'm a little bit puzzled now with the whole dynamic website thing. Lets say all the members pages could be accessed by type www.domain.com/members/$id where the $id is the members ID number. These pages do not exist statically but if you were to type in a number such as 45 for example and that member existed, if would run the function and generate a publically accessible page (based on the website settings). So how would google know that member 45 existed? $id could be any number or string?

[eluser]CroNiX[/eluser]
[quote author="Dandy_andy" date="1337714131"]So how would google know that member 45 existed? $id could be any number or string?[/quote]It wouldn't, unless there was actually a link like www.domain.com/members/45 on your site that the public has access to.

Search engines and visitors have no knowledge of whether your site is dynamic or not. They only go by the links and the actual content on those pages.

[eluser]Dandy_andy[/eluser]
That particular page, www.domain.com/members/45 would exist if you were to type it in to the browser, but doesn't exist as a static page that can be crawled. So based on this, if I would like to have all the member details accessible by all the search engine robots (for example each page is unique in some way), then I would need to have a link to it or generate a static page of some sort? I presume this is why my current system (I didn't design the website by the way) creates an index.php file in a publically accessible folder so that when accessed, it runs the relevant function and generates the page. Are there any other ways to do this?

[eluser]CroNiX[/eluser]
As I said in my first post above, if you want the membership links crawled and the content indexed, you need to create a page with the links, such as a membership roster page that would list all members with links to their profiles.

[eluser]Dandy_andy[/eluser]
Got you. Thanks. I may have to re-think this a bit. Turns out my current website (after checking site maps etc.) doesn't have any links to members pages even though there is a file created for each one. Only the main links of the website have been included in the site map and that is generating enough traffic for me anyway.