Question...

metguitarist · 15th June 2003, 12:24 PM

What steps would I have to take to make a search engine, I have a huge team assembled (coders, graphic artists, web designers, etc.) and we are trying to assemble a search engine that will search our own index. What steps will we need to take?

**Mr. Tech** · 16th June 2003, 02:33 PM

Well first I suggest you write out a basic plan of what this script needs and needs to do.

Then, what i normally do is get the basic code structure written and then you keep adding things as you go.

Then, you can get the design made etc. and fill it all in.

Hope that helps! :wink:

PY 222 · 19th July 2003, 07:57 AM

Are you going to search the entire Net or just within your domain?

metguitarist · 19th July 2003, 03:26 PM

entire net

PY 222 · 29th July 2003, 12:09 PM

Quote:

Originally Posted by metguitarist

entire net

That is quite a massive undertaking.

As far as I know, no single search engine has done that yet. Not even the mighty Google.

The Net is big and huge. Good luck in trying to store URLs for all the sites.

What kind of hardware are you going to be running your searches on? And how much hardware are you going to provide?

I am very curious because I wrote a technical paper on Google a while back and it was a massive undertaking even for Google. So I would like to see what kind of implementation that you might have.

metguitarist · 29th July 2003, 01:44 PM

I dont mean the entire net, I just mean a spider that would follow links

PY 222 · 29th July 2003, 04:43 PM

And then are you going to store the links?

f-w-r · 30th July 2003, 01:14 AM

Quote:

Originally Posted by PY 222

And then are you going to store the links?

haha what would you do with them if you didn't store them? If you didn't store them(or atleast if noone at all stored them) your spidering would be a random waste of time.

This is a rather funny topic, because I never pay for my hosting I always go into popular forums and talk webhosts into giving it to me for a link to their site, which benefits them and me more than if I just paid for their hosting...

So anyway I got a free search engine script from hotscripts.com that spidered sites. I finished setting things up and went to bed....

HOLY BANDWIDTH BATMAN! :lol:

The next day I had no more hosting =( used all bandwidth and probably almost all my storage space. I'm not sure how much my quota was, all I know was it was enough to get my host too angry at me to ever respond to another message from me ever again lol

metguitarist · 30th July 2003, 03:59 AM

haha... We'd probably need an unlimited in space and bandwidth host...

f-w-r · 1st August 2003, 09:13 PM

you could probably market a well written search engine script if you failed to be able to get hosting though. Well written search engine script would take INSANE amount of time though if it included factors popular search engines include... In testing of the script you could find a free unlim bandwidth/space hosting provider... they're all over the place some of them are slow, actually most of them are slow lol....but, if you find one that forces ads into webpages you may get luckier with speed then if it supports whatever language the search script would be then you'd be all set for this idea....

free hosts that force ads can be easilly avoided if the website pages are on another site and all you want from the site is data. So you could just use a professional host to pull the search engine's spider data out of your free server and display it at no extra bandwidth/space costs. Might not be a good idea for permanant use, beecause free servers die constantly, and more than likely beecause if they trtuly dont have a quota on your bandwidth and space that they will die quicker lol... but it would be an idea for testing how much bandwidth it would take....

to get an idea of bandwidth though, just factor 100's of the fastest internet connections possible, connected at multiple locations to the internet all going back to google... It has been over a month since my site has been re-indexed by google and google's spider cycle doesn't end.... once they're done spidering sites data gets indexed and the spiders start the next giant spider cycle. So if you factor in just a month of spidering and saving data with just one insanely fast connection(which i'm sure wouldn't even cut it) that would give you a slight idea of how insane it would be....