creating web crawler and search engine

creating web crawler and search engine

• bsports3

  Hi Sandeep,

  I understand your frustration - I am trying to create a small indexer for my website and a search feature for it.

  i am doing some research right now - if you would like, i would have no problem letting you know what i find out, and once my engine is complete, i will post it somewhere public(like CodePlex or something). my email is dacoty@gmail.com if you would like to correspond further.

  best of luck - David

  
  
  

• danpoolshark

  i didn't ask whether god exists or not. this is a technical forum and anyone can ask technical questions. if you donot have answers to the questions; don't even bother looking at them. you wasted my time...
  
  

• Lynn_global

  sandeep -

  i also noticed that there are a bunch of snobs who think they have the answers for everything floating around on this and many other forums. people who are sick of answering the same questions over and again and want to refer you to the FAQ or have something otherwise un-constructive to say. as sad as it is, these people DO exist here too.

  this includes the moderator (don't like your job, pal QUIT!)

  makes me wonder why the bother trying to help - looks more like they want to be top-dog and want to feel better about themselves."lookat the big brain on brad!!" LOL 

  i never know programming became a staus symbol. just keep plugging away at what you do and help others as you want them to help you. what goes around, comes around!!

   

  all the best- dave

   

  
  
  

• RavensAngel

  Sandeep
  This is such a big question that it is not suited for forums.
  

   

  
  
  

• Nigel Watling

  i am so surprised at the reply of the forum's moderator. i donot mean to offend you all... take it in advance. MS has provided us with a great forum Windows developers to seek out help from fellow programmers to seek out solutions to problems when developing with Windows and the fact that the moderators and MS personnel deter us programmers from querying answers to todays pratical programming problems is quite offending. i find myself remebering the fact that Linus Torvalds might just have started the same way as me... a bit confused at first but in a few years time creating something that changed the dynamics how the IT/ Computer industry works... but even if this forum is not suitable for any discussions of this scale please point me to other forums/newgroups where i would be able to pose questions of this scale.

  thanks for reading....

  
  
  

• dv83d

  i am so surprised at the reply of the forum's moderator. i donot mean to offedn you all... take it in advance. MS has provided us with a great forum Windows developers to seek out help with fellow programmers to seek out solutions to problems when developing with Windows and the fact that the moderators and MS personnel deter us programmers from querying answers to todays programming problems is quite offending. i find myself remebering the fact that Linus torvalds might just have started the same way as me... a bit confused at first but in a few years time creating something that changed the model how the IT/ Computer industry works... but even if this forum is not suitable for any discussions of this scale please point me to other forums/newgroups where i would be able to pose questions of this scale.

  thanks for reading....

  
  
  

• Encaps

  i have a developed a simple user directed web crawler, that searches the web using google as a seed url. anybody interested i will mail the source code to you email me at connect2sandeep@gmail.com. its just basic stuff so don't expect some fireworks.
  
  
  

• rings777

  i have found an article that describes a web search agent:

  www.acm.org/crossroads/xrds5-4/webhunting.html

  i am planning to develop a simple crawler.

  
  
  

• sobo

  This is a broad question with many aspects. As Durga states, this is a tough one to get a single answer to on a web forum but here it goes:

  You will need to be ready to do a lot of reading. I suggest:

  - Serach MSDN for articles relating to building scalable network applications (specifically servers). This will lead you to System.Net, asynchronous I/O and the underlying Winsock technologies

  - For the DB aspect you will want to look at fault tolerance, load balancing, DB schema design, normalization and query analysis documents

  - For the UI aspect, see www.asp.net

  - For crawling you should look at httpwebrequest and webclient classes. You will also want to reserach both the HTTP and HTML protocols to determine how to interpret the content returned.

  
  
  
• creating web crawler and search engine