Olaf: You’re certainly correct in that the developer can make some effort to help resolve bad links from search engines and my statement that “there really isn’t anything I can do about it”? is technically not accurate. I have made attempts to remove URLs from search engines, but in my opinion, working through the search engine’s web site, looking for a link to add/remove URLs, then actually going through the process takes too much time and effort. Not to mention that the list of search engines I check against already numbers 150.
Stefan: Personally, I would never automatically re-direct a user from a 404 page back to the home page (I’m assuming that you were referring to doing this automatically). If the user doesn’t realize that they’ve been redirected, they’ll think the index page has the information that he’s looking for, and this probably won’t be the case. However, the message provided to the user could easily contain a link to the site’s home page in addition to the search engine and site map that I showed in my article.
Peter: I can appreciate your concerns. I have had discussions with other developers who suggest that this will generate an inordinate about of e-mail, however, from my experience on a relatively large site, this has not been the case. If a spider follows a link to a page that doesn’t exist, then the e-mail message from that will allow me to correct the link and the 404 goes away. If the error is the result of a missing file that should be available for download, either because it’s moved, or was never uploaded, this becomes very helpful in identifying and resolving those issues. However, if someone, or something, is just hitting the site looking for pages that don’t exist, no e-mail is even generated.
Clerkendweller: Excellent comments. The original intent was to provide immediate feedback to the user, then I thought “well, why not inform the developer of the problem”? and that led to what I have. The idea of logging these errors has been discussed for exactly the reasons you mention. I’m leaving it for phase II.
Kevin: Thanks for your input. I realize that server logs contain this same information, however, I don’t want to look through any log report, and more importantly, I want to know right now that I have a problem on my site. A simple e-mail does the trick.
Daniel, Chris, George: Thanks for your kind words. Yes, a large part of this effort was driven by trying to provide the user with accurate and specific information with links that will help them find what they’re looking for.