The keyword is "favicon." You do a google search on that, and you'll see that webmasters are always very interested in howto enable them from their site.
For your use, PCmag's Favorg, as brought up by Smitee, is a popular utility and would probably be fun for you to play with.
A couple of additional Windows notes. Remember that any individual favorite [*.url file] on your machine allows you to customize its icon, just as you can do with local shortcuts [*.lnk files).
1. Context-click (or shift-F10) the favorites/*.url file
2. choose "Properties,"
4. then go into the "Change Icon" dialog box.
5. (you might want to assemble a collection of icons you like somewhere on a set folder for this)
Also, you can do away fully with that msft "e" thing being the default for your URL files.
1. In Explorer, choose _View, Folder Options, File Types [*]
2. In the File Types dialog, go down to select this type:
URL:HyperText Transfer Protocol
3. Click _Edit
4. In the Edit dialog, you will find the "Customize Icon" dialog.
From there you can change the default icon to display for favorites files that do not have custom or favicons associated with them.
Those two notes are just for general Windows customizations. Complementary activity is collecting icons that you like that are downloaded from Websites that use favicons. For keeping the associations between special favicons and their original *.url files, the PCMag utility would probably work best.
--
[*] re "In Explorer, choose _View, Folder Options, File Types." I think the exact sequence for getting to the File Types dialog might vary a little depending on which Windows OS. ? I don't have the info for instance for XP (But in the meantime, I'm sure most anyone here already is familiar with getting to it on their machines.)