.tar.gz is actually two extensions.
.tar is an archive of files. You can think of it as a way of treating a set of files as a single entity.
.gz is the compression part; .gz means compressed with gzip. (Other common extensions indicating other types of compression are .bz2, .Z, and .zip).
So a .tar.gz is a tar file that has been compressed with gzip. You might also see something like .html.gz (compressed HTML file) or .ps.gz (compressed PostScript file).
.tar.gz files are commonly used for distributing the source code and documentation for an application or library. You can compile this source code to obtain a program which you can then install. That lengthy name is often the name and version of the software it contains.
A .rpm is, as mentioned above, a Red Hat Package Manager package. It contains already-built programs or libraries, so it does not need to be compiled before it is installed. This means the install will be faster, but you have less chance to customize the program to your particular needs; you just get what Red Hat decided you should get (which is often right).
Red Hat's package manager also keeps track of the files installed on your system, making it easier to uninstall software from a .rpm than it would be to uninstall software compiled from source and installed "manually."
A .bin file is normally a script that, when executed, unpacks itself and goes through a process for installing some kind of software. It may or may not compile source code; it may or may not keep track of where it installed files so you can remove them later.
