The passage of video and data on digital wiring can be done fairly
easily if you digitize the video AND if your network supports 'isochronous'
real-time data transfer. Alas, Ethernet doesn't necessarily support real-time
data (thus, VoIP is still somewhat tricky, IP video is flaky).
The gamers that share dungeons over IP use shorthand (the IPX protocol),
not really sharing the full video stream.
Firewire, aka iLink, aka 1394b, is intended for this kind of thing.
Firewire DOES support isochronous, and there's a variant of the standard
that works on Cat5 twisted pairs. It's common (in the Macintosh world)
to use Firewire ports for TCP/IP, too, which would cover the data needs of most computers, and would share with a video stream.
The bad news: that Cat5 element of the standard requires Firewire to slow
to 100 Mbit, and I've never actually seen it implemented. Firewire ports are usually 400 Mbit, or 800/400 Mbit. Also, the video
stream for digital transmission is typically HD video, which requires some
hardware to encode (not easy to splice in to a VGA channel). Both ends of the
channel are Firewire (so it can't go to or through an Ethernet switch any more). That means you'd get TCP/IP connection between the two, but
one would have to host bridge software to connect to Ethernet for any kind of
network beyond those first two machines.
No one else has mentioned it (so I guess I will); the same common-mode
connection that provides Ethernet power without disturbing the differential
mode network signals, can be used to make an additional 'wire' of each of the
(four, presumably) pairs in a Cat5 cable. This would amount to an unshielded, untwisted quad of not-well-specified wiring, and one could
conceivably send signals through it. It would radiate badly, and the crosstalk would be significant, and it requires wiring to the central tap of the Ethernet
interface's magnetics (to the center tap of the transformer output).
That's pretty ugly.
Another non-Ethernet way to proceed, would be to use three pairs to send
RGB video signals (with suitable 75 ohm VGA/ 120 ohm baluns and
amplification), and the fourth for a modem connection (alas, probably that'd be limited to 33 kbaud unless you can get some pricey DSL modems
that can really use the cable bandwidth). It used to be easy to get sync-on-green from a video card, and maybe it still is. The receiver would have to regenerate
the sync signals in addition to buffering the balanced RGB. It's not gonna be
a simple wire solution.