Metal:
maudlin of the Well - The band members in Kayo Dot's old band. Basically the same concept except with much more death metal and jazz elements. (See Kayo Dot)
http://www.youtube.com/watch?v=13GyZh_if20
Cynic - Jazz death metal. One of the first bands that heavily use robotic vocals.
http://www.youtube.com/watch?v=yRbLaNqsxhM
Disillusion - Prog metal that doesnt sound like a big wankfest. Riffs are extremely catchy. (Note: they seem to experiment a lot with genres, as their newest release is very industrial)
Equilibrium - Folk metal done right
Lykathea Aflame - Atmospheric Brutal Death Metal (yes, it does exist). The drummer has got to be God himself.
Some non metal-related bands:
Kayo Dot - Avant-garde-jazzy-progressive-classical-rock with huge amounts of experimentation and a little grind influence for good measure
http://www.youtube.com/watch?v=geJnvIQ7HnY
Portal - Spacy, atmospheric, jazzy group with female vocals
http://www.kringelbass.com/mp3/Other/Portal-Cosmos.mp3
(entire album is free to download at that site)
Stars of the Lid - Ambient music with a huge classical influence
http://www.youtube.com/watch?v=s3CoihmU1ok
Streetlight Manifesto - Technical 3rd wave ska. Literally shits over all other 3rd wave ska bands.
Other bands:
Ulver - Started as a black metal band, changed into neofolk, went back to black metal, and now kinda floats around the electronica, ambient and experimental genres. The posted one is a later work, and its kinda electornica-classical-experimental.