It seems like, if you stick three 360-degree (4pi steradian?) cameras in a triangle, a computer should be able to reconstruct a stereo pair of viewpoints in any direction (by reverse mapping the shape of the space and re-projecting the scene onto it). Cameras occluding each other would be a problem, though...

Ironically (?), pannable stereo is much easier for constructed environments than for captured video. And in either case it requires hardware that's up to reconstituting the scene in realtime.