What's the deal with new grads at places like Goldman Sachs and Morgan Stanley leaving after 2 years?

I've noticed this trend with a few different people I've met recently. Just one example, I met someone recently who graduated at the top of her class from a top Ivy, was hired by GS as an IB working in M&A (I thought this was the creme de la creme of IB?), but left after two years for an IB job at a SWF. I've seen this pattern too many times now for it to be purely coincidental.

Is having those companies on your resume that much of a draw for even higher TC elsewhere with better WLB? Even after just a 2 year stint? Is it a case where only the middling performers stay at BBs, while the top performers go elsewhere for faster raises and the bottom performers go to less demanding firms?