Chuck Bass never left him