You didn't leave him.

He let you go. 

If you meant anything to him, he would have kept fighting for you.