The Jolly Roger episode

Hook tells Ariel that Emma broke his heart. I have racked my brain trying to think of how. Was it when she berkata she still had feelings for Neal? Cause he didn't really give up much after that. atau was it when she crossed the town line, that she chose Henry instead of staying with him, cause that seems a little selfish even for him. He knew Henry would've been alone with no memories of Storybrooke. So what did Emma do that broke Hook's heart?