It's always risky to second guess the meaning of the lyrics by any poet/songwriter. What seems most obvious is often not the actual meaning. Case in point, Springsteen's "I'm on Fire" which seems to be about passionate, unrequited love but, he explained, is actually about his anger at his mother's having to borrow money so her children could have school clothes and Christmas presents.
Since none of us can see into RDJ's memory and heart, we really don't know what he was writing about. The human heart can be broken by many people and events other than a lost love.
My take on the song is that RDJ's heart had been broken at some time, by some person or some event, but it's not for us listeners to know the, probably very painful, details.