However, the veteran midfielder insists he's happy with his decision as he helps revive Milan this season and mount a serious title challenge.
He told Marca: "After Real Madrid, wherever you go, it's a step backwards. That's beyond question, and all the players will be able to confirm it.
"However, I think I've arrived at a club that, in terms of history and name, is very close to Real Madrid, and for me, it's the ideal situation. Especially because I grew up watching Italian football; Milan was my favourite club..."
Modric also said: "The whole world knows what Real Madrid is, a club where mediocrity is not tolerated. The mere fact that we've maintained this level over the years is the most incredible thing about all our successes.
"I've always said it: my dream was to retire at Real Madrid. But everything simply has a beginning and an end; sometimes, not everything happens the way you want it."