Mordred's relationship to Toshi is relevant, yeah. I think whether or not Toshi is a legitimate prince isn't, really, because he has no kingdom to rule anyway. All that really matters is how Saber views him, and Saber definitely sees him as her heir.