No, it really doesn't take longer to build software right. For example, when you code test-first, you're only running the piece of the code you're directly working on vs bootstrapping the whole app, then navigating to your area of interest. And your focus on just one small part of the problem gets you to a solution much faster. So if you build software right, it is faster and you'll have tests as part of the process--they're not an extra you do after-the-fact, if you have time.
The problem is most people are told that it takes longer to build software right, so they don't practice building software right most of the time. Since they don't practice it, they are indeed slower when they try to do that, as with anything you don't practice. If you _always_ build software right, you will be faster at it (presumably, but since you never do it wrong you don't have a benchmark, just as those who never do it right have no benchmark either).