Hi, David, thanks for taking the time to comment. I appreciate your perspective. I think a lot of the difference in our perspectives comes down to how you're different from the audience I was writing for and the fact that I almost always am coming into a project that has significant technical debt that has been created over time based on the decisions of both the business and previous developers.
I was writing for an audience that probably does not have great refactoring skills and also probably is not aware of that fact. So they're not going to be able to efficiently set up testing infrastructure where none exists, learn how to develop with tests, and fix a bunch of bugs "under the radar." It's not going to happen. I, too, have done significant refactors "under the radar" and sometimes against the explicit request of the business, but I have a tremendous amount of confidence in my ability to do that without making things worse and I'm always ready to get fired at any time. Not everyone is in that position, and in any environment with code reviews (which I always advocate for even if that's not part of the process when I arrive), people will know you did work that wasn't what you were assigned. In many places this will get you in trouble if not fired, regardless of your opinion that the business shouldn't tell you how to code. They have their sprint plan and they're expecting that developers will be concentrating on the assigned work, not doing other work that might enable that work, but wasn't scoped.
I vehemently disagree with you that many people spend their lives refactoring because they don't know how to read code. I have never worked anywhere where the requirements were captured outside the code. Therefore, if you can't read the code, you can't refactor it. But if code is hard to read, it takes longer to work on and it's difficult to reason about what it will do, so illegible code should be refactored (maybe it's not a priority for refactoring, but it should be on the list).
You're right that code that's riddled with bugs, etc., should be refactored, but the problem is that the bugs in it may be the reason for some other code, so once you fix the bugs you may find a whole lot of other bugs you have to fix you hadn't counted on in your refactor. In addition, since requirements are often only captured in the code, if there's a requirement that's difficult to see in code that bounces back and forth all over the place, that requirement could be missed. A lot of time, all of this was hashed out through weeks or months of QA, and now you're diverting QA resources unexpectedly to code that the business had not realized was problematic.
An example I recently had is I fixed a grid so it was now getting updates when the data updated, and then all the item editors that were managing keyboard handling suddenly were throwing errors when they were calling setTimeout to reference DOM elements that were no longer present after the grid updated. So then we had to fix that, and the business requirements were deceptively tricky for those editors. And that wasn't even because of the actual business requirements, but because logic elsewhere in the system that was set and would have been its own can of worms to refactor.
I don't think that simply prioritizing simplicity and maintainability works for most developers without the discipline of TDD to teach them how to do it. I have literally never seen a developer who lacked the skill of writing simple code out of the gate learn it without doing TDD, and even then I've only seen one or two manage it. Sadly, it's a skill that is essential to our profession that almost no one has.
The only managers I've ever seen demanding tests are ones that see how much better my code is than others', and they think mandating TDD will translate to the entire team. This actually can work, given a robust code review process, but if the developers don't understand and internalize the reason for it, it won't work. Managers seem not to realize that's the key part and you can't mandate it. Nevertheless, the intended audience of my article is _not_ the people who aren't internalizing the concepts of using TDD to help them learn to code well the first time. It's the people who want to write good code and are looking to understand how to get the business to let them.
I do agree with you that many developers do not feel it's their job to push back on the business over code quality and scope. However, not everyone thinks it's worthwhile to protect a business from itself at the risk of their own livelihood. And knowing how companies don't give a f about anyone that works for them in the end, it's hard to say they're wrong.
IMO it's almost never appropriate to keep the old code in place after a refactor. I've tried it both ways, and it's more trouble than it's worth. The new solution is far and away better 99% of the time. Keeping the old code just results in confusion and tech debt. It's like the people who leave giant blocks of commented-out code all over the place. However, this may be because when I refactor I almost always entirely reimagine the problem into something that's way simpler than what was there rather than taking a cleaner version of the original solution. That can't really be done incrementally in most cases.
I disagree with you about keeping the business out of refactors for a couple of reasons. I've already touched on the fact that most developers aren't independently wealthy and probably don't want to get fired over not doing things the way the business says it wants. But also most developers also don't have access to the company financials or understand the impact _to the business_ of doing this work vs. that work. By the time I have joined every project I've been on for the past 5+ years, there was significant technical debt all through the projects. I couldn't possibly prioritize all of the refactoring that needed to be done by myself--I lacked the context--and clearly the existing teams either didn't prioritize it highly or didn't know how to deal with it before I got there.
