sound system on a budget

If you remove those 'just a few drivers' and put new ones in without knowing what you're doing, then the chance you've got a box that performs well enough to justify the time, effort and cost is incredibly low. The crossover, woofer, and tweeter will all have been balanced with each other and to the box. Mix and match without the same care and a decent level of understanding, and you upset that balance, which gets more delicate the closer you get to the ear's most sensitive points (1-3kHz especially but basically everything from about 200Hz to 20kHz).

Irregularities in a sub design like -5dB at 40Hz and +5dB at 70Hz, or vice versa, may not be optimal but they'll be tolerable, and easily fixable with EQ. Room modes mess up that region a lot anyway so it's a standard issue to have to deal with. Our ears are much more forgiving at those frequencies, and most music is mixed and mastered with a lumpy bass frequency response in mind.

Similar bumps and dips higher up are a lot harder to fix and a lot harder to ignore.Crossover issues can be even worse. You could end up with components that are completely incompatible with one another or with the crossover - sensitivity or frequency response issues that have been compensated for with the original components in mind, that totally screw up the performance of the new drivers. End up with a woofer that crosses at 2.5kHz that shouldn't be crossed higher than 500Hz, or a tweeter crossed at  at 800Hz that shouldn't be crossed until 1.5kHz and you've basically got junk, and that's just looking at the most obvious issues with slotting new drivers into two way hifi boxes.

It's not that it's impossible to do it right, not at all, but the assumption you can just drop new components in so long as they physically fit and get good results because 'it's just a few drivers in a box', or that the same issues that speaker designers have to deal with 'on the regular' won't apply because you're not designing a box from scratch is dead wrong. You might get lucky and get a tolerable result, but you might not.

Building a midtop from scratch is easier to do properly than retrofitting an old midtop with new components, unless part of the retrofit includes a new crossover network (not easy to do properly), and even still, most hifi speakers have some form of irregular baffle cutout (for a tweeter horn ie, but sometimes also the bass driver cutout, or some proprietary mounting system) which finding an adequate replacement for which also performs near enough the same won't be easy, and adjusting could be almost as involved a process as building a new box.

Otherwise they may have a driver mounted to a horn integral to the box, which again may not be with a standard fitting and which will not generally be a listed component. You'd need to model for suitability with an alternative driver, or just guess and pray, which is really not a very effective design philosophy and not one you should be recommending to people just because you can't think of a better one.

The net is full of reasonably priced component kits for hifi speakers, including box panels and/or suitable box design documentation. If you've got the money to buy quality drivers for a retrofit, you may as well buy a kit like that, unless you're willing to put in the grind necessary to understand how to develop your own. You'll get much better results that way.