Refinancing is math

Pure and simple. How much will you save on interest, vs. how much will you pay in penalties? Reading the fine print before you sign your mortgage in the first place will help immensely, because then you will know what sort of penalty to expect. But the bottom line is that you should never assume what your penalty might be, because it’s almost always going to be a bit higher.