21” to 20”

So I know a lot of people go from 21s down to 19 because it saves a lot of money on the tires and less chances at cracking rims, etc…. But what about going down to 20s? has anybody had experience and did it make much of a difference from the 21s!?