It can be a big deal, it doesn't HAVE to be a big deal. Primers have their own little spot amongst the cardinal rules of handloading, at least IMO.How big a deal is switching primers? I get it powders have different properties, but do primers?
As an example, I shoot pretty much any range ammo but carry Speer Gold Dots in any caliber. Do you, say, only use a Federal primer across all loads or do you follow the primer recommendation in the loading manual all the time?
The cardinal rule with primers is merely a sub-rule: when you've worked up ANY load, you can't switch anything without preparing yourself for DIFFERENT results. This is not to say that you must re-work the load, but if you are running some loads at/near/(*gasp* beyond) published maximums, then it's a dangerous and move to just SWITCH one part of the load without doing the necessary steps to re-work that load. And by re-work, I mean LOWER the powder charge, make the change, then use proper methods to work BACK to where you were or wanted to be.
And yes, a primer is a huge (massive!) part of that equation. So for sure, you can buy every different brand on the market and use each/any as you please, however if you build a HOT load and then run out of SuperZoom primers and just stick some MegaBlast in there instead, you can find yourself in a bad place with pressure, where you should not ever be.
My recommendation with primers is to find a brand that you believe you can get and more or less marry yourself to that brand. It's not necessary, but it simplifies things for you and takes the primer swap equation out.
Now, with all that said, I'd also submit that handloading for handgun rounds with max pressures running from 17.5k to 35k (generally speaking) isn't really a place most folks are likely to get in to trouble. Things like warm loads, excessive pressure and small changes giving BIG result swings is far more likely & common in high pressure rifle rounds.
As to which primer to pick? Well, Federal has the long and well known reputation of having the softest cup of all pistol primers. What that means is that if you're ham-fisted with your equipment, a Federal is the first one you'll inadvertently -POP- at the load bench. You'll smear your shorts. Always wear the safety glasses when priming, no exceptions. But Federal primers are THE primers to use if you have a handgun that exhibits signs of weak or questionable striker/hammer hits. And this doesn't have to mean "broken gun", but many folks who tune revolvers for ultra-slick and extremely light trigger and smooth actions make use of the Federal primer to get reliable ignition.
Me, I'm pretty much married to CCI primers because that's what I started with when the old gun crank at the local shop pushed them across the counter at me. And for fun/sighs, that 1,000 count brick was $9. CCI has the opposite reputation -- hardest cup in all of pistol primerdom. So if your gun can reliably make 'em go bang, you're gun is running as it ought to.