I have done a LOT of magnet swaps. It's kind of a rabbit hole but a lot of fun. It will change your sound and feel to an extent, but also be aware that other factors such a baseplate material affects the tone too. You can even put some copper, bronze, cobalt, silver, nickel or various other metals on the baseplate and sometimes get an effect as they usually have some kind of paramagnetic property. However, the effect of magnets or other materials really depends on the pickups.
You really need to know the various magnets and effects they produce, and know exactly what you want to achieve. Second, you need to have a careful setup to record and listen to before/after clips. Pay attention also to the feel. Some magnets have an aged/degaussed effect that softens things and causes a bit more articulate response. Other magnets like A8 are very strong and thickens the midrange. If you have a Dimarzio, for example, A8 may not be a good option if the pickup already has a thick midrange. Dimarzio, Duncan, etc. typically use different baseplate materials.
Note the magnets can be polished, roughcast, unoriented, etc. This affects the pickup characteristics.
Probably the most useful for experimenting are roughcast for stronger magnets like a5 and polished for weaker magnets like A2. Rough magnets have a more unpredictable magnetic field so it can impart some nice individuality to your tone.
In my experience, A4 and UOA5 are the most pleasing magnets to swap. You don't hear a lot about A4 but it's really underrated. A4 gives you the power and low end of A5 but some of the sweetness of A2. It's my favorite magnet. UOA5 is a very popular type but it's effects are usually very mild or sometimes negligible.
Also, using A2 in really powerful pickups with lots of low end/low mids can have a very desirable effect and make a one-trick pony into a more versatile pickup.
Lastly, the ultimate pickup tweak is combining font/back coils from different pickups. There are a few general guidelines for getting a useful sound from this (that is, make sense to try) but you'd be better off having a lot of pickup tinkering experience before undertaking such a thing.