When shifting (up or down) the trick is to 'float' off the throttle when you go from being in-gear, into neutral, to reduce the load on the cogs as much as possible. This makes for easier gear dis-engagement, and greatly reduces the chance of chipping the cogs
As you say, to 'find' and engage another gear without using the clutch you need to be really precise about matching the revs
exactly to what the new gear demands - because there's no slop and no cushioning and no room for error
As mentioned, the big risk is chipping teeth off the gears, and likewise giving the synchros a hard time. When dis-engaging, slide it out promptly. But be gentle when attempting to re-engage from neutral. Never force the gearlever - use your fingertips rather than your fist. If you muscle the shift, or
have to force the shift -which indicates poor rev-matching, this greatly increases the chance of causing damage - including simply bending the gear linkages if your car has those
So the key imo is being gentle. If you have to force it you're doing it 'rong'
Doing rapid drag upshifts in a conventional gearbox without using the clutch would pose the worst of risks imho
Btw I've driven big trucks with Spicer/Road-Ranger non-syncro gearboxes, and could shift those with only two fingers resting on the top of the gear-knob