As long as you are using riding specific shoes, it doesn't matter so much that you aren't using 510s. Ideally you have something that is as grippy as 510s, but even if you don't, technique is going to be the biggest factor.
The grippiest soles won't grip the pedals if you don't have enought pressure on them.
Pressure can come from 2 things:
1. Your body weight
2. The force you apply by pushing your hands against the grips and your feet back into the pedals. Imagine you are picking up a bowl by touching only the inside surfaces. How would you do it? You push against one side with one hand the other one against the opposing side to create tension.
If you are being pitched forward on rough stuff it sounds like your pedal may be sitting too far forward on your feet. Also you may be mistiming your body weight shift.
It's great advice to drop your heels, but you can't be doing that all the time; sometimes you have your toes downwards but you are pushing back into the pedals with your hands simultaneously pushing forward into the bars (creating pressure on the pedals).
Check out some videos on how to do a correct bunny hop; that will be the best way to visualise.
It will take you a good 10+ rides of perseverance and likely riding slower and clumsier than usual before you get back to the same level of confidence and speed as you were on clipless. From then it can only get better.
I made the transition from clipless to flats after about 7 years, so it definitely is possible. Don't give up!