I've seen this before and played with it some. Had a client who sold over $2million a year shut off his ppc. Here's what will happen if you're not the top seller on page 1: you won't see much impact at first, but you will slowly loose rankings and relevancy for keywords (and BSR) over roughly 6-8 weeks and you'll notice the difference in sales. At first the increased profit from no ppc will seem like it was worth it but over time your loss of relevancy for important keywords will affect BSR in a big way and of course ranking on import keywords. It takes twice as long once you turn ppc back on to get back to where you were. Also Amazon's constantly changing algorithm doesn't help matters either.
If you do turn it off, just pause it so you can at least keep some of the history you built in your campaigns.