Ok, so what went wrong here?

It sounds like your fwb is only a convenience to you and that if you found someone else more exciting you would leave your fwb again. It also sounds like your friend's feelings are deeper than yours and he almost may be afraid of starting up with you again for fear of getting hurt again. You may want to yze whether this up and down relationship is worth it.