My (47f) boyfriend (40m) refuses to let me break up with him

I have had months of issues with him being controlling (what I wear, who I see/talk to. In one instance, he stated that I should be wearing a bra in my own home and got upset that I didn’t. I have had really bad panic attacks with him and have tried previously to break up with him. This is attempt #2, and my guard is falling again as he keeps saying “love should be enough” and I’m feeling guilty of abandoning him. He is saying that I am not giving him a good enough reason to break up with him as he is working on fixing the controlling factors .

How can I get him to understand that sometimes love isn’t enough for a relationship to survive?

Any advice is appreciated, thank you