This sounds like you are just annoyed with him. In fact something could be bothering you outside of your relationship that could cause it. Like has an ex ever done the same thing?
If that’s not it. I would talk to him. But it’s all you. You have to be patient and gentle and learn to love his flaws. I’m sure he has the same issue. Love is unconditional, remember that.
This is tough.. Have you tried speaking to her? Have you put forth effort and showed that you really do love her? If not, was you’re relationship the same way? If a girl isn’t getting the attention she needs, she may be interpreting that as to thinking you don’t love her. I’ve took that as the guy didn’t know how to treat me right before. Show her that you are what she needs. Show her you can be her everything because she is your everything. If she’s all you want, then prove it.
I would suggest talking to him again if you feel it’s a mistake. If you don’t want to talk to him still after you’ve brought him back in to your life, no one is forcing you to continue to keep him there. Better Safe than Sorry.