What to do when a broker pushes back on a break fee?

I included a break fee in an LOI I recently sent. The broker said they their buyer will not accept a break fee.
For me this is a red flag because why would the buyer or broker not accept a break fee unless they had
an idea to not complete the transaction?

Has anyone been in this situation before and if so, how did you hande it? I am worried that I will do alot
of work and then have the owner pull out of the transaction.