Dealing with A Seller Reluctant To Allow Customer / Employee Diligence

I am working on a deal where the seller is reluctant to have me speak to any customers or employees as part of diligence. He has been extremely cooperative on all other requests and is open-book everywhere else. He has no NPS/customer sentiment tracking so all feedback is anecdotal from him. On the plus side, his customer churn is low and most have been with him for 10+ years. For context, this deal has 51% customer concentration with their largest customer who has been with the company for 12 years. He is pointing to their track record as sufficient backing for his claims of their satisfaction, and the deal is structured where he is financing 45% of it himself which is forgivable if revenue drops due to the main customer (or other large ones) leaving, which keeps him on the hook for retention.

I can empathize with his hesitation given the sensitivity of the sale to customers and employees, but want to make sure I am doing proper diligence. I even suggested using a 3rd party to conduct (e.g. Strategex) surveys but he doesn't like that idea either as he feels it is too formal/"heavy handed" and will spook customers since they have never done anything like it. His company is only 4 employees (though doing $13M in revenue) so he doesn't want to come across as "bureaucratic" to his clients.

Any advice around how to proceed? One advisor of mine suggested I threaten to walk from the deal if I can't speak with some clients, but I am not sure if that is overkill if I like the deal on all other fronts. That said, I don't want to go into this blind or be overly trusting and pay the price later. Curious to get some more opinions on it. Thanks!