Broker is threatening that I cannot buy a flat without him and if I try to he'll create problems

I got a flat on rent via a broker in Kandivali, Mumbai. Later the landlord approached me with an offer to sell it. Now we've decided to do the deal and the broker got to know about it and he called me saying that I cannot buy the flat without him and I have to pay him brokerage because he got the flat for me to rent and gave me the landlord's contact.

He's threatening that he won't let me buy if I don't involve him. I know legally I'm not obliged to involve him. My concern is that I live alone and he's threatening physical violence and asking me to meet him.

I'm not going to meet him or have any further contact. What can I do legally to get him out of my hair? Also can filing a case later show up in my record for background checks for jobs and visas? I'm skeptical about legalities because I don't want them affecting my job or visa.