Two men have been arrested after a siege in Port Melbourne.

Police were called when an argument broke out between the men, believed to be brothers, one of whom was armed with a knife.

The men refused to leave the Clifford St unit for two hours.

Tensions between residents on the street boiled over as the brothers were taken away.

Some begun shouting abuse and, at one point, a woman was dragged across the street by the hair.

Neighbour Matthew Pylker, who lives across the road with his wife and two young children, said the family has been terrorising the street for at least two years.

“My wife can’t go to the park by herself with her kids, she’s always scared,” Mr Pylker told 3AW.

The situation has now been subdued, but Mr Pylker fears it is only a matter of time before tensions escalate again.

“A beautiful area like Port Melbourne gets ruined by a few individuals like this,” he said.