Sydneysiders paying their respects to siege victims at Martin Place have shouted down a group of anti-Islamic protesters who appeared at the memorial on Friday afternoon.

Four men waving Australian and Southern Cross flags shouted slogans including "Islam is evil" and "not all cultures are equal - Islam is inferior".

The men also accused Muslims of being "murderers" and supporting female genital mutilation, according to a witness.

The group bore signs proclaiming: "Muslim terrorists not wanted here - neither are their leftist supporters", "Tony Abbott - Will you protect us from multiculturalism?" and "We didn't start the fire!"

One of the men was Nick Folkes, who has previously been associated with the fringe Australian Protectionist Party and is now the chairman of the "Party for Freedom", which calls for an end to Muslim immigration and "state-sanctioned multiculturalism".

A post about the siege on the group's website calls for a "tribute to remember the fallen" at 5pm on Friday, referring to Tori Johnson and Katrina Dawson, who were killed in the siege.

Mr Folkes told Fairfax Media the demonstration was peaceful and that he received support from some members of the public.

"We saw it as a tribute, mate, but it turned into a heated exchange with different groups," he said.

"The police moved us on. They wanted to move us on because…some of the Muslims and lefties didn't want us there."

A witness said the crowd booed the men's speeches and was universal in its condemnation of the tirade, except for one man who appeared to shout in support of the protesters. A member of the public confronted the men at about 6.20pm and the altercation was broken up by police, a witness said.

There has been a high police presence at Martin Place since Monday.

Mr Folkes denied he was a racist or white supremacist.

"Islam is not a race. We're criticising a political ideology that is totalitarian in nature," he said.

The Party for Freedom website refers to playing "spot the Aussie" in Australia's cities. It criticises the national immigration policy, and contains racially motivated statements, including that Australia is "over-run with third world intruders".

With Nicole Hasham