The problem is not refugees. The problem is fear and cowardice.

In early 2013, my editor sent me to Penrith to write a colour story about the marginal Western Sydney seat of Lindsay.  On a sunny weekday afternoon I wandered in and out of shops on the main strip, talking to business owners and customers about politics.

Inevitably the subject of asylum seekers would come up, and as I trudged in and out of local businesses I was told over and over again that they were stealing jobs and taking handouts from the government.

About half way down the strip I stepped into a delicatessen that was basically two shops in one space. Half the shelves were stacked with eastern European canned goods, jars of pickles and an array of crackers I’d not seen before. Behind one counter a middle-aged Polish woman was making sandwiches heavy on deli meats and gherkins.

Across the floor, behind another counter groaning with freshly julienned carrots, fluffy rolls and cold pork cuts, was a Vietnamese woman around the same age as her shopmate.

They both told me they'd arrived in Australia decades ago. The Vietnamese woman then said she thought we shouldn't let boat people come because they weren't hard workers. They were getting everything handed to them by the government.

"We worked hard", she said. "Why don't they?"

I asked her why she thought that, and she told me she'd read it in the newspaper.

So when it emerged this week that the majority of Australians don't want more asylum seekers to come here I wasn't surprised. For years, we've been fed messages about refugees and the dangers of letting them in.

We've been taught they're dole bludgers, or job snatchers, or even terrorists.

Our government once accused them of throwing their babies in the ocean. For the past 15 years the way we've talked about asylum seekers has helped to breed a climate of fear and reluctance to help.

So it's not at all surprising that Immigration Minister Peter Dutton felt comfortable going on television and calling them (I'm paraphrasing, although only a little) illiterate dole-bludging job-thieves.

It was absolutely wrong. But it wasn't surprising. Neither was Julie Bishop backing him in, nor Malcolm Turnbull.

The way Australians see refugees is influenced by the messages our politicians and the media send.

On some issues it is important to lead, not follow the public orthodoxy. We should have learnt that lesson decades ago, when we were faced with the mass exodus of Vietnamese people from their war-torn homes.

While the Fraser Government's approach may not have been perfect, it paved the way for the resettlement of thousands of refugees, and Fraser himself encouraged Australians to embrace a more multicultural country.

He did not bow to xenophobic concerns, and the fabric of our society is the better for it.

Where are the leaders today who will make the arguments people need to hear? That refugees are actually a vital part of our economy and society. That they are some of our best and brightest, but that they also fill jobs Australians don't want to do.

Like the hundreds of refugees who live in regional Victorian Shepparton who work on farms and in the nearby abattoirs.

Just this week we've seen the Government's plan for a so-called "backpacker tax" thwarted by farmers worried they wont have enough workers. At the same time the Government is making it easier for the Brits to come over here and work menial jobs, Dutton is scaremongering about refugees who can't read stealing our jobs.

It's a mixed message, designed to tap into fears on all sides.

Australia has a long history of racially fraught immigration policy. From the white Australia policy that was a founding principle of our oldest political party (the Labor party), to early 20th century bans on Chinese working men bringing wives or families over, followed up by mid-century screening of potential immigrants leading to a heavily euro-centric intake, we've never excelled at this.

And yet the true face of Australia today is incredibly diverse, and so much richer for it. We've taken more and more refugees over the years, and we have always found room for them. There's no reason to assume that will be different this time around.

The truth is that people are scared of what they don't know. It's the job of leaders to step into that unknown and reassure us. Find a new path, and lead us down it.

That hasn't happened with refugee policy. We're all still wandering around in circles waiting for someone to have the guts to stand up.

Late last year Canadian Prime Minister Justin Trudeau made headlines around the world because he met Syrian refugees at the airport, welcoming them to their new home.

Trudeau embraced not just an increased intake of refugees, but he literally went to the airport and embraced the refugees.

It was a small gesture, but it sent a very big message.

In 2014, Australia recognised or resettled less than half of one percent of all refugees in the world.

We are not punching above our weight, we are not being inundated, but we do hold some responsibility.

This is a complex issue with no simple solution. But there is no place for scaremongering and point-scoring.

What we have seen from the Immigration Minister, the Foreign Minister and others plays to our basest fears.

We see that in the polling. We see it in the streets. And if we want to change it, we must tackle fear head on. Because this debate is not just about livelihoods, it's about lives.