After winning the Golden Ticket, sending him through to the next show, he told the hosts: "Mate, we live in a world where the people we elect don't want to feed hungry children. This (routine) is the least crazy thing."
Rufus was referring to the ongoing school meals issue, after shocking pictures emerged of the "unacceptable" food parcels some people were being given.
Rufus himself hit back at backlash, while thanking fans for their support, in a tweet that read: "To the people who want to be cross with me - you were never going to be my friend. I wish it were different, but such is the world we now live in.
"To everyone else, thanks for all your lovely messages. I will try to keep doing funny ice dances for you x."
He told The Mirror: "I think they were absolutely appropriate. I've said this before, he's a comedian, he's an intelligent and emotional man who takes cues from the real world around him, to formulate an opinion. That's what we all do.
"He said something in response which was the way he felt, of course he was allowed to say it."
Ashley went on: "If people agree or don't agree, the idea that people are not allowed to express an opinion because the show is a 'family show' is nonsense.
"I think what people are saying is its' a family show and they only want opinions on the telly that they agree with, which is a different thing."
Ashley himself faced backlash along with his dance group Diversity, after the group performed a moving and powerful routine on Britain's Got Talent last year, which referenced the Black Lives Matter movement.
Ofcom received thousands of complaints over it being included, as many hit out over "political views being aired on a family show" - while ITV, the show and the majority of viewers stood by the group and their incredible performance.
The same goes for Rufus' remarks, with many viewers and fans showing their support and backing what he had to say.
Dancing On Ice airs Sundays at 6pm on ITV.