The president of the United States Hispanic Chamber of Commerce says he will not resign from the president's National Diversity Council despite a mass exodus by other executives from similar appointments even though he calls Trump's response to Charlottesville a "monumental failure."

"These are difficult times for the American people," Javier Palomarez said in a WFAA interview to explain his stance. "We are not jumping ship at this point because we believe there is work yet to be done."

However, he made it clear the president's delayed response to the Charlottesville violence, and Mr. Trump's insistence on placing "blame on both sides" troubles him deeply.

"Something needs to be very clear to our president. We're not splitting the atom here. If you have an opportunity as the president of the United States to denounce Nazis, you take it. Period. End of paragraph. He had an opportunity to openly, unequivocally to denounce bigotry, racism, hatred, Nazis, and he didn't take that opportunity," he said.

Palomarez, a successful businessman from Dallas and the son of migrant farm workers, has never been a fan of President Trump. In fact, during the election he was very clear with his opinions.

"I called him a buffoon. I called him a clown. I said he was unfit for the party, that he had neither the talent nor the temperament. I called him a piaso on national television," he said.

Trump called him to serve on the National Diversity Council anyway, and he accepted, to give a voice to the 4.2 million Hispanic owned businesses in the country that contribute more than $668 billion to the American economy. And that's why he says he chooses to continue serving.

"We have to fight to have a room, to have a place at the table. That's my focus. We are resolute that the American small business community has to have a voice within this administration. I accepted that challenge. As an American, that's my job. And I have to stay focused on that job, even at times like this," he said.

Palomarez says his own rise from poverty is proof anything is possible in America. He says he's a man with hope, who hopes Trump can change. But if not, the man who once called Mr. Trump a buffoon says he will try to keep working by his side.

"We have great conversations with the people around him. People that are fair-minded, intelligent, progressive thinkers that understand the criticality of American small business," he said.

"This isn't about serving a president. This is about serving the American people and those job creators and business owners who put me in this post. Until they tell me it's time to step off, I'm on the job."