Byron Allen‘s Comics Unleashed took over the timing of Stephen Colbert‘s talk show following its conclusion on May 21. In his recent interview, Allen opened up about his series replacing The Late Show’s slot.

Byron Allen clarifies he’s ‘not trying to replace’ Stephen Colbert

In his interview with NPR’s Newsmakers, Byron Allen talked about Comics Unleashed taking the time slot of Stephen Colbert’s show. “At the end of the day, I’m not trying to replace Colbert,” Allen said.

The comedian continued, “I am not trying to hold on to his audience because Comics Unleashed has been around 20 years and has its own audience.” He also shared how the audience will take time adjusting to his show at The Late Show’s slot.

“Not everybody’s gonna love me,” the 65-year-old shared. He further explained, “But there is that one or two percent that would be like ‘hell yeah, I’m rolling with you’ and I learned that at an early age.”

Moreover, Allen stated that he has “had nothing to do” with the cancellation of The Late Show while praising Colbert. “Stephen Colbert is an American treasure,” he expressed. The father of three continued, “He is a phenomenal human being. He is truly special.”

Moreover, he shared his thoughts on Colbert’s next project after the conclusion of his decade-long chat show. “Here’s the good news. He owns his talent. And he has an audience that loves him. He just has to decide where he wants to plant his flag next,” Allen noted.

He added, “Quite frankly, there are so many places he can go.” The television producer also shared that he would support Colbert “on anything he wants to do.” Allen continued, “He doesn’t need me. But I would do it in a nanosecond.”

In his interview with PEOPLE, Colbert explained how CBS might have “saved” him by cancelling his show. The 62-year-old said, “It takes a lot of bone marrow to do the show every day, and now I’ll be stepping down with enough time, enough energy to do other things that I want to do.”