Cowell hasn't spoken to Cheryl since sacking

Simon Cowell, whose new talent competition debuts this fall, is willing to give into the demands of the pop diva, whom he says "wanted to be involved in the show from day one."


"You can't say no to Mariah," Cowell, 51, tells PEOPLE. "We have to find a role for her. I'm a huge fan of hers – not just as an artist, but as a person, and I love her to death."


But Cowell also says that having Carey on the show poses a slight problem.


"We just can't shut Paula Abdul and Nicole Scherzinger up," he joked. "Add Mariah into the mix as well and you won't even have anyone singing.


Trying to get her to understand what I did, it was never meant to be negative. It was a decision I thought was the right one and I think she is more comfortable in England than I saw her here (in US)," he said.


Cole, who had judged the British version of the show, filmed four episodes of the US version of "X Factor" before she was sacked.


Cowell also admitted his decision to replace Cheryl with Nicole Scherzinger might have been rash.


"Maybe I didn't give her enough time, and you have to be in the moment. I offered her the show, and initially she accepted it. Then it went public that she didn't and had the chance to still come back, but for whatever reason I couldn't get her (on the phone). Unless I spoke to her, there was no way I would have her back," he said.