Ben Shephard, who hosted the X-Factor spin-off show the Xtra Factor, has quit the show to work on a BBC1 programme.
An X-Factor insider told the Sun, "Ben was not that happy when Dermot [O'Leary] got the job. He thought he should get a shot."
A spokesman for Shephard said there was no truth in the suggestion and that he wishes the X-Factor staff all the best.
Ben will now be working on the BBC show Dance X, which is currently looking for UK's best dancing talent to create a new dance group.
Original X-Factor host Kate Thornton was axed along with judge Louis Walsh.
Danni Minogue could step in as a new judge while Simon Cowell is still looking for a fourth.