Speaking on Sky News, Lord Mandelson, the remain campaigner, accused Vote Leave of "distorting" facts and said that while it was right to raise the 50 criminals, people must take into account the 6,500 criminals rounded up and deported from the UK since 2010 because of the European Arrest Warrant.
He said: "They take a germ of truth, they then generalise from it and in the process they distort the real picture."
However, Mr Raab told Sky News Lord Mandelson was conflating two issues and pointed out that extradition was different to deportation.
Video:Cameron's EU Event: What Happened?
Mr Raab said that the UK removed eight times as many non-EU nationals as EU nationals from the country, highlighting the restrictive nature of the rules.
It comes as it emerged pro-European MPs could stage a campaign to reverse Brexit in the event the public votes to leave – or limit the impact by retaining membership of the single market and therefore retaining freedom of movement rules.
Around two-thirds of MPs are in favour of remaining in the EU and could use their weight to put pressure on the government to stay in the single market, therefore keeping a raft of EU regulations.
However, if they did manoeuvre to soften the extent of Brexit in a move being referred to as "reverse Maastricht" they would risk the wrath of a public that had voted differently – with a General Election in 2020.
The Labour MP and remain campaigner Stephen Kinnock told the Times: "MPs will be presented with a very difficult choice in the event of a Leave vote, because we have no idea what a post-Brexit UK looks like, and the referendum will not provide a specific mandate in terms of which Brexit model has the support of the electorate."