However, moments later a penalty was awarded to Newport when the ball struck Marc Albrighton's arm, which was converted by Padraig Amond in the 85th minute. John Marquis gave the visitors a fifth minute lead before Andrew Hughes equalised 10 minutes into the second half but Grant McCann's men stormed into the fourth round thanks to further strikes from Thomas Anderson and Mallik Wilks.
Newport, who took Tottenham Hotspur to a replay previous year, held on to beat a top-flight side for the first time since 1964.
Oldham had to come from a goal down in the final 14 minutes to beat Fulham at Craven Cottage.
Amond stepped up calmly to send his penalty past Leicester keeper Danny Ward, once a team mate at Morecambe. He faced a few of my penalties in training so I was hoping he wouldn't remember.
Morgan, though, believes the minnows' win was undeserved, explaining: "It was a hard day and the magic of the FA Cup happened, definitely".
The Foxes go to Rodney Parade in the third round on Sunday with the Exiles chasing a top flight scalp for the first time since 1964.
Groundbreaking flyby image of Ultima Thule, on edge of our solar system
So what is Ultima Thule? In the meantime, they have released the closest image taken of Ultima just before the flyby. Traveling at 31,500 miles per hour (50,700 kph), the spacecraft could easily be knocked out by a rice-size particle.
"I thought we played well, especially the second half".
Fulham were awarded a soft penalty but Alexsandar Mitrovic, brought off the bench to take the kick, had his effort superbly saved by a Danish keeper Daniel Iversen.
Claudio Ranieri's Fulham reign has got off to a hard start and it got worse for the former Leicester boss at the hands of Pete Wild's Oldham.
Flynn added: "It's nearly Roy of the Rovers stuff, I can't believe it". It is one you have to savour. Barnet is now managed by Darren Currie, the nephew of Sheffield United great Tony Currie.
I don't know if we deserved it or not.
"We have put Newport County on the map and I am so proud".
However, in the league, his side have registered wins against Chelsea, Manchester City and Everton during the festive period. Championship strugglers Rotherham offered some resistance, but any hopes of a giant-killing were ended by half-time - Phil Foden and a Semi Ajayi own goal putting the result beyond doubt.