The League Two side had appeared on course for a huge shock when Padraig Amond broke the deadlock after 38 minutes, heading home Scot Bennett's cross.
West Ham were without several injured first-choice players and an upset always looked on the cards.
Newport, a massive 72 places behind a Spurs side who are fifth in the Premier League, sparked delirious scenes at Rodney Parade when Amond gave them a 38th-minute lead.
Although it eventually led to correct decisions being made, the time it took, with nearly four minutes elapsing between Mohamed Salah being fouled and Craig Pawson awarding a penalty which Roberto Firmino missed, left supporters and players on all sides angry and frustrated.
Newport captain Joss Labadie and Ben White returned, while Matty Dolan was also back, but only on the bench.
And he almost made an immediate impact, feeding Son Hueng-min into the left side of the area and he tested Day.
England Women make sure not to leave Sydney empty-handed
As a team we have seen progression in such a short amount of time, although potentially not reflected in our finishing position. The Olympic gold medallists will take on the US in the Cup quarter-finals while New Zealand play arch-rivals Australia.
An ignominious exit loomed for a reasonably strong Tottenham outfit but, with the visitors seemingly fast running out of ideas, substitute Son Heung-min flicked on a corner and Kane ended a frustrating night in front of goal with the simplest of far-post tap-ins.
Spurs were now dominating possession but clear-cut chances were hard to come by with the Newport defence standing firm and Kane's assured touch for once letting him down.
- Robbie Willmott has assisted seven goals for Newport in all competitions this season, more than any other Exiles player.
Pochettino made another switch, replacing Llorente with Dele Alli, as the game moved into its final half an hour with his side still trailing in Wales.
"If the replay is anything like the second half then I'm not relishing it". "For me, sometimes I miss personally this kind of atmosphere and football". Moving to the London club was seen as the player's "imminent destiny".
"We kept it as we were, we just had to get them back focus again and let's be honest, it was the only mistake we made all day and we got punished".
Pochettino says his team just did not match the motivation and energy of Newport. "I hope we have learned today that we have to fight to win things. We need to be more motivated though".