Oldham are flying high in League One but the visitors from the Championship brought them down to earth as Potter opened the scoring and Craig Davies made sure of the win.
The sides had drawn 2-2 at Molineux last week and Oldham again gave Wolves a scare by bossing the first half.
But top-scorer Chris Porter missed a great chance and Wolves eventually made their quality tell as they earned a mouthwatering fourth round clash against local rivals West Bromwich Albion.
Oldham's football gained them an opening in the fifth minute which Porter drilled over the bar from the edge of the box.
And centre-back Stefan Stam found plenty of space in the Wolves penalty area midway through the half but saw his shot charged down by Gary Breen.
Home striker Paul Warne got on the end of three crosses despite the physical presence of the visiting defence but was unable to guide any of them on target.
And Oldham missed a clear chance just before half-time as Porter charged to the edge of the six-yard box as he tried to reach a teasing cross from Warne but his header lacked any sort of direction.
Wolves keeper Matt Murray saved from Will Haining on 54 minutes and Oldham thought they should have had a penalty for Potter's challenge on Richie Wellens.
Mick McCarthy's men, however, took the lead when Oldham could only clear a corner to Potter 25 yards out.
The former Liverpool midfielder unleashed a tremendous first-time shot with his right foot which gave Les Pogliacomi no hope as it flew into the top corner.
It was soon 2-0 as Rohan Ricketts crossed from the right-hand side and Davies climbed to get the final touch with a glancing header six yards out, although it was unclear who the scorer really was.
Wolves went on to dominate the closing stages and should have added more to their tally.
Davies brought a low save from Pogliacomi, who also kept out Ricketts following a powerful drive from 25 yards.