Royle made an emotional return to Boundary Park after the sacking of John Sheridan but his homecoming was spoiled by goals from Rovers duo Charlie Barnett and Ian Moore.
Oldham also had Stefan Stam sent off for two bookings as Tranmere deservedly won the battle of play-off rivals.
Adding an extra twist, it was a sweet victory for visiting manager Ronnie Moore, who was sacked by the Latics three years ago.
Tranmere got off to a good start and carved out the first opening when Moore volleyed wide from a great position.
There was a worse miss on the quarter hour as fine work by Bas Savage and the excellent Barnett ended with Edrissa Sonko heading badly off target.
Sonko wasted another chance before Oldham hit the woodwork through Lee Hughes, who skipped round keeper Danny Coyne after chasing Deane Smalley's flick.
Hughes had been forced into a tight angle and was denied by the inside of the post, with Ben Chorley scrambling the ball to safety.
Savage and Chris Taylor went close from distance at either end, while the second half began with the game's first real save of note.
Oldham midfield man Danny Whitaker fired towards the bottom corner from 12 yards, only for Coyne to tip away at full stretch.
Tranmere, however, broke through on 56 minutes as full-back Ryan Shotton was given space to cross from near the right touchline.
Barnett arrived at the far post and hit an unstoppable first-time volley which flew into the top corner from ten yards.
Oldham should have been level within eight minutes but the off-colour Hughes blazed over after being set up by substitute Steve Kabba.
And their day went from bad to worse after Stam, already in the book for a foul, chopped down Barnett deep inside the box.
Referee Graham Laws had no choice but to send off the Dutch defender and award Tranmere a penalty, which Moore slotted into the bottom left-hand corner.
Oldham toiled in vain to get back into the match and are now on a run of three straight defeats.