The goal came in the 32nd minute when Paul Warne side-footed home from close range.
The Swans showed four changes from the side beaten at Cheltenham and only a superb tackle by Will Haining in the fifth minute prevented Pawel Abbott from a clear run on goal as the home side pressed the visitors in the opening stages.
The Latics hit back and a cross from Andy Liddell was met at the far post by Warne but Willy Gueret blocked the header.
Chris Taylor beat Alan Tate down the left flank before cutting inside but his low drive went straight to Gueret.
It was end-to-end play and Ian Craney fired in a 30-yarder that Oldham keeper Les Pogliacomi had to tip around the post.
Midway through the half, Taylor skinned Tate once again but his shot drifted wide of the far post.
Swansea almost made the breakthrough in the 28th minute when Leon Britton found himself alone in the goalmouth but Taylor hacked the ball off the line.
The visitors deservedly broke the deadlock four minutes later when a Chris Porter shot came back off Gueret but Warne was on hand to slide it into the net.
Swansea made two changes at the interval in an effort to get back into the game with Adebayo Akinfenwa and Marcos Painter on for Tate and Abbott.
In the 53rd minute Lee Trundle was pulled back by Gary McDonald just outside the penalty area only for Andy Robinson's free-kick to strike the crossbar with the Oldham keeper beaten.
The Swansea changes made a difference and Trundle found himself free on the right but his cross was cleared for a corner by the impressive Haining.
The home side were almost presented with an equaliser when Pogliacomi and Sean Gregan collided in the penalty area but the loose ball ran wide of the far post.
As time slipped away, Robinson's volley was deflected wide and, when Akinfenwa won a free-kick just outside the area, Pogliacomi leaped across to gather the kick from Trundle.
The defeat now leaves Swansea a long way behind the division's frontrunners and an automatic promotion spot now appears beyond them.