The visitors went down to ten men with just nine minutes on the clock when Dean Hammond was shown a straight red card for his two-footed tackle on Deane Smalley.
It was an uphill task for Brighton, but they battled hard to stay in the game after Oldham went in front before half-time.
Oldham were FA Cup giant-killers last week when they beat Premier League side Everton away from home, but they missed chances for three league points and were punished in injury time.
Oldham created almost all the openings, starting when Craig Davies drilled over the bar and last week's match-winner Gary McDonald headed wide.
Brighton had already lost Hammond when Davies fired straight at defender Wes Fogden when he could have done better.
Oldham's breakthrough came on 40 minutes when Kelvin Lomax passed to Lee Hughes, who made progress into the box.
Hughes held off his marker and beat keeper Michels Kuipers from close-range to register his eighth goal of the season.
The Latics should have kicked on but McDonald was denied before Brighton went close with a snap-shot from Nick Forster.
The home side could not find a way to double their lead and midfielder Jean-Paul Kalala was next to be thwarted after he led a counter-attack and exchanged passes with Davies.
There were further openings for defenders Reuben Hazell and Lomax, but Brighton kept their heads and finally carved out a chance deep into stoppage time.
Oldham's defence failed to clear a corner, Nicky Forster hit the bar and the loose ball was rammed home from eight yards by Elder, who had only been on the pitch since the 87th minute.