Bhagyaraj was born in Vellakovil near Gobichettipalayam in Erode district in Tamil Nadu. His parents were Krishnaswamy and Amaravathiamma. He married twice. His first marriage was with actress Praveena, in 1981. She died due to an aggravated case of jaundice in August 1983. His second marriage was with actress Poornima Jayaram on 7 February 1984. She was his co-star in Darling Darling Darling in 1983.The couple have a daughter Saranya Bhagyaraj and a son Shanthnu Bhagyaraj.