The Georgia Supreme Court yesterday refused to reconsider last month's decision that would require Betty Talmadge to turn over more than $500,000 to her former husband, Sen. Herman E. Talmadge (D-Ga.).

Paul Howell, a lawyer for Mrs. Talmadge, said he would file another petition for rehearing with the court today.

Although rehearings are seldom granted, Howell noted that one of the Supreme Court justices, Thomas Marshall, had discussed the Talmadge case with a reporter for the Atlanta Consititution while the petition for reconsideration was before the court.

That could lead the justices to grant another hearing, Howell said.

The money at stake is the profit from an 8 percent interest in 1,175 acres along Interstate 75 near Atlanta. Talmadge invested about $119,000 in the land beginning in 1967 and the share was worth $756,000 when the land was sold in 1972.

The Talmadge investment was made in Mrs. Talmadge's name, although her husband provided the money. Mrs. Talmadge contends the property was a gift, but the Georgia Supreme Court ruled June 27 that she was only holding the property as a trust for Talmadge.

Talmadge listed the property as belonging to his wife on financial disclosure statements filed with the Senate. Mrs. Talmadge paid the capital gains tax when the property was sold. The couple filed separate returns for 1972, although they later filed an amended return jointly.

The Talmadge's were granted a divorce last year.