Los Angeles County's 55,000 employes have pulled out of the Social Security system, the largest defection so far from the troubled federal pension plan.

Local and state governments and nonprofit organizations have been taking employes out of the system since they won the right to do so in 1959, but Social Security Administration officials say increased payroll taxes and local government deficits have turned a trickle into a growing flood. "It could really botch us up royally," said Susan Dower, a program analyst for the National Commission on Social Security Reform which is struggling to save the system.

The trend also worries union leaders, who see the withdrawals as a callous way for local governments to save money on Social Security employer taxes they used to pay to the federal government while leaving their employes with no pensions. Employes of organizations that have left the system will be eligible for retirement benefits only if they paid Social Security taxes for about 10 years before they withdrew.

Supporters of the change said employes could now invest toward their retirement the money that used to be taken out of their checks for Social Security. But Ophelia McFadden, general manager of the 6,000-member Los Angeles County Employes Union, said, "Many of these are single women with children who are going to spend any extra money on bread and butter for the table."

Last year, Social Security Administration spokesman Jim Brown said more than 100 counties, cities, school systems, hospitals and other nonprofit organizations representing 172,000 employes withdrew from the program, an estimated loss of $200 million to $250 million annually. Another 387 local government entities representing 167,000 employes have informed Social Security they plan to drop out in the next two years, and officials warn that a panic could soon greatly inflate those numbers.

When Rep. J.J. (Jake) Pickle (D-Tex.) suggested legislation in 1981 to curtail the right of localities and nonprofit organizations to leave the system, many moved quickly to give Social Security the required two-year notice of withdrawal.

The National Commission on Social Security Reform, with its 15 members appointed by the president and congressional leaders to find solutions to the system's problems, has voted unanimously to recommend that no more employers be allowed to defect, according to Dower.

Some lawyers doubt that the federal government would have the constitutional right to keep state and local governments in the system against their will. But the threat could produce what Dower calls another "run on the bank," a panicky withdrawal by many more local governments and nonprofit groups.

About nine million people now paying Social Security taxes work for local and state governments and nonprofit organizations which could drop out if they chose. If all had left the system in 1981, Brown said, Social Security would have lost $18.2 billion in 1982.

Of 881 governmental entities that withdrew from the system from 1959 through 1981, Brown said, 606 left after 1975, when payroll taxes began to climb steeply with inflation. Another 204 local governments, including New York City, changed their minds after applying to withdraw in the 1975-1981 period, however. The governments of Virginia, Maryland and the District have stayed in the system, but some local government organizations have filed notice that they intend to withdraw. By the end of this year, 2,600 employes of Prince George's County are scheduled to leave the system and 11,000 employes of Fairfax County are scheduled to leave in 1984.