The Justice Department decided in January to set up a $4 million school safety center -- and a close friend of White House counselor Edwin Meese III agreed to head it -- even before the department had received a formal application for the grant, a Senate hearing was told yesterday.

In another case, the Justice Department in February approved an $800,000 grant for a study of the effects of magazines like Playboy, Penthouse and Hustler on juvenile delinquency even though a department analyst had advised that the study could be done for $60,000 or less.

Those contracts and several others came under scrutiny yesterday by a Senate Judiciary subcommittee hearing at which Sens. Arlen Specter (R-Pa.) and Howard M. Metzenbaum (D-Ohio) questioned why the majority of grants from the department's Office of Juvenile Justice and Delinquency Prevention were awarded without competitive bidding.

Alfred Regnery, administrator of the office, said noncompetitive bids got 57 percent, or $19.3 million, of the money awarded since he took over in November, 1982. He said competitive bidding on a grant can take up to six months and cost as much as $10,000.

"You're spending the government's money," said Specter, chairman of the juvenile justice subcommittee. "The Congress has said we want competitive bidding."

Metzenbaum questioned Regnery about what he called the "highly unusual" award -- of $2 million in the first year with a $2 million renewal optional -- to George Nicholson, who once worked with Meese in the Alameda County, Calif., district attorney's office.

Nicholson earns $65,000 a year as head of the National School Safety Center at Pepperdine University. Meese and President Reagan are associates of the school.

Metzenbaum read a Jan. 3 memo from the Cabinet Council on Human Resources that said the Justice Department intended to set up the center and Nicholson "has agreed" to direct it. On Jan. 7, Reagan announced creation of the center in a nationally broadcast radio address. Nicholson did not formally apply for the grant until Jan. 9.

Specter and Metzenbaum also questioned the $800,000 grant for a pornography study to Judith Reisman, who said she is examining the influence of the increasing depiction of children as sex objects in sexually explicit magazines. She has been hired temporarily by American University to carry out the study.

Regnery testified that the grant was awarded to Reisman after his deputy heard her last year on a radio talk show and allowed agency staff members help her write a grant proposal.

American University President Richard Berenzden testified he got telephone calls praising Reisman, who had applied for a research opening at his university, from Regnery's deputy and from Bruce Chapman, an aide to Meese at the White House.

Reisman testified that although she does not have an undergraduate degree, she received a doctorate in communications in 1980 and had previously worked for the "Captain Kangaroo" television series.