Landers speculates that people confused astrology with astronomy:
“Surely,” I said to myself, “it’s not that Americans believe astrology is scientific. Instead, they must be confusing astronomy with astrology, like I did those many years ago.”
He then did a small study of 99 respondents on Amazon’s MTurk to explore that possibility, paying each respondent 5 cents.
The first problem with Landers’ study is that he claimed to have used the same wording for the astrology-as-science question as in the NSF survey, but he made three wording changes: He added two responses “pretty scientific” and “not too scientific” to the response set for the astrology question and deleted the response “sort of scientific.” The NSF/GSS question—which he rewrote in three ways—had asked:
“Would you say that astrology is very scientific, sort of scientific, or not at all scientific?”
Second—and more important—he omitted the immediately prior question in the NSF study:
“Do you ever read a horoscope or your personal astrology report?”
This question clearly ties astrology to horoscopes and people don’t get personal astronomy reports. Accordingly, people should have been aware from the context of the question that astrology involves horoscopes.
Landers also asked his 99 respondents to define astrology before asking the other questions. He then interpreted some responses that astrology is a science or is the study of the stars as evidence that they had confused astrology with astronomy.
From Landers’ study here are some examples of supposed confusion of astronomy with astrology:
- Astrology is the study of the stars.
- The study of the stars.
- The study of the stars and constellations.
- It is the study of the stars.
- The study of the planets and stars.
- Astrology is the study of galaxies, stars and their movements.
- Astrology is the scientific study of stars and other celestial bodies.
But some followers of astrology think of their discipline as scientific, although any proper definition ties astrology to life on earth. Without at least probing further, Landers shouldn’t assume that someone who calls astrology “the study of the stars” is confusing the pseudo-science of astrology with the science of astronomy.
I decided to explore the same issue as Landers did with my own small MTurk/Qualtrics study, paying 51 cents. (It appears that I paid more than the market-clearing price because my data collection was complete in just 26 minutes.) I got 109 responses and excluded one respondent for the same reason that Landers excluded one respondent, the response was gibberish. Unlike Landers, I used both questions from the original NSF study, and I did not change the response set on the astrology-as-science question. My first four questions were:
1. Do you ever read a horoscope or your personal astrology report? [Yes/No]2. Would you say that astrology is very scientific, sort of scientific, or not at all scientific?3. In just a few words, please describe what “astrology” is?4. In just a few words, how might ordinary people who follow astrology gain information about what it might mean for their lives?
The responses were quite revealing. With the proper NSF/GSS setup tying astrology to horoscopes, only 11 of 108 respondents in my survey gave responses to the question on what astrology is that Landers probably would have classified as indicating astronomy, for example:
1. DEFN: “Astrology is the science of the stars.”
When I asked how people would gain information about astrology, this respondent answered:
INFO: “I am not really sure because I don’t really believe in horoscopes.”
Obviously, this person is not confusing astrology with astronomy.
Similarly, compare these definitions of astrology with the same respondent’s explanation of how to get information about astrology:
2. DEFN: “study of stars”
INFO: “it tells the future and your personality traits.”
3. DEFN: “I believe it’s the study of the stars. study of the alignment of the stars and planets.”
INFO: “Maybe the stars can tell the future?”
4. DEFN: “Astrology has to do with characterisitcs [sic] and stars”
INFO: “They might understand their personality or why things happen”
5. DEFN: “The study of stars and their formations.”
INFO: “It can help them cope with how they are feeling and make decisions.”
6. DEFN: “study of the alignment of the stars and planets.”
INFO: “astrology sometimes tells you what to look out for in the next month.”
7. DEFN: “Grasping for concepts in outer space”
INFO: “They are using astrology to gain information for what it means by searching for advice and support through hard times throughout their lives.”
Only two of 108 respondents gave responses to the open-ended questions that might indicate that they confused astronomy with astrology, and one of them answered that astrology was not at all scientific.
8. DEFN: “Astrolo[g]y is a study of star and planet alignment.”
INFO: “Ordinary people would have to pub [put] a lot of belief in information to believe in it. It is someone else opinion your relying on.” [also answered “Not at all scientific”]
While not entirely clear, the respondent considered astrology not at all scientific (which would be odd for someone who thought that astrology was astronomy) and she seems to be talking about not putting faith in the advice of other people (probably astrologers). So I classified this respondent as not confused in the way Landers contends is widespread.
The only one who reasonably might well have confused astrology with astronomy was this person:
9. DEFN: “astrology is the study of movements and the central body of the solar system. mainly how the univers[e] works and how does it change in time. the movement of the stars, moon and other planets not to mention the lunar and solar eclipse the milky way.”
INFO: “they can search websites and some documentations in the library about what they want to know about astrology some say it can help you realize who and what is [sic] you are in this universe.”
Even though this respondent says that “some say it can help you realize who and what . . . you are in this universe,” which sounds like he is talking about astrology, he may well be just talking about the wonders of science and the universe. I say this mostly because he was one of only six respondents who believed that astrology was “Very scientific.” So I’ll resolve the reasonable doubt in this case in the direction of Landers’ hypothesis. But even this one case of probable confusion is at least doubtful.
I followed up Landers’ small study with one of my own, trying to correct for some of the deficiencies in his study. In my sample of 108 respondents, the evidence suggests that probably only one of the 108 respondents was confusing astronomy for astrology. Given that, I would say that there is no reason to believe that the results of the NSF/GSS study reflect a substantial conflation of astrology with astronomy.
It is indeed ironic that in a study designed to probe question wording, Professor Landers could have made such a serious error himself by interpreting statements such as “Astrology is the study of the stars” as if they indicated a confusion of astrology and astronomy. Such a statement is indeed inadequate as a definition of astrology, which is why it’s a good idea to follow up before jumping to unwarranted conclusions. To avoid misinterpreting the answers he received, Landers needed both to probe further and to test the actual questions used on the NSF survey.
As my follow-up study suggests, a statement that astrology is “the study of the stars” is quite consistent with an understanding that astrology is about horoscopes and predicting the future. Given that the NSF/GSS study explicitly tied astrology to horoscopes, it would be surprising if a substantial number of respondents conflated astrology with the science of astronomy. Thus, from the evidence of my study, it is extremely unlikely that a substantial number of respondents in the NSF/GSS study failed to understand that they were answering a question about astrology, not astronomy.
I do want to note that I have come around to believing that the NSF wording is less than ideal in a different sense. By calling astrology “sort of scientific” the respondents may be describing the pseudo-scientific trappings of the field, rather than expressing an opinion of its possible soundness.
Last, I don’t want to be too hard on Professor Landers. His instincts were right to want to test his hypothesis, which appeared plausible if you didn’t know about the immediately prior question tying astrology to horoscopes. In comments on his blog, however, Landers has dug in his heels, asserting without any evidence whatsoever that using the actual NSF questions wouldn’t have made a difference. Yet overall Landers’ NeoAcademic blog is a good one, and we all make mistakes. Certainly, I do.
For more on which groups think that astrology is unscientific, see my brief report at SSRN.