The Uses and Limitations of Free Listing in Ethnographic Research
Lance Gravlee, Dept. of Anthropology, University of Florida
Free listing is one of several structured interviewing techniques designed to elicit systematic data about a cultural domain. A cultural domain may be defined as "an organized set of words, concepts, or sentences, all on the same level of contrast, that jointly refer to a single conceptual sphere" (Weller and Romney 1988:9). Many of the emic mental phenomena that interest anthropologists can be conceived of in terms of cultural domains. For example, some of the domains that anthropologists have studied include kinship terms, diseases, child-rearing practices, plant terms, and types of home remedies.
There are generally two things anthropologists want to know about a cultural domain: (1) what belongs in it? and (2) how are its contents structured? Free listing, in which the interviewer simply asks each informant to "list as many X's" as he or she can, is specifically designed to answer the first question: What are the contents and boundaries of the domain being studied? But free lists also contain information about how people perceive the relationships among items in a domain, and therefore can be used to answer the second question as well. In this essay, I will briefly discuss the uses and limitations of free listing to answer both types of questions. In addition, I will present an example from my own research to illustrate the use of free listing to compare cultural domains.
The first step in any study is to define the contents and boundaries of the subject matter. This general principle applies not only to cultural domain analysis, but to any type of research.
One of the hallmarks of ethnographic research is the commitment to defining the subject of study in emic terms—using the language, concepts, and categories of the people being studied. The importance of this commitment makes free listing an essential tool for any ethnographer, not just cognitive anthropologists or ethnoscientists. Asking a set of informants to "name all the X's you know" provides a systematic way of eliciting the emic concepts and categories that belong in a domain. In short, it is the best way to ensure that the anthropologist is dealing with culturally relevant items (Bernard et al. 1986).
The first question many researchers are likely to ask about free listing is, "How many informants do I need?" The best answer is that it depends on the coherence of the domain. As Borgatti (1998) notes, a single informant would do if everyone agreed on the contents of the entire domain. Since few domains are this coherent, the best strategy is to keep track of how the frequency distribution of terms changes as new informants are added.
For instance, suppose we want to collect free lists on "things you take to the beach." We would want to monitor the number and relative order of items listed as the number of informants increased. If few new terms were added and the relative order of items didn't change from 10 to 15 and again from 15 to 20 informants, it would be safe to assume that 20 is an adequate sample size for this domain. For most coherent domains, it turns out that 20 to 30 informants are in fact sufficient (Weller and Romney 1988). But there is no substitute for determining sample size by monitoring the stability of the domain as the number of informants increases.
The most important result from free listing is the frequency with which each term is mentioned across all informants. For most domains, one of the first things to notice from the frequency results is that many of the items are listed only once. There appears to be a sort of core-periphery structure to most domains, so that a small set of core items will be mentioned by most informants, while a much larger set of peripheral items will be listed by only a single informant.
My own research on the domain of ethnic categories in the US, for example, produced a total of 100 terms. Only nine terms were listed by at least 10 of the 35 respondents; 66 terms were listed by a single informant. The balance between core and periphery terms in any given domain depends on the domain's coherence, but it is generally true that a large proportion of the terms mentioned in a free list exercise are idiosyncratic.
For purposes of defining the domain, it is important to exclude these terms, since, by definition, cultural domains are shared. It usually makes sense also to exclude terms that are listed by only a few informants. One approach to reducing the number of items in a domain is to look for a natural break in the frequency distribution. Such a break might reflect the boundary between terms that belong to the culturally shared domain and terms that are idiosyncratic.
Setting the domain boundary is important in its own right for determining the final set of culturally relevant terms. But it also has the practical advantage of reducing the number of items to a level suitable for pile sorts, triad tests, rankings, or other types of data collection. Most researchers who use free lists use them for just this purpose: identifying the terms that should be included in later parts of the study. After all, that's what free lists do best. But there is also a lot to learn from free lists themselves. The next section will address this point.
Psychologists and anthropologists have derived a number of important measures from free lists (Brewer 1994; Robbins 1997). Most of these measures depend on two results: the frequency and order in which items are mentioned in the free listing exercise.
Fifty years ago, Bousfield and Barclay (1950) hypothesized that the order in which a term appeared in an individual's list would be related to the frequency with which it occured in the aggregate. They were right. Their results showed that, on average, the closer to the beginning of an individual's list a term appeared, the more often it would be listed across individuals. In other words, mean rank order is negatively associated with frequency.
Later researchers suspected that this relationship holds because both order and frequency say something about the cultural or psychological salience of a given term. Romney and D'Andrade (1964) made this assumption explicit in a study of American kinship terms when they proposed an item's mean rank across lists as a measure of the item's salience. Smith (1993) later pointed out that the problem with taking mean rank as a measure of salience is that not all items will show up in all lists. If a term is mentioned only once but appears at the top of an informant's list, its mean rank would be one—not a useful measure of salience.
Smith's solution was to use both the frequency and order measures to calculate a gross mean percentile rank for each item across all lists. For a given informant, the percentile rank of an item A is calculated by the formula:
The average percentile rank of an item across all lists is the item's gross mean percentile rank—its salience index. This measure takes into account the open-ended nature of free listing, and it incorporates both how often and how early items occur in informants' lists. Even better, recent versions of ANTHROPAC calculate the salience index automatically for free list data (Borgatti 1992). Salience is a useful measure because it indicates which concepts or categories should be given the most attention in later parts of a study. It also sets up a theoretically important question: Why are some items more salient than others?
Another set of measures that can be derived from free lists provides information about the categorization principles for a given domain. In a study of animal terms, Henley (1969) discovered a clustering effect in the order of occurrence of items in a free list. First, Henley asked 21 adults to list as many animals as they could. Then she examined the lists to find the average distance among 66 pairs of animals like cat-dog, cat-cow, bear-lion, and bear- pig.
The lowest average distance, 1.8, was that between goat and sheep; the highest, 56.1, was that between cat and deer. Henley concluded that, the closer any two items occurred in the free lists, the more related they were perceived to be. The relationship between interitem clustering and perceived similarity has also been demonstrated for other domains (Romney et al. 1993; Sanday 1968).
Another approach to domain structure is to consider the co-occurrence among items in a domain. Borgatti (1998) describes a method for creating an item-by-item matrix in which the cells indicate how often informants listed both the row item and the column item. This matrix can be represented graphically with multidimensional scaling (MDS), which usually reveals the same core-periphery structure mentioned earlier. Items that occur frequently in the same list—the core items—appear at the center of the graph, with peripheral members radiating out from the core.
There is some evidence of a relationship between the core items and the concept of prototypes. Rosch et al. (1976) demonstrated that people are more likely to list items with high prototypicality ratings when asked spontaneously to give examples of a category. These core items, or prototypes, would appear at the center of the MDS graph of co-occurrences.
I should be careful to point out that free listing is not a cure-all. If the real question is how people perceive the similarity among items in a domain, then it is best not to rely on free listing for an answer. More direct techniques using pile sorts, paired comparisons, or triad tests provide better data about the structure of domains. The utility of free listing for defining a domain also can be limited by the difficulty of finding appropriate superordinate categories. In some cases this difficulty can be overcome by eliciting domain labels from unstructured interviews; in other cases it might suggest that the domain simply doesn't exist. Finally, the fact that free listing is an open-ended task creates problems when trying to compare responses across individuals.
The major application of free listing is to define the contents of a cultural domain. But suppose we are interested in the contents of two related domains and would like to know how they are distinguished. One possibility is to collect free lists for each of the domains and compare their contents for similarities and differences. In this section, I will present an example from my own research that takes this approach.
In 1996, I conducted a small study of ethnic and racial categories in the US. Initially, I set out to study ethnic categories, but as I began talking to people I soon realized that the concept of "race" was undeniably important from an emic perspective. I decided to conceive the problem in terms of two tentative cultural domains: I asked one sample of university students to list all the "ethnic groups or categories" they could, and another to list all the "racial groups or categories" they could. The relevant question about the two domains was whether there was really any difference between them, or whether the two sets of free lists would produce the same results.
As it turned out, it didn't matter which free list question I asked; the answers were more or less the same. There are several ways to compare the contents of two cultural domains. The most obvious comparison is a simple count of the terms that appear in both domains. It turned out that 77 of the 143 total terms were mentioned in only one of the domains. But 70 of the 77 were mentioned by only a single informant. In other words, if a term was salient enough to be mentioned more than once, chances were that it would be listed in both domains.
There are also systematic ways to compare the frequency and rank order distributions of two domains. Figure 1 shows a scatterplot of the frequencies for items that were listed at least twice in each domain, along with the linear regression line and equation. Two things are noteworthy. The first is the fairly strong Pearson's correlation, r = .72, which suggests that there is a strong positive association between the frequency distributions of the two domains. The more often a term was listed as a racial category, the more likely it was to be listed as an ethnic category, too.
The second thing to note is that, despite overall similarity, there are some exceptions to the agreement between domains. The most striking case is the term "African-American," which was listed far more often as an ethnic rather than racial category. By contrast, the term "Black" was mentioned more often as a racial category. This pattern draws our attention to the different meaning of alternative labels for the same category, and we might want to follow up on this issue in more in-depth interviews.
Another way to compare the two sets of free lists is to look at the similarity among individual lists. We can do this by pooling together all the lists and creating a list-by-list similarity matrix, in which the cells represent the proportion of items that any two lists shared in common. Then, we can use multidimensional scaling (MDS) to look for pattern in the new similarity matrix.
Figure 2 shows the results of this analysis. The points in this figure represent individual free lists. The distance between points reflects similarity among the lists, such that points close together represent lists with many items in common. If racial and ethnic categories belonged to two distinct domains, we would expect to see two groups of lists corresponding to the two domains. In fact, we see the opposite. There is no discernible difference in the similarity among respondents to the two free listing tasks, suggesting that the contents of the two domains are essentially the same.
Finally, we can use Spearman's rank-order correlation to compare the relative rankings of the items in each of the domains. Including only the 33 items listed by three or more informants in at least one of the domains, this analysis produces a rank-order correlation of .67. Thus, in terms of frequency, content, and rank order distribution, the domains of "ethnic categories" and "racial categories" appear to be overwhelmingly similar. This conclusion, based on analysis of the free lists, is important for understanding the emics of ethnicity in the US.
Free listing is a deceptively simple but powerful tool. Its most important application is the identification of culturally relevant items and the definition of domain boundaries. Free listing is thus a simple technique for eliciting emic concepts and categories; as such it belongs in every ethnographer's toolkit.
The identification of a cultural domain and its contents is especially important to ensure the validity of other data collection techniques, including both structured and unstructured interviewing techniques. But there is also a lot to learn from free listing itself. The relative order and frequency distributions are the basis for measures of an item's cultural salience and the domain's categorization principles. There is also evidence that the cognitive processes involved in free listing are linked to other cognitive effects, notably to rankings of prototypicality.
At the same time, free listing is not the one-stop-shopping option for cultural domain analysis. If the central question concerns prototypicality effects or the perceived similarity among items, other techniques are more appropriate. One novel use of free listing is the comparison of related domains when the emic distinction between concepts is not clear. We can evaluate the similarity of domains by comparing the presence of items in each domain, the association between frequency distributions, and the association between rank orders. This analysis can provide significant information about the relationship among salient emic concepts.
Bernard, H.R., et al. 1986. "The Construction of Primary Data in Cultural Anthropology." Current Anthropology 27(4):382-395.
Borgatti, S.P. 1992. ANTHROPAC 4.0 Methods Guide. Columbia: Analytic Technologies.
Borgatti, S.P. 1998. "Elicitation Techniques for Cultural Domain Analysis." in Ethnographer's Toolkit, edited by J. Schensul. Newbury Park: Sage.
Bousfield, W.A.and W.D. Barclay. 1950. "The Relationship Between Order and Frequency of Occurrence of Restricted Associative Responses." Journal of Experimental Psychology
Brewer, D. 1994. "Patterns of the Recall of Persons in a Student Community." Social Networks 15:335-359.
Henley, N.M. 1969. "A Psychological Study of the Semantics of Animal Terms." Journal of Verbal Learning and Verbal Behavior 8:176-184.
Robbins, M.C.and J.M. Nolan. 1997. "A Measure of Dichotomous Category Bias in Free Listing Tasks." Cultural Anthropology Methods 9(3):8-12.
Romney, A.K.and R.G. D'Andrade. 1964. "Cognitive Aspects of English Kin Terms." American Anthropologist 66(3, Part 2):146-170.
Romney, A.K., D.D. Brewer, and W.H. Batchelder. 1993. "Predicting Clustering from Semantic Structure." Psychological Bulletin 4(1):28-34.
Rosch, E., C. Simpson, and R.S. Miller. 1976. "Structural Bases of Typicality Effects." Journal of Experimental Psychology: Human Perception and Performance 2:491-502.
Sanday, P.R. 1968. "The 'Psychological Reality' of American English Kinship Terms: An Information-Processing Approach." American Anthropologist 70:508-523.
Smith, J.J. 1993. "Using ANTHROPAC 3.5 and a Spreadsheet to Compute a Free List Salience Index." Cultural Anthropology Methods 5(3):1-3.
Weller, S.C.and A.K. Romney. 1988. Systematic Data Collection. Newbury Park, CA: Sage.