The Debate On The Status Of Data

Published Date: 02 Nov 2017

Qualitative research is a method of inquiry employed traditionally in the social sciences (e.g. psychology), that aims to gather an in-depth understanding of human behaviour and the reasons governing these behaviours (Denzin & Lincoln, 2011). It has since spread across many academic disciplines and even in market research. Typically, psychologists have sought to collect quantitative (numerical) information that can be statistically analysed in their pursuit to establish the study of Psychology as a scientific, albeit social, discipline. However, an increasing number of researchers have begun to collect other forms of data that can be analysed qualitatively in which the issues of meanings are richly explored and studied. One example of a qualitative research question might be "understanding how the meaning of ‘love’ changes as a relationship progresses".

Nevertheless, the qualitative research field has engendered an endless list of criticisms and ridicule from the academic community such as the lack of rigour (Sandelowski, 1986), the lack of validity (Cho & Trent, 2006), and even the inconsistent definitions of certain terminologies (such as the definition of the status of data; Speer, 2002). In addition, literature regarding qualitative research has focused so much on the different methods of analysis (such as Grounded Theory, Discourse Analysis, etc.) to the extent that collected data, in its entirety, has been saliently overlooked. In lieu of the fact that data collected from qualitative research may be completely different from data collected during a quantitative research, this gap in the available literature serves as one of the biggest sustaining criticisms regarding the rigour (or reliability) and validity of qualitative research findings.

This essay thus seeks to, humbly; fill in the gap by providing insights regarding the on-going debate pertaining to the terminologies used describing the status of collected data in qualitative research: "Natural", "Contrived" and "Naturalistic" data. In addition, it attempts to highlight the controversies and limitations surrounding ‘naturalistic data’.

Conversation Analysts versus Discursive Psychologists

One of the most consistent and troubling findings of qualitative research is that the context of the research project (i.e. the circumstances that form the setting for the research which can be understood and fully assessed) has a significant effect on participants’ behaviour (Banyard & Hunt, 2000). These circumstances may affect the overall quality of the data collected. Because of the nature of the data collection techniques, researchers worry that numerous potential sources of ‘bias’ may lead to invalid and erroneous results such as ‘the Hawthorne effect’, ‘reactivity’, and so on (Hammersley & Gomm, 1997). A good example would be the classic Hawthorne Works research conducted from 1924 to 1932 where the purpose of the study was to see if workers would become more productive in higher or lower lighting. However, researchers found that worker productivity increased despite dim working conditions, and these results were due to the presence of the research team rather than the actual experimental variable (i.e. the intensity of lighting). This implied that the data collected from the research was invalid and completely ‘contaminated’ because the presence of the researchers interacted with the environment of the experiment. This has led to the awareness and necessity of eliminating such biases and context effects from qualitative research. Conversation analysts and discursive psychologists have hence advocated a distinction between: (i) ‘natural’, ‘naturally occurring’ or ‘naturalistic’ data; and (ii) ‘contrived’, ‘researcher-provoked’ or ‘artificial’ data (Speer, 2002).

Conversation analysts have always conveyed a preference for working with ‘naturally occurring’ data such that the term has become a slogan constructed into many definitions of the conversation analysis (CA) approach. Hutchby and Wooffitt (1998), for example, states that CA is "the study of recorded, naturally occurring talk in interaction"; while Psathas (1995) notes that ‘data may be obtained from any available source, the only requirement being that these should be naturally occurring’. A variety of terms have also emerged and been used interchangeably, with references to ‘naturally occurring talk’ such as ‘natural interaction’, ‘natural conversation’ and ‘natural human interaction’.

The word ‘natural’ in these cases is implicitly and explicitly compared with data that are ‘contrived’ or ‘researcher-provoked’. Simply put, natural data is data that must not have been produced for the purpose of study or collected for any pre-formulated investigative research purposes (Drew, 1989). Such definition then begs the question, "Are data collected from focus groups or interviews considered natural data?" This question adequately addresses the fundamental issue of ‘naturally occurring’ or ‘natural’ data for recordings from interviews and focus groups can be considered as co-produced. After all, a researcher has to be present to ask questions which in itself, is not entirely ‘natural’. Conversation analysts answering this question will hence argue that recordings of focus groups or interviews can still be considered ‘natural’ unless the interviewer deliberately asks leading questions, or that participants’ are treated as ‘appropriate substitutes’ for reports of actual events.

While discursive psychologists have philosophical roots in social constructionism, they generally conceive the natural and contrived distinction in strikingly similar terms as conversation analysts. For example, Potter (1997) argues that ‘naturally occurring talk’ is talk produced entirely independent of the actions of the researcher’. In other words, the interaction that was recorded has to pass the ‘dead social scientist test’ to be considered as ‘natural’ data: the interaction must have taken place even if the researcher got run over on the way (Potter, 2004). To answer the previous question, interviews and focus groups are hence not considered as ‘natural’ data due to the fact that an interview would not take place without the researcher there to ask questions; while a counselling session would still happen whether the researcher turns up to collect the recording or not. From this perspective, mundane talk occurring in the classrooms, conversations between friends, courtroom trials, and business meetings are all considered ‘natural’.

This distinction differs from conversation analysts in that discursive psychologists view the interview method as contrived and not natural. ‘Contrived’ data are considered to be data that have been ‘got up’ by the researcher such as those collected in an interview, an experiment, or a survey questionnaire. Potter (1996) has critically pointed out that the interview is subjected to powerful expectations by participants where the interaction with the interviewing researcher modifies the context such that it is no longer completely natural; and there are particular difficulties in extrapolating from interview talk to activities in other settings. Instead, he suggests that the term ‘naturalistic data’ should be used to describe such recordings as interviews can be conducted in naturalistic settings, which are mainly adjunct to naturalistic studies. Discursive psychologists thus propose to look at the natural and contrived distinction on a continuum instead of definite terms. At one end of the continuum, there is research with a high level of researcher involvement (i.e., the researcher is a complete participant), and at the other end there is ‘completely natural’ research, where interaction is accomplished with no researcher involvement.

To summarize, although both approaches prefer natural data, conversation analysts consider recordings as ‘natural data’ only if they are as uncontaminated as possible by researcher intervention. Conversation analysts hence regard the terminologies regarding the status of "natural" and "naturalistic" data as interchangeable. Conversely, discursive psychologists argue that the "natural" data lies on one end of continuum, "contrived" data on the other, and any data in between this continuum is considered as "naturalistic". Therefore for both discourse and conversation analysts, the ‘naturalness’ of data can be decided on the basis of the method used to collect it, and more specifically, the degree of researcher involvement or social scientific ‘intervention’ within it.

Implications and Limitations

Based on the two distinctive viewpoints above, it is imperative to note that one of the biggest discrepancies in present available literature regarding qualitative data is that the terms ‘natural data’ and ‘naturalistic data’ have either been used interchangeably, or argued that a distinction exists between the two. Researchers have not come to an agreement on these two standpoints such that textbooks written or edited by different authors offer different definitions of "naturalistic data", giving the impression terms such as "natural interaction" or "naturally occurring interaction" are the same (e.g. Forrester, 2010). In his introductory textbook on qualitative research, Silverman (2001) uses the terms ‘naturally occurring talk’, ‘naturally occurring settings’ and ‘naturally occurring situations’ interchangeably and on the same page (p.159). This lack of agreement is crippling because these textbooks are used by novice researchers and even undergraduate students to learn about the unique methodologies in qualitative research – inducing confusing conceptual resources for making decisions about the most "appropriate" data sources for their research questions (Lynch, 2002). In addition, these inconsistencies may have exacerbated the on-going debate such that attempts to impose consistency on ordinary language have had little to no success.

At the same time, several researchers warn against drawing hard and fast boundaries in social interaction research, and assuming that certain types of ‘talk’ are more legitimate or ‘natural’ to study than others. Schegloff (1989), for example, argues that labelling and announcing an occasion of talk-in-interaction as an interview does not, in fact, make it one; neither does it guarantee that what began as a ‘natural’ interview will remain as one throughout the course of the study. The problem with the natural, naturalistic, and contrived distinction is that the method is treated as a means to get the data, in which the data are then viewed as an effect of the way they were collected, rather than constituting the method or data collection technique. In other words, data collection method and the data collected are seen as completely distinctive entities as compared to the data collected making up part of the data collection technique. This cause-effect model seems peculiar in the context of a research field that spends much of its time criticizing such frameworks as both deterministic and simplistic (Speer, 2002).

Therefore the relationship between the method (be it an ‘interview’ or ‘naturalistic observation’) and the ‘type’ of data collected (be they ‘natural’ or ‘contrived’) may have been embellished in discursive and conversation analyst studies. As researchers, we need to be careful not to reach premature conclusions about the effects of any particular research methodology on our data – especially given that our data collection method is just one of an infinite number of potentially relevant contexts that may be made relevant in an interaction.

Conclusion

At this point, it should be clear that the notions of what is (i) ‘naturally occurring’, ‘natural’ or ‘naturalistic’; and (ii) what is ‘non-naturally occurring’, ‘researcher provoked’, or ‘contrived’ are being used in ill-defined and theoretically inconsistent ways. Although incompatible uses of the same term can be a source of debate and confusion, efforts to impose consistency on ordinary language may have ironically compounded the confusion. In fact, to further complicate matters, the natural and contrived distinction is often conflated with and mapped onto the distinction between mundane and institutional talk, both of which are ‘naturally occurring’ in the sense that they have not been ‘got up’ by the researcher (Drew and Heritage, 1992).

All these inconsistencies thus indicate that discursive psychologists and conversation analysts have not thought about the ‘naturalistic’ conceptualization so much as reproducing it. ‘Naturalistic’ has become a term with ambiguous boundaries and little in the way of specificity, which will in turn, only contribute to the confusion of those new to the field and even those already working within it! Qualitative researchers should thus seek to attain a more sophisticated understanding of the relationship between method, context, and data if we are to make informed choices about the data we use, and – perhaps more importantly – produce theoretically sound justifications for those choices. The umbrella topic of ‘conducting a qualitative research’ will then be clearer, richer and more fulfilling.