Retributions  

If you're new here, you may want to subscribe to my RSS feed. Thanks for visiting!

In its survey on Karnataka polls, CNN-IBN has predicted a Congress majority with major losses for both B.J.P as well as the JD (S). In his post discussing the CNN-IBN results, Offstumped has questioned the appropriateness of the survey methodology. In his critique, he has raised two questions: a) Problems with systematic random sampling b) Low response rate and if that adversely affects the survey finding. 

Ideally, a researcher would like to survey the entire population to be absolutely sure of his finding. However, in case of large populations, it is simply not possible for the entire population to be surveyed. Hence, a sample, supposedly representative of the population is selected and it is hoped that findings from this sample can be extrapolated to the population within an acceptable margin of error. 

Before coming to CNN-IBN’s survey, Offstumped has a minor error in his analysis. Discussing Probability Proportionate to Size (PPS), a version of Cluster analysis used by CNN-IBN, he writes, 

The bottomline is when CNN-IBN picked the 75 assembly seats for the survey from a total of 224 it did so by giving smaller constituencies a higher weightage of being selected when compared to bigger constituencies so that when ultimately sampled there is no bias based on size on who gets selected to participate in the survey.

Actually, it’s the opposite. PPS is employed in Cluster sampling when population within different clusters are heterogenous in terms of size. Therefore, by lowering the probability of a small cluster being selected ensures that an individual household within two different clusters of varying population size would have the same probability of being selected. Otherwise, an individual in a smaller cluster would have a greater probability of being part of the sample than an individual in a larger cluster thus introducing potential bias. 

Now. let’s look at CNN-IBN methodology detailed here. 

a) Prima facie, the methods used are sound as both PPS and systematic random sampling are standard industry techniques appropriate for surveys of this nature. 

b) As Offstumped points out, systematic random sampling may introduce some potential biases. However, no sampling method in entirely free of error. Each has its own strengths and weaknesses and which method is used depends on a host of factors including study design, population, cost and time constraints e.t.c. The use of systematic random sampling, in a survey of this nature, is appropriate. 

c) CNN-IBN claims a response rate of 50%. For a face-to-face interview, a 50% response rate would be considered an achievement. No survey achieves a 100% response rate and surveys are designed or should be designed keeping non-response in mind. However, it is equally true that if not accounted for, non-response may bias the final results. 

Bottomline: It’s the fine print, stupid! CNN-IBN methodlogy’s is simply not detailed enough to pass a judgment on its appropriateness. To give just one example, was a self-completed questionnaire used or the interviewers asked the questions? In the case of the latter, considering the survey used student interviewers, there is a high chance of introducing interviewer bias. Similarly, while the methodology talks about using PPS, it hasn’t specified the basis on which clusters were decided. Was it geographical location, caste or religion? Needless to say, they all may impact the result differently. 

Now, let’s turn to a broader issue: Do pre-poll surveys serve any purpose? Should they be banned? There is some evidence that opinion polls impact voter behavior more specifically through what has been termed the bandwagon effect when voters favor parties which are ahead in opinion polls. Potentially, a deliberately biased survey may affect election results. 

The CNN-IBN survey has been conducted by Dr Yogendra Yadav and his team at CSDS. Dr Yadav’s ideological biases are well know. To be fair, it must be pointed out that it would be difficult to find a social scientist without any ideological leanings of one kind or another. However, when engaged in research it is expected that those biases would be kept aside and the study would be conducted to advance human knowledge and understanding. How much of that admittedly ideal goal is translated into reality is entirely up to individual researchers. Social science research is also notoriously liable for manipulation; how the dataset is constructed and which statistical tools are employed has a huge impact on final results. 

Nevertheless, banning polls is no solution. Those who advance the agenda of choice–the market– must attempt to seek redressal in the multiplicity of surveys and hope any potential biases would be balanced out. Equally, it is quite likely that a researcher who deliberately manipulates data to advance his personal agenda would be ultimately discredited. 

Nevertheless, a more transparent system would help. The law can amended to ensure that all polls must explain their survey methodology in greater detail. For example, in case of CNN-IBN, it is not enough to state that they used PPS; the methodology should be explained in sufficient detail so that anyone so inclined would be able to replicate the survey. The polling companies may not fear greater competition; not everyone has the resources to attempt interviewing 5000-10000 people. It would simply allow interested parties an opportunity to attempt a dispassionate analysis without being hampered by a lack of data. It would also help decide if errors in predicting final results are the result of flawed methodology, deliberate bias or simply the heterogenous and fickle nature of Indian voter which makes guessing game, which ultimately all opinion polls are,  a difficult exercise.