What are we really measuring?

What are we really measuring?


Oct 07, 2014

Share It

As Sample Manager at iModerate, I oversee quotas for every project that we field. The most common quotas are focused on age, gender, region, and ethnicity; all important demographics, but perhaps not comprehensive enough. Admittedly, there is not one demographic variable that will capture the rich mixture of factors that influence what, how, and why people think and behave the way they do. Gender, age, region, ethnicity, income, education, religion, sexual orientation, political viewpoint, and so on all contribute to the messy conglomeration that make humans such complex creatures (and makes doing research on/with humans tricky). We don’t want to let our own biases – many times subconscious and culturally based – cause us to lose focus of important variables by putting too much emphasis on others.

A while back I wrote a blog teasing apart some important distinctions between two terms, race and ethnicity, we mistakenly use interchangeably. While I don’t want to completely rehash what was previously written, it is critical to remember that if we talk about race, we are implying some sort of biological essential and unchangeable characteristics about a group of people. Skin color, or any genetic trait for that matter, does not pre-determine the outcome of the cultural characteristics (i.e. specific shared behavior, perceptions, beliefs, language, knowledge, etc.) of an individual. For this reason, using race as a category to analyze social group similarities and/or differences among or between people’s perceptions, opinions, experiences, and identity is not useful. What would be more effective would be to focus on ethnicity: that is, cultural similarities based on a shared background.

Whilst making sure that we’re focused on ethnicity as a way to get a good picture of diversity within demographic groups, it isn’t the only demographic variable that affects how one experiences and perceives their world. For various historical and political reasons that I don’t have the time to get into, Americans tend to avoid discussions of the effects of socioeconomic class. As a result, other categories of difference (e.g. ethnicity or race) could be given more weight at the expense of this variable. There is a debate in the socio-cultural research on health outcomes (e.g. asthma)[1] and education as to whether socioeconomic class is a better predictor of outcomes than ethnicity. In a 2012 National Public Radio interview, researchers explained that despite closing the black-white education gap over the past 50 years, the gap between rich and poor students has been increasing[2]. They found the experiences of poor students and their parents to be similar, regardless of ethnicity.

Although complex, it is worthwhile to pay attention to these debates and how they may relate to the ability to more accurately predict marketplace behavior[3]. Mirroring studies on the relationship of income and ethnicity to health and education outcomes, many of the differences in consumer behavior that appear between different ethnic groups disappear when income/socioeconomic class are factored in[4]. Could the differences we’re seeing between ethnicity groups be only skin deep?

This brings me back to the importance of not giving too much weight to one specific variable, at least not until its significance has been fully tested. While we might see differences between say Whites and Non-Whites we must ask, are these differences based on solely on ethnicity or are there other factors involved? Even if there is a correlation between ethnicity and socioeconomic class, we cannot assume differences are due to what might seem like the most obvious demographic variable. Differences may be attributed to ethnicity when they’re actually due to socioeconomic class. We need to know what we’re measuring. So, what are we to do?

The most evident step we can take is to set quotas based on some calculation of socioeconomic status and analyze data accordingly. Why isn’t income enough? While income does relate directly to socioeconomic class, socioeconomic class or status includes other factors such as education, occupation, and overall wealth. To get a more accurate picture, we need to think of some creative ways to operationalize socioeconomic class as a variable. With the programming sophistication of online surveys, we could easily create variables based on the answers to a number of demographic questions. We could also cross quotas by socioeconomic class and ethnicity to make sure we can easily compare data within and between socioeconomic class and ethnicity respectively. While these steps might take a bit more time and, as a Sample Manager, I know it could make fielding a bit more difficult (or at least take a little longer), the quality of data coming out of the study would improve.

Regardless of the steps we take, we must be sure we know what we are measuring. For example, if we see differences between White and non-White respondents, we must ask whether the differences are due to ethnicity or some other set of factors. We cannot assume that what is on the surface and seems most obvious is actually correct. We must make sure that we understand the complex factors that inform one’s opinions, perspectives, desires, experiences, and so on. If we mistakenly attribute differences to race or ethnicity, but in reality it has more to do with economic position, this would obviously have negative effects on the strategic decisions informed by our research results. What’s the point of spending all the time, energy, and money on research that isn’t giving us the complete picture?


[1] Miller, Jane (2000) The Effects of Race/Ethnicity and Income on Early Childhood Asthma Prevalence and Health Care Use. American Journal of Public Health 90(3): 47-63

[2] http://www.npr.org/2012/02/13/146816813/income-more-than-race-is-driving-achievement-gap

[3] Coleman, Richard P. (1983) The Continuing Significance of Social Class to Marketing. Journal of Consumer Research Vol.10: 265-280

[4] Cicarelli, James (1974) On Income, Race, and Consumer Behavior. The American Journal of Economics and Sociology 33(3): 243-247

The insights I received from iModerate really brought our NPS program to life. While it was always highly-visible and important to key stakeholders it did not resonate as well with the majority of employees. The iModerate piece rounded out the NPS program and brought it to a place where it is now more valued, transparent and salient across the organization. Having the consumer’s voice and that context has helped us build business cases and impact operations in a way that has led to great success.

Adriana Smith, Manager, Brand Strategy, NRG Energy