Much ado about nothing

As part of their Education and Workforce Initiative, the US Chamber recently came out with a survey on the business community’s opinions on education reform. It has a weighty title: “Education Reform: Insight into the Business Community’s Views About the U.S. Education System.” They’re touting this as a significant new piece of research, promoting it through external press releases, membership communications, and even a national conference call to highlight the results.

The problem is that it’s simply not valid research, and the Chamber should know better than to present it as such.

The 571 respondents were self-selected twice over, first opting in to the Chamber’s email lists (either as a US Chamber member or as someone interested in their education initiatives), and then opting in to participate in the survey from an email solicitation. This can hardly be called a representative sample: these are people with a demonstrated interest in Chamber representation as well as in education and workforce development, and hardly a cross-section of American businesspeople in general.

You can also review the respondent breakdown in the report to see how far apart we are demographically from having a representative group. Aside from the double-self selection mentioned above, consider that 68% of respondents are in organizations of 50 employees or less, and that the organizational breakdown indicates 52% of respondents are in small businesses and 20% are in chambers; large businesses didn’t make the top four, coming in less than 9% - below nonprofits, which shouldn’t be counted if the survey is supposed to gauge the thinking of businesspeople. Further, beyond these two questions, and one question each about job title and geographic region, there aren’t any other questions to help with analysis. How old are respondents? Do they have any children in school (public or private)? How knowledgeable are they about education issues? Do they have any experience working with public education? Just who are these people, and what is the basis for their opinions?

And we won’t get into the design of the questions itself – suffice it to say that they would have benefited from a round of focus groups before developing their questions.

As one of those 571 respondents, I have a problem with the US Chamber presenting these survey results as nationally representative of the business community’s thinking. The chamber is perfectly capable of performing a valid and representative report – they apparently chose to take a shortcut and perform a little sleight of hand instead.

This kind of survey may have had value as an internal research piece, helping the chamber find out what was on members’ minds as part of a larger focus group effort. But does it have value beyond that? It clearly does not – and I hope that in the future, the US Chamber uses its formidable resources and capabilities to produce valid and representative research that can advance the field of education reform.