There's a new player in the compensation data space that has all of us at the Cafe thinking about the future of pay transparency, the rise of big data, and what goes on in the minds of venture capitalists. It's called Comparably.com and according to its co-founders it "will be the leading resource for gender pay gap data, detailing what compensation women vs. men are making across hundreds of job titles."
That's a very bold claim... one that has attracted $6.5 million in funding from venture capitalists. The world of VC seems taken with pay transparency right now; earlier this month Glassdoor.com announced that it raised $40 mil in funding to "grow out its content areas and invest in machine learning tech" to detect bogus postings and improve matches between employers and candidates.
While I applaud the efforts to make pay transparency mainstream and ensure internally equitable pay practices in all organizations, I have some concerns...
The World's Best Mom Award
We're all familiar with these awards; you may have received one from your child or you may have given one when you were a child. They're very endearing, but they're completely meaningless for any kind of application or inference because they're based on a very small sample size.
Sample size is a big problem for many of these pay transparency sites. For example, Comparably.com just released a Salary Negotiations Study based on a survey of 5,000 people working in Tech. According to the Cyberstates 2016 report published by CompTIA, the Tech industry employs more than 6.7 million people, making the sample size about 0.075%! From a statistical perspective, this study is the equivalent of evaluating candidates for the "World's Best Mom" award based on the mothers you know, likely a very small proportion of the billions of other moms in the world...
Meet Chip, a (Survey)Monkey Suffering From Response Bias
The dangers of self-reporting are well documented, yet many of these pay transparency sites continue to use self-reporting as their main source of data. Why? Because it's cheap (in terms of time and money) and easy, especially with the advent of online tools like SurveyMonkey. It's not the fault of the tools - I am a user of SurveyMonkey. It's the way in which the tools are used. They're often used without reliability and validity checks. Anyone can log on to these pay transparency sites, create an account (or accounts, which is a different problem...) and self-report that they work for Company X as an (insert job title here) and earn $1 mil per year. I haven't seen any documentation on any of these sites regarding how the data is vetted. I assume it's not being vetted, and that's a problem.
The Emperors, Dressed in their New Clothes, Are In The Wrong Kingdom
I don't presume to know what goes on in the minds of venture capitalists when evaluating an entrepreneur's pitch. I understand the scope of my expertise, and don't stray outside of that scope. My fear is that some may be wandering outside of their scope of expertise and (un)intentionally muddying the waters of pay transparency.
Take Comparably.com as an example. The resumes of the four founders are quite impressive. Individually, they've launched companies like DocStoc and Yammer, been involved with Paypal, Invested.in and DebtMarket, Inc., and the CEO is currently serving as one of the "Entrepreneurs in Residence" for the City of Los Angeles. There's no doubt that these guys are the real deal, right down to their hoodied headshots, in the world of entrepreneurship. If I was launching a social media start-up, these are men I would call.
What's missing from the Executive Team is any specialized knowledge of compensation, survey design, or quantitative analysis of economic data. It's unrealistic to think that anyone could become the leading resource for gender pay gap data without having at least one person in senior management who knows something about compensation. Descriptive statistics like medians and ranges are a good place to start, but the complexities behind compensation decisions cannot be adequately captured with broadly aggregated descriptives generated from very small samples of self-reported data. Some level of expertise is required.
Data curation is not the same thing as data analysis.
It seems to me that these pay transparency sites are nothing more than curators of questionable data. I'm deeply concerned, as are others, that more and more of these sites will pop up, winning the day with visually stunning curations of data with no quantitative substance. While appealing, these curations can't assist organizations with compensation decisions and issues of internal pay equity. I'm not convinced that they will facilitate a meaningful discussion of pay transparency and pay gaps - gender, race, age, disability, etc. At the end of the day, it's just lipstick on pigs.
(Image: painting by Delilah Smith)
Stephanie Thomas, Ph.D., is a Lecturer in the Department of Economics at Cornell University. She teaches undergraduate and graduate courses on economic theory and labor economics in the College of Arts and Sciences and in Cornell’s School of Industrial and Labor Relations. Throughout her career, Stephanie has completed research on a variety of topics including wage determination, pay gaps and inequality, and performance-based compensation systems. She frequently provides expert commentary in media outlets such as The New York Times, CBC, and NPR, and has published papers in a variety of journals.