When Numbers Don’t Tell the Whole Story: What Geology Can Teach Us About Social Science

I recently sat down with Marc Defant, a volcanologist who spent his career studying the Earth’s most violent processes. We talked about masculinity, feminism, and why he thinks the social sciences don’t measure up to the standards of hard science. He wasn’t wrong about the measurement problem. But he was missing something important about what measurement can and can’t capture. This gap between how we study rocks and how we study people matters more than most of us realize. It shapes policy decisions, public discourse, and how we understand the problems we’re trying to solve.

The Replication Problem Is Real

Let’s start with the uncomfortable truth: social science has a credibility problem. In 2015, researchers attempted to replicate 100 psychology experiments. Over two-thirds failed to produce the same results when other scientists tried to reproduce them using identical methods. A separate effort focused on high-profile studies published in Nature and Science between 2010-2015. Only 62% successfully replicated. The effect sizes averaged about half of what the original studies claimed. This isn’t a small problem. These are the flagship journals. The studies that shape policy and public understanding.

Research shows that social sciences had 2.3-fold increased odds of reporting positive results compared to physical sciences. That suggests fewer constraints on researcher bias in softer fields. When Marc brought up his skepticism about social science claims, he was pointing at something real. The numbers in the humanities often don’t hold up under scrutiny.

How Geology Gets It Right

Earth scientists have built-in verification methods that cross-check each other. They use radioactive dating to establish absolute ages. Rock sequences provide relative chronology. Fossil correlations validate both approaches. A geologist studying ancient volcanic eruptions can measure chemical composition with mass spectrometers. They can analyze minerals at the atomic level with scanning electron microscopes. They can record seismic activity with instruments that detect movements smaller than a human hair. The humanities simply can’t compete.

Different measurement techniques must align. If they don’t, something is wrong with your hypothesis. This layered approach creates falsifiability that’s harder to achieve when studying human behavior. Rocks don’t change their properties based on who’s studying them or what the researcher hopes to find. James Hutton established the principle of uniformitarianism in geology: the present is the key to the past. Geological processes observed today worked similarly throughout Earth’s history. This gives geologists a methodological anchor. You can make testable predictions about past events by studying current processes.

Social sciences lack such universal principles. Human behavior and social systems are adaptive, context-dependent, and not rigorously rule-bound. This makes it very hard to replicate every finding and often those will change based on a variety of social factors. Rocks tend to stay the same.

The Problem Marc Missed

During our conversation, Marc kept coming back to the numbers. Feminism’s claims didn’t add up when you looked at the data. Take the gender pay gap. He’s right that the 77-cents-on-the-dollar figure is misleading. Women tend to choose jobs in helping professions that pay less than the jobs men typically take. Female teachers work in every school in this country. Female underwater welders and power line technicians are rare. It has been illegal to pay women less for the same job since the 1970s. The gap isn’t about discrimination in pay for identical work. But here’s what the numbers alone can’t tell you: why society values childcare and social work so poorly in the first place.

That’s not a measurement problem. That’s a values problem. The numbers don’t always tell the whole story. Much of what matters has to do with lived experience, values, and societal priorities. The humanities talk about these things by talking to people. You can’t always fit that in a graph.

When You Can’t Run Experiments

Geology shares a methodological limitation with social sciences: you can’t perform controlled experiments for most hypotheses. You can’t recreate a volcanic eruption in a lab to test your theory. You can’t rewind Earth’s history and change one variable to see what happens. Astronomy faces the same constraint. You observe and study natural experiments instead.

The difference is that geologists and astronomers can still measure physical properties independently using mathematics and physics that operate under consistent laws. Social sciences must measure subjective phenomena. Political beliefs, behavior, emotions. These require creative solutions that measure non-physical concepts while maintaining scientific standards. That’s a fundamentally harder challenge.

Physicist Eugene Wigner described the “unreasonable effectiveness” of mathematics in natural sciences. The natural world operates according to strict mathematical laws that are ultimately discoverable and consistent. Physics can predict the electrical current in a wire with great precision. It can calculate the motion of objects with accuracy. Human communities don’t obey simple mathematical laws. They sometimes come close in narrow ways, providing just enough intermittent reinforcement to keep researchers hoping the answer is around the next corner. Often it’s decades of hard work away.

The Incentive Structure Problem

There’s another issue Marc’s critique doesn’t address: the system that produces social science research is broken. Research published in 2024 shows that while transparency measures like pre-registration and data sharing have advanced openness in social sciences, they often fall short in mitigating questionable research practices due to persistent incentive misalignments.

The culture still prioritizes quantity over quality and innovation over rigor. Researchers face pressure to produce statistically significant results rather than prioritizing methodological soundness. One statistician who has studied this issue across multiple disciplines called it “an existential crisis” for science. This isn’t unique to social sciences. But it hits harder in fields where the phenomena being studied are already more difficult to measure and replicate.

Context Sensitivity Matters

Research analyzing the Reproducibility Project found that context sensitivity negatively correlated with replication success. Effects that depend on specific circumstances are significantly less likely to replicate, even after adjusting for sample size, effect size, and methodological similarity.

This presents a fundamental challenge for applying social science findings to policy decisions across different contexts. A study showing that a particular intervention works in one city with one population at one time might not work somewhere else. That doesn’t mean the original study was wrong. It means human systems are complex and adaptive.

Geology doesn’t face this problem. The principles governing how rocks form and erode remain consistent across time and geography.

Why This Gap Matters for Public Discourse

In any given issue, there’s a delta between the reality of the situation and how we talk about it. I try to talk about things in that space between the narrative and the reality. I saw this almost every week on The Cameron Journal Newshour. When social science research can’t meet the evidentiary standards of hard science, people lose trust in expertise. They start dismissing all research as biased or politically motivated. That’s not the case, the humanities work different than the hard sciences.

That’s dangerous. We need good social science. We need to understand human behavior, social systems, and how policies affect people’s lives. But we also need to be honest about the limitations. About what numbers can and can’t tell us. About where lived experience and values matter as much as measurement.

Marc’s skepticism about feminism’s claims reflects a broader frustration many people feel when narratives don’t match observable reality. The response can’t be to abandon measurement and retreat into pure subjectivity. But it also can’t be to dismiss everything that doesn’t fit neatly into a regression analysis. But I’m so glad that he brought that expertise to the conversation on feminism because it gives us a chance to change.

What We Should Do Differently

Social scientists need to stop overselling their findings. Stop making sweeping claims based on studies that barely replicate and depend heavily on context. Be honest about uncertainty. Be clear about limitations. Stop letting the pressure to publish drive questionable research practices.

Policymakers need to stop treating social science research as if it provides the same level of certainty as physics or geology. Factor in context sensitivity. Recognize that what worked in one place might not work in another. The rest of us need to get comfortable with nuance. With the idea that some questions have clear answers and others require judgment calls based on incomplete information. We need to value both measurement and meaning. Both numbers and narratives. Both what we can prove and what we can only understand through lived experience.

The Real Standard We Should Hold

I don’t think social sciences should try to be geology. They’re studying fundamentally different things that require different approaches. But they should adopt geology’s commitment to multiple verification methods. Its willingness to say “we don’t know yet” when the evidence isn’t clear. Its focus on building knowledge slowly through careful observation rather than chasing headline-grabbing findings.

Most importantly, they should adopt geology’s humility about what can and can’t be measured with precision. Some rocks formed billions of years ago. Geologists can tell you their age within a few million years. They can describe the conditions under which they formed with remarkable accuracy.

But ask them to predict exactly when the next major earthquake will hit, and they’ll tell you they can’t. The systems are too complex, the variables too numerous. That honesty about uncertainty doesn’t diminish their expertise. It enhances it. Social sciences would benefit from the same approach. Less certainty about things we can’t really measure. More honesty about what we don’t know.

And more recognition that some of the most important questions can’t be answered with numbers alone. Context is vital. Understanding the bigger picture and how we got here matters as much as any data point. Maybe more. That’s the standard worth holding.