McNemar's Test in Python: Comparing Paired Categorical Data

McNemar’s test is a statistical method used to compare paired categorical data with two categories, often used to evaluate the effectiveness of a treatment or intervention. It involves creating a 2×2 contingency table, calculating the test statistic, and comparing it against a chi-square distribution to determine statistical significance.

Recommended: Understanding Bootstrap Statistics

Understanding McNemar’s Test for Paired Categorical Data

As mentioned above, McNemar’s test is done on paired data with two categories after a certain event. For example, let us suppose that a pharmaceutical company wants to test the effectiveness of a new treatment for a disease. We state our null and alternate Hypotheses.

H_o: New treatment is not as effective as standard treatment

H_a: New treatment is more effective than standard treatment

The next step is the data collection. In our scenario, we will ask our patients whether their condition improved or not improved. We will thus create a 2×2 contingency table like the one below.

McNemars Test Contingency Table — ***McNemar’s Test Contingency Table***

a: number of patients with improvement under both standard treatment and new medicine.
b: number of patients with improvement under standard treatment, but no improvement under new medicine.
c: number of patients with no improvement under standard treatment, but improvement under new medicine.
d: number of patients with no improvement under both standard treatment and new medicine.

The next step is to create a chi-squared test statistic. The calculated χ² is then compared against a chi-square distribution with a degree of freedom (df) as 1 ( df is taken as 1 because we have two parameters in our test ). Given below is McNemar’s test statistics.

Assuming that the confidence interval is 95%, then if the p-value is less than 0.05, then we reject the null hypothesis, otherwise we accept the null hypothesis. Let us understand this further with Python code.

Step-by-Step Implementation of McNemar’s Test in Python

In the code below, we have modeled the above situation and randomly generated the data for 20 patients before and after the new medicine. Thereafter, we calculated the test statistic and compared it with Chi-squared distribution.

import random

# Simulate data for 20 patients
data = []
for _ in range(20):
    # Randomly assign initial improvement (True/False)
    improved_standard = random.choice([True, False])
    # Randomly determine change in improvement with new medicine (True/False)
    change_in_improvement = random.choice([True, False])
    # Assign new medicine improvement based on change
    improved_new = improved_standard ^ change_in_improvement
    data.append((improved_standard, improved_new))

# Create contingency table
contingency_table = [[0, 0], [0, 0]]
for standard, new in data:
    if standard and new:
        contingency_table[0][0] += 1  # Both improved
    elif standard and not new:
        contingency_table[0][1] += 1  # Standard improved, new not improved
    elif not standard and new:
        contingency_table[1][0] += 1  # Standard not improved, new improved
    else:
        contingency_table[1][1] += 1  # Both not improved

# Print contingency table
print("Contingency Table:")
print("     | New Medicine - Improved | New Medicine - Not Improved |")
print("-----|-------------------------|----------------------------|")
print("Standard Treatment - Improved |", contingency_table[0][0], " |", contingency_table[0][1], " |")
print("Standard Treatment - Not Improved |", contingency_table[1][0], " |", contingency_table[1][1], " |")

# Calculate McNemar statistic
b, c = contingency_table[0][1], contingency_table[1][0]
mcnemar_statistic = ((abs(b) - abs(c))**2) / (abs(b) + abs(c))

# Calculate degrees of freedom (always 1 for McNemar test)
df = 1

# Use chi-square distribution to find p-value (assuming significance level alpha = 0.05)
from scipy.stats import chi2_contingency
chi2, pval, _, _ = chi2_contingency(contingency_table)

# Print results
print("\nMcNemar Test:")
print("McNemar Statistic:", mcnemar_statistic)
print("Degrees of Freedom:", df)
print("p-value:", pval)

# Interpretation (replace with your own interpretation based on p-value)
if pval < 0.05:
    print("There is a statistically significant difference between the new medicine and the standard treatment.")
else:
    print("There is no statistically significant difference between the new medicine and the standard treatment.")

Let us look at the output of the code.

We accept the null hypothesis from the output since the p-value is greater than 0.5.

Conclusion

Here you go! Now you know how to perform McNemar’s test. You also learned how to implement McNemar’s test in Python. Do keep in mind that McNemar’s test is only applicable to categorical variables. Hope you enjoyed reading It!!