Revisiting The Infamous Hawthorne Effect

You may have heard of The Hawthorne Effect, a phrase that lives on as one result of a series of studies that took place between 1924 and 1927 at the Hawthorne Works of Western Electric Company in Cicero, Illinois. This article is a review of "Shining New Light on the Hawthorne Illumination Experiments," an article appearing in Human Factors (2011), by Masumi R. Izawa, Michael D. French, and Alan Hedge, through which they describe "an historical and statistical analysis of archival data from the Hawthorne illumination experiments." The researchers relied on copies of the "original documentation and much of the data from all three illumination experiments," which had previously been thought destroyed, but which had actually found its way into an archive at Cornell University.

Note: HFES and its publishing partner, SAGE, is offering free access to all HFES periodicals, including the article reviewed herein, through the end of November, 2012. Details appear at the end of this article.

Background

Drawing on previously published accounts of the controversial Hawthorne studies, Masumi et al provide a fascinating 'who dunnit' review of their background:

• General Electric (GE) provided $50,000 to the Committee on Industrial Lighting (CIL), a division of the National Research Council, "to scientifically prove to skeptical factory management executives that more lighting would increase productivity;"
• the honorary chair of the CIL at the time was Thomas Edison;
• study findings that suggested artificial lighting "improved industrial productivity would have been apt for" a celebration of the 50th anniversary of Edison's invention of the incandescent light bulb, scheduled for 1929, as well as improving the bottom line for electric companies;
• CIL chairman Dugald Jackson (Massachusetts Institute of Technology (MIT)) managed the experiments, though Charles Snow, the source of the data Masumi et al relied on for this article, actually conducted the experiments;
• lighting patented by Ward Harrison, an engineer at National Lamp Works of GE, were used in the study;
• the results were expected to demonstrate an increase in productivity related to the increased illumination, but the first experiment failed to do so;
• $90,000 was raised by the National Electric Lighting Association to combat adverse public opinion should the unfavorable results become public;
• by the end of the third experiment, GE and the CIL concluded that lighting had no effect on worker productivity, but a final report was never published;
• Snow resigned from MIT in 1927 and joined Western Electric Company in 1928;
• George Pennock, Hawthorne superintendent, ordered all data from the experiments destroyed in 1928;
• the data that were sent to MIT were also destroyed;
• without the data, no further analysis or re-analysis has been possible since, and "subsequent accounts of the illumination experiments have been based on secondary sources and anecdotal accounts;"
• further confusing the issue, subsequent accounts of the original three studies were incorrectly described as being part of the original experiments conducted by Snow;
• for example, Snow's research assistant Homer Hibarger, conducted a different study, sometimes called the 'Moonlight Study', in which Hibarger reported that 'the test was successful, demonstrating as it did the possibility of maintaining a uniform output under greatly reduced illumination intensities,' again suggesting that additional lighting was not necessary for the manual task investigated;
• yet another study by Hibarger, known as the 'relay assembly test room experiment,' which became the source of the first published statistical analysis related to the Hawthorne facility, and is also the data source for later statistical analysis by other researchers.

Masumi et al conclude their historical review with this statement, arguing that none of the studies they reviewed conduct a comprehensive analysis of the results of the three original experiments:

More recently, Levitt and List (2011) claim to have uncovered the original data from the illumination experiments and conducted a statistical analysis to verify the elusive Hawthorne effect. A close review of their study reveals that critical details of the illumination experiments were glossed over. 

This statement left me wondering what Levitt and List claim to have uncovered, and what they "glossed over," but Masumi et al do not provide any additional detail.

Masumi et al also claim to have access to original documentation, authored by Snow, that ultimately found its way to Cornell University. They report the path to Cornell as:

• Snow had kept documentation from his work, including log notes, photos and draft reports, with appendices, of the three experiments in question;
• in the late 1950's a graduate student named Charles Wrege contacted Snow and borrowed the documentation;
• in 1961, Snow gifted the archive to Wrege;
• in 1980, Wrege donated the archive to Cornell, where they are now held in Collection No. 5167 at the Kheel Center for Labor-Management Documentation and Archives in the M. P. Catherwood Library;
• "Despite this scholarly windfall, it seems that the existence of these documents has been generally unknown to researchers," state Masumi et al.

The Original Experimental Methods

Reviewing the archived materials, Masumi et al found that Snow had reached his conclusions by "eyeball analysis of vector graphs that plotted the percentage changes in production against lighting treatments and light levels." In the 1920's, statistical theory and methodologies where not very sophisticated, so Masumi et al first sought to understand the experimental designs, then applied appropriate modern statistical tests to the available data. The authors provide a great deal of detail in the full article, and interested readers are encoured to read it.

Next, Masumi et al extracted and organized original data from the graphs and tables contained in Snow's draft reports and re-analyzed the data using modern statistical methods, which they describe in detail in their paper. I will not review the detailed findings here, but for those with a solid interest in and understanding of experimental design and statistical analysis, it is a worthwhile read. Instead, I will now turn to Masumi et al's findings and interpretations.

The First Illumination Experiment

Contrary to Snow's original conclusion that light level did not affect productivity, Masumi et al found that in one experimental treatment there was a positive effect but that positive effect did not repeat in the second, similar treatment. The authors attribute this to a potential seasonal effect, because the experimental treatments took place at different times of the year, each with different lengths and intensity of natural daylight that could have had an influence. "Unfortunately, it is not possible to further disentangle these effects because of the limited data collected and summarized by Snow," they report.

The Second Illumination Experiment

Snow reported that there was a positive effect on productivity, but believed 'that the effect of increased supervision and the psychological factors incident to test conditions are of such magnitude as to completely mask any effect of illumination on production.' On the contrary, Masumi et al's statistical analysis shows no significant effect on productivity, and other than Snow's suggestion, they were unable to recover any data surrounding the question of supervisory changes.

The Third Illumination Experiment

Snow had concluded that, 'This test in a attempt to find the critical point of illumination . . . has demonstrated that the operatives [workers] will maintain their productive output at that level or in the neighborhood of that level which is customary, even in the face of insufficient illumination and the attending discomfort.' In other words. Masumi et al's analysis, however, found "that test group productivity was higher as the difference between test and control group light levels was greater (i.e., test group light levels were lower)," a statement that is unclear to me. At another point they say, "A negative association between light level and productivity was found in the third experiment."

So, in summary, Masumi et al's analysis found that there was:

• a positive effect of lighting on productivity in one treatment of experiment one, though it was not sustained in a second treatment;
• there was no association between lighting and productivity in the second experiment; and
• there was a negative association between lighting and productivity in the third experiment.

They suggest that these inconsistent results could be due to confounding factors, including uncontrolled work-related issues, experimental design issues, data factors, and the lighting levels that were tested in the experiments.

Lighting Level Issues

The lighting levels used in the different experiments varied, and the levels in experiments two and three were "mostly" lower than in experiment one. The highest lighting levels in experiment three, for example, were 1/5th the level of those in experiment one, and in all experiments, the lighting levels were lower than modern standards would dictate.

Experimental Design Issues

In order to adequately compare productivity across the three experiments, the durations of the lighting treatments in each would need to be consistent. However, the treatment periods ranged from 1 day to 24 days. The 1 day duration was driven by management concern that the lighting level tested would affect employee relations; the 24 day treatment was driven by the complexity of installing and uninstalling the lighting used for that particular treatment.

The layout of the building could also have affected the experiments, because each was conducted in different locations featuring different daylight exposures from windows. Compounding this effect, Snow notes that changes were made to the window treatments during the experiment, modifying the effect of daylight mid-experiment. The first two experiments, due to their building locations, exposed workers to a mix of daylight and artificial illumination. In the first experiment, lighting fixtures were also modified mid-experiment to reduce glare. Taken separately or together, these types of mid-experiment modifications confound baseline and experimental condition comparisons.

There were also inconsistent lighting levels at different periods in experiment one due to variations in power voltage. Masumi et al also noticed that task lighting was available to some workers in the Inspection department, though it is unknown if they were used during the experiments.

Psychosocial changes were also introduced mid-experiment in experiment three with the abrupt introduction of increased supervision.

Data Factors

Snow's log notes indicate lighting levels were sampled at different times of the day and at different locations. However, the individual measurements data was not contained in the archive, so Masumi et al had to rely on Snow's summary data consisting of daily group averages and general lighting levels, substantially reducing the power of their statistical analysis.

Work-Related Issues

Masumi et al point to several important issues that could have significant effects on these experiments, including:

• different employee pay methods, including hourly pay that varied on skill requirements, incentive pay based on individual piecework, and incentive pay based on 'gang' or group piecework, each potentially influencing worker motivations and ability to produce;
• different types of work with different visual demands, including:
  • inspection that required sufficient lighting to discriminate defects; and on the other extreme,
  • assembly work that was more motor memory in nature, and perhaps could even have been done in the dark or while blindfolded.

Even though the different types of tasks likely required different lighting, each was treated the same in the experiment.

So, What is The Hawthrone Effect?

Masumi et al note that The Hawthorne Effect has been interpreted in different ways at different times since the original experiments were conducted in the late 1920's. For example:

• some tie it to motivation, suggesting that productivity improves when management shows increased interest in workers;
• others suggest that it's an unavoidable outcome related to experimental participant knowledge that their behavior is being closely monitored.

Masumi et al suggest that, based on Snow's own writings, it was originally tied to the simple realization that the experiments, by their very nature and in the way they were carried out, increased supervision. Their research also suggests that the phrase Hawthorne Effect first appeared in writing in 1950, when John R. P. French Jr. wrote:

The experiments did demonstrate in a dramatic way, however, the importance of social factors in group productivity. From a methodological point of view the most interesting of these findings was what we might call the ‘Hawthorne effect’.

Regardless of where the phrase originated, they caution against interpreting it to mean that lighting does not affect productivity, due to the many study limitations described in their analysis. In their conclusions, Masumi et al state:

Our analyses of the Hawthorne illumination experiments have shown the value of understanding the historical, organizational, and business context in which this work was undertaken …

I enjoyed reading and reviewing this article, because the topic and story is full of intrigue and has direct impacts and implications on ergonomics. It's amazing to me that after all these years, we are still debating the studies themselves, as well as their meaning and application. It's truly a cautionary tale of what can happen when preconceived expectations influence scientific discovery; how important experimental design, data collection techniques, and interpretation methodologies are to our understanding of the world around us; and what happens when evidence is destroyed in an ill-conceived attempt to cover-up the truth, whatever truth that may be.

What specific truths the Hawthorn studies may have provided has been a subject of debate ever since, and speculation, interpretations and anecdotal accounts can get us only so far. Masumi et al's review of the history and analysis of the treasure trove in the Cornell archives takes us further toward the truth, yet still leaves much mystery.

It still leaves us wondering … does lighting affect productivity? If so, in what ways, and how could we measure and prove it in an industrial setting?

Reference

Masumi R. Izawa, Michael D. French, Alan Hedge, Shining New Light on the Hawthorne Illumination Experiments, Human Factors, October 2011, vol. 53, no. 5, pp. 528-547, doi: 10.1177/0018720811417968

At the time of this writing, this article was accessible online to HFES members and SAGE subscribers at: http://hfs.sagepub.com/content/53/5/528.full

HFES Nonmembers: Free access to HFES periodicals for the month of November 2012! Gain access at: https://online.sagepub.com/cgi/register?registration=HFES112012

This article originally appeared in The Ergonomics Report™ on 2012-11-07.