br Introduction The assessment of tissue protein expression

Introduction The assessment of tissue protein expression by immunohistochemistry (IHC) is widely used in both the clinical and the research settings. IHC combined with tissue microarray (TMA) technology (Wan et al., 1987; Kononen et al., 1998) provides an efficient approach to the study of multiple molecular markers in hundreds or thousands of tumors. TMAs are produced by removing cylindrical cores of tissue from up to donor paraffin blocks and embedding these into a single recipient paraffin block at set array coordinates. Several hundreds of tumors may be embedded in a single TMA. This has the potential to reduce inter-assay variability and to reduce the cost of research (Camp et al., 2008). Consequently, the large sample sizes required for robust inference in clinical epidemiology are achievable. A typical study may include over 10,000 cases (Ali et al., 2014). However, the process still relies on manual scoring of labeled sections by trained researchers. This is time consuming and scoring remains a rate-limiting step in this type of research. One solution to this bottleneck is to scan the labeled sections and to use automated analysis of the digitized images of each core. Several image analysis algorithms have been shown to perform reasonably well for some IHC markers (Giltnane and Rimm, 2004; Bolton et al., 2010; Ali et al., 2013; Howat et al., 2015). While automated image analysis remains promising, its implementation may be complex and it has not yet replaced manual scoring in large scale molecular epidemiology studies in cancer. An alternative approach to automated image analysis is crowdsourcing in which a function – here scoring of IHC labeled sections of tumor cores – is outsourced to an undefined and generally large group of people in the form of an open call. The crucial prerequisites are the use of the open call format and the large network of potential contributors (Howe, 2006). Crowdsourcing relies on parallel independent inputs from individuals allowing for large group size, maximizing cognitive iap apoptosis and enhancing group performance (Page, 2008). The Citizen Science Alliance (http://www.citizensciencealliance.org) is a collaboration of scientists, software developers and educators, who use the concept of crowdsourcing to develop, manage and utilize internet-based citizen science projects in order to further scientific research and to promote the public understanding of science. Through citizen science projects, thousands of Citizen Scientists have collected, organized and classified data for research purposes. Some successful initiatives are: the investigation of galaxy morphology (Lintott et al., 2008), the prediction of protein structures (Cooper et al., 2010) and the alignment of multiple sequences in genomic studies (Kawrykow et al., 2012). The Cell Slider project was established to enable the scoring of tumors labeled using IHC by untrained members of the general public – Citizen Scientists – through an internet-based interface. In this paper we report the results of the first Cell Slider project in which Citizen Scientists scored estrogen receptor (ER) expression in images of tumor cores from a large number of breast cancers arrayed in TMAs.
Methods
Results
Discussion Assessing individual Citizen Scientist performance is challenging in group aggregate work. In this study, a user performance score was developed to assign weights to Citizen Scientist results according to the level of their agreement with a specialist. However the overall performance of the group was not improved by weighting individual results. This is because of the effective loss of data for those Citizen scientists assigned a weight of zero. Our results agree with previous observations that the average of decisions from a group aggregate is accurate and multiple readings by a large number of individuals can correct for divergent results (Mattingly and Ponsonby, 2014). In a direct comparison in which the immunohistochemistry staining of over 3000 breast tumors cores arrayed on a TMA were scored by a pathologist and the Citizen Scientists, the Citizens performed well. The major weakness of the Citizen Scientists was in the identification of cancer cells on any given image. In the construction of tissue micro arrays small cores (typically 0.6mm) of representative areas of tumors are selected. However, the density of cancer cells may vary substantially across a tumor. Consequently, some cores contain no cancer cells at all (Ali et al., 2011). Other cores may include cancer cells but these may be unevenly distributed so that some of the sub-images of a core may not have cancer cells. Citizen Scientists tended to overestimate the presence of cancer cells, probably because of the presence of stained normal cells or technical artifacts. Lymphocyte expression of ER can cause false positive results (Pierdominici et al., 2010). The poor calling of cancer cells is also reported in studies with automated algorithms (Ali et al., 2013, Howat et al., 2015). This limitation is, perhaps, not surprising given that breast cancer cells are morphologically heterogeneous. Furthermore, the pattern recognition skills needed to identify cells with a large nucleus, irregular size and shape, prominent nucleoli, and scarce cytoplasm are greater than those required to simply identify the presence or absence of IHC staining.