Back to top

The Evaluation of Citizen Science. Recommendations for Policymaking from the Austrian Citizen Science Conference 2017

Sylvana Kroop, Barbara Kieslinger, Teresa Schäfer, Ursula Holtgrewe

Zentrum für Soziale Innovation (ZSI)

“Citizen Science” has increasingly gained attention over the last years. On a European level there are in the meanwhile a variety of citizen science activities, e.g. Horizon 2020’s Science with and for Society, where citizen science is embedded in the broader context of Responsible Research & Innovation (RRI). However, its evaluation is still in the very early stages of development. Although citizen science has raised to the top of research funding agendas at EU- as well as Member State level neither policy makers nor scientists currently have much empirical evidence about its impact on science and especially on society in order to justify that resources are efficiently spent and to sustain citizens’ engagement. Thus a sound approach of the evaluation of citizen science projects in its concepts, processes, feasibility, outcomes and impact is needed.

In the frame of a series of SIC’s participatory policy workshops we provided a workshop at the Austrian Citizen Science Conference 2017 on this topic.  An international group of 44 people primarily from Austria but also from Germany, Switzerland, and the United Kingdom had registered to our workshop, mainly scientists from natural sciences but also the humanities; a few participants came from government, schools, small businesses and civil society. We presented and discussed a framework for a holistic evaluation of citizen science whereas in the following at hand report we only highlight the issues that the workshop participants basically concerned most in terms of citizen science projects concluding with some policymaking recommendations at the end. It should help policy makers to have a better basis for balancing the decisions on the establishment of citizen science.


Figure 1: Impressions from the Workshop “How to evaluate citizen science?

Surprisingly: Although most participants had excellent knowledge about the challenges to implement a citizen science project 90% clearly stated to have no experience at all in the evaluation of citizen science. A key discovery of this event was the fact that scientists do have knowledge about evaluating the scientific results but do not know how to evaluate citizen science by itself. So what does it mean to evaluate citizen science and not just the scientific results of a citizen science project?

While citizen science is enjoying great popularity predominantly in natural sciences to gain large amounts of data and to make progress much faster with the help of the crowd for counting, locating or identifying animals, trees, proteins or galaxies, the current renaissance of citizen science goes much beyond this crowdsourcing approach. Citizens are not seen to be only external suppliers for research and development and the society is even less seen to be just a passive recipient of scientific knowledge. The current idea of citizen science is to have citizens and scientists actively meet on eye-level with the double purpose of developing both the “Scientific Literacy” of citizens and the “Societal Literacy” of scientists. The Horizon 2020’s Science with and for Society programme aims to reshape the relations of science, the economy and society and prioritises the encouragement of a sustainable dialogue between the academic and non-academic worlds. In this way, the knowledge of society as a whole can be pooled and made usable for research and development and eventually, the system of innovation can be extended throughout society. Moreover civil society organisations and citizens can benefit from research, e.g. in their fight to protect the natural environment.

This is a big challenge for scientists. Never before did the communication between citizens and scientists play such an essential role; mostly it did not play any role in research activities. As scientists with their standardised research activities are generally not used to such a citizen science approach there is a high risk of failure. If citizen science projects fail and citizens get frustrated and disappointed with the outcomes of their efforts there is a considerable reputational risk for scientists and institutions involved, and a risk of losing trust in citizen science projects as a whole. Therefore we want to know how to perform in a citizen science project to be successful.

In our workshop we presented and discussed general indicators (independently of any specific research discipline) to measure the quality of a citizen science project in terms of the process & feasibility as well as the outcome & impact. In the following we highlight the issues participants concerned most:

The conflict of focussing scientific excellence and the need of societal integration

The extra effort and expense for the communication between scientists and citizens remains one of the biggest challenges. This effort is known to be considerable compared with conventional science projects and may conflict with the actual research work. It generated lively discussion in the workshop, whether scientists should do the communication work with citizens. Some participants argued that researchers should focus their expertise on scientific work, doing research, generating new scientific insights and publishing them. However, the large majority of participants argued to not release scientists from their responsibility towards society. Scientists are supposed to experience the relevance, acceptance or refusal, and in general the social impact of their work. They are not to conduct their work in their ivory tower anymore; scientific work increasingly has to include society at large through the exchange with citizens. However, the question of how to improve mutual dialogue and to keep research activities efficient at the same time remains unsolved.

The vagueness of the existence and long-term impact of citizen scientists’ communities

Another question addressed the sustainability of a community of interested citizens established during a research project. After the end of a project there are no resources left to sustain a community and long-term impact is hard to assess. Is this community frustrated and breaks up because researchers don’t care about them anymore or does a joint research event have a long-term effect on the newly established community of citizen researchers? Citizens do not understand when the research (project) just stops and is discontinued. Citizen science has to be understood as a continuous enterprise and operation.

The challenge of evaluating the societal impact

The evaluation of citizen science including its impact on the whole of society is a big challenge for all participating citizen science experts. Participants asked and discussed the question: “How many citizens have I reached with a specific activity? Maybe there were 50 citizens participating but only 1 or 2 citizens were really reached.” Also statements regarding multiplier effects (e.g. How many people will spread an idea by using a certain communication instrument?) are mostly vague. There was full agreement to support the development of easy to handle evaluation instruments to explore and measure the potentials for social innovations by a certain citizen science approach.

Based on the workshop discussion we conclude with the following ideas for policy recommendations:

1. Meaningful interaction between science and society needs powerful support.  Stronger integration of science and innovation in society is envisaged (e.g. the Austrian ministry runs a programme called “Sparkling Science” to test the implementation of citizen science projects at schools). Since this requires considerable additional time and effort and may require but also promote the involvement of various disciplines and professions, it needs powerful support in research funding on both the EU- and Member State level for the dialogue between researchers and citizens. Established communities of researchers and especially among citizens have to be observed, engaged and attended to beyond a project’s lifetime to be successful in activating the innovative abilities of the (knowledge-) society. Thus funding programmes have to guarantee minimum resources and a minimum benefit system to support the needed process of sustainability.

2. Social Impact Award. A further important task is a rebalancing of the excellence and relevance of research. Current indicators of scientific achievement (especially publications and competitive funding) need to be extended to measure the societal impact of research activities (e.g. a social impact factor). The huge work to establish (and maintain) a dialogue ability between citizens and researchers, and the community building in citizen science projects, are currently unrewarded “extra” activities; such engagement may even disadvantage active promotors in relation to those researchers who do not lose time for dialogue and community work with citizens. An idea is to establish a “Social Impact Award”. Similar to the “agreement on achievements” (a usual instrument at Austrian university funding) a similar instrument could be developed to recognise the social impact achievements of institutions’ and individuals’ research activities.

3. Emphasising the evaluation of citizen science, not only the content-related scientific results. To demonstrate the added value of this participatory approach to science there is a need to evaluate not only the content-related scientific results but also citizen science in terms of the social, economic and environmental impact; to have some common indicators that should be applied across different projects. Built on the evaluation framework presented and discussed in our workshop, the idea to have such a framework as a “self-assessment-tool” could be further developed. It could include the possibility to make a “social impact check“ or a “participatory check” (similar to an “innovation check”) of research proposals, ongoing projects or the output of research activities. The anticipated benefit of making such a check could enhance awareness building and encourage scientists to adopt more inclusive and participatory research designs.