The Oxford English Dictionary defines impact as a 'Marked effect or influence', this is clearly a very broad definition. The Goldsmith report concluded that general categories of evidence would be more useful such that indicators could encompass dissemination and circulation, re-use and influence, collaboration and boundary work, and innovation and invention. Evaluation is the systematic collection and inter- pretation of evidence leading as a part of process to a judgement of value with a view to action., Evaluation is the application of a standard and a decision-making system to assessment data to produce judgments about the amount and adequacy of the learning that has taken place., 1. The quality and reliability of impact indicators will vary according to the impact we are trying to describe and link to research. Wigley (1988, p 21) defines it as "a data reduction process that involves the . The verb evaluate means to form an idea of something or to give a judgment about something. It can be seen from the panel guidance produced by HEFCE to illustrate impacts and evidence that it is expected that impact and evidence will vary according to discipline (REF2014 2012). Evaluation research aimed at determining the overall merit, worth, or value of a program or policy derives its utility from being explicitly judgment-oriented. The introduction of impact assessments with the requirement to collate evidence retrospectively poses difficulties because evidence, measurements, and baselines have, in many cases, not been collected and may no longer be available. If metrics are available as impact evidence, they should, where possible, also capture any baseline or control data. For example, the development of a spin out can take place in a very short period, whereas it took around 30 years from the discovery of DNA before technology was developed to enable DNA fingerprinting. This is being done for collation of academic impact and outputs, for example, Research Portfolio Online Reporting Tools, which uses PubMed and text mining to cluster research projects, and STAR Metrics in the US, which uses administrative records and research outputs and is also being implemented by the ERC using data in the public domain (Mugabushaka and Papazoglou 2012). Collecting this type of evidence is time-consuming, and again, it can be difficult to gather the required evidence retrospectively when, for example, the appropriate user group might have dispersed. 60 0 obj << /Linearized 1 /O 63 /H [ 1325 558 ] /L 397637 /E 348326 /N 12 /T 396319 >> endobj xref 60 37 0000000016 00000 n The Payback Framework has been adopted internationally, largely within the health sector, by organizations such as the Canadian Institute of Health Research, the Dutch Public Health Authority, the Australian National Health and Medical Research Council, and the Welfare Bureau in Hong Kong (Bernstein et al. There is a great deal of interest in collating terms for impact and indicators of impact. What are the challenges associated with understanding and evaluating research impact? Although it can be envisaged that the range of impacts derived from research of different disciplines are likely to vary, one might question whether it makes sense to compare impacts within disciplines when the range of impact can vary enormously, for example, from business development to cultural changes or saving lives? HEFCE indicated that impact should merit a 25% weighting within the REF (REF2014 2011b); however, this has been reduced for the 2014 REF to 20%, perhaps as a result of feedback and lobbying, for example, from the Russell Group and Million + group of Universities who called for impact to count for 15% (Russell Group 2009; Jump 2011) and following guidance from the expert panels undertaking the pilot exercise who suggested that during the 2014 REF, impact assessment would be in a developmental phase and that a lower weighting for impact would be appropriate with the expectation that this would be increased in subsequent assessments (REF2014 2010). In developing the UK REF, HEFCE commissioned a report, in 2009, from RAND to review international practice for assessing research impact and provide recommendations to inform the development of the REF. In education, the term assessment refers to the wide variety of methods or tools that educators use to evaluate, measure, and document the academic readiness, learning progress, skill acquisition, or educational needs of students. It is now possible to use data-mining tools to extract specific data from narratives or unstructured data (Mugabushaka and Papazoglou 2012). According to Hanna- " The process of gathering and interpreted evidence changes in the behavior of all students as they progress through school is called evaluation". It is very important to make sure people who have contributed to a paper, are given credit as authors. SROI aims to provide a valuation of the broader social, environmental, and economic impacts, providing a metric that can be used for demonstration of worth. x[s)TyjwI BBU*5,}~O#{4>[n?_?]ouO{~oW_~fvZ}sCy"n?wmiY{]9LXn!v^CkWIRp&TJL9o6CjjvWqAQ6:hU.Q-%R_O:k_v3^=79k{8s7?=`|S^BM-_fa@Q`nD_(]/]Y>@+no/>$}oMI2IdMqH,'f'mxlfBM?.WIn4_Jc:K31vl\wLs];k(vo_Teq9w2^&Ca*t;[.ybfYYvcn The reasoning behind the move towards assessing research impact is undoubtedly complex, involving both political and socio-economic factors, but, nevertheless, we can differentiate between four primary purposes. The understanding of the term impact varies considerably and as such the objectives of an impact assessment need to be thoroughly understood before evidence is collated. An evaluation essay or report is a type of argument that provides evidence to justify a writer's opinions about a subject. Definitions of Performance Appraisal - By McGregor and Dale Beach . 0000007559 00000 n Where quantitative data were available, for example, audience numbers or book sales, these numbers rarely reflected the degree of impact, as no context or baseline was available. %PDF-1.4 % Perhaps, SROI indicates the desire to be able to demonstrate the monetary value of investment and impact by some organizations. The Social Return on Investment (SROI) guide (The SROI Network 2012) suggests that The language varies impact, returns, benefits, value but the questions around what sort of difference and how much of a difference we are making are the same. Authors from Asia, Europe, and Latin America provide a series of in-depth investigations into how concepts of . The traditional form of evaluation of university research in the UK was based on measuring academic impact and quality through a process of peer review (Grant 2006). These sometimes dissim- ilar views are due to the varied training and background of the writers in terms of their profession, concerned with different aspects of the education process. The term "assessment" may be defined in multiple ways by different individuals or institutions, perhaps with different goals. The following decisions may be made with the aid of evaluation. In this sense, when reading an opinion piece, you must decide if you agree or disagree with the writer by making an informed judgment. Systems need to be able to capture links between and evidence of the full pathway from research to impact, including knowledge exchange, outputs, outcomes, and interim impacts, to allow the route to impact to be traced. 5. If knowledge exchange events could be captured, for example, electronically as they occur or automatically if flagged from an electronic calendar or a diary, then far more of these events could be recorded with relative ease. Case studies are ideal for showcasing impact, but should they be used to critically evaluate impact? Evaluation of impact in terms of reach and significance allows all disciplines of research and types of impact to be assessed side-by-side (Scoble et al. Wooding et al. 2007; Nason et al. Impact is assessed alongside research outputs and environment to provide an evaluation of research taking place within an institution. This raises the questions of whether UK business and industry should not invest in the research that will deliver them impacts and who will fund basic research if not the government? Clearly there is the possibility that the potential new drug will fail at any one of these phases but each phase can be classed as an interim impact of the original discovery work on route to the delivery of health benefits, but the time at which an impact assessment takes place will influence the degree of impact that has taken place. If this research is to be assessed alongside more applied research, it is important that we are able to at least determine the contribution of basic research. Assessment refers to the process of collecting information that reflects the performance of a student, school, classroom, or an academic system based on a set of standards, learning criteria, or curricula. From an international perspective, this represents a step change in the comprehensive nature to which impact will be assessed within universities and research institutes, incorporating impact from across all research disciplines. Baselines and controls need to be captured alongside change to demonstrate the degree of impact. Here we address types of evidence that need to be captured to enable an overview of impact to be developed. Replicated from (Hughes and Martin 2012). 0000334683 00000 n Even where we can evidence changes and benefits linked to our research, understanding the causal relationship may be difficult. In the UK, UK Department for Business, Innovation, and Skills provided funding of 150 million for knowledge exchange in 201112 to help universities and colleges support the economic recovery and growth, and contribute to wider society (Department for Business, Innovation and Skills 2012). There has been a drive from the UK government through Higher Education Funding Council for England (HEFCE) and the Research Councils (HM Treasury 2004) to account for the spending of public money by demonstrating the value of research to tax payers, voters, and the public in terms of socio-economic benefits (European Science Foundation 2009), in effect, justifying this expenditure (Davies Nutley, and Walter 2005; Hanney and Gonzlez-Block 2011). SIAMPI is based on the widely held assumption that interactions between researchers and stakeholder are an important pre-requisite to achieving impact (Donovan 2011; Hughes and Martin 2012; Spaapen et al. The origin is from the Latin term 'valere' meaning "be strong, be well; be of value, or be worth". In this case, a specific definition may be required, for example, in the Research Excellence Framework (REF), Assessment framework and guidance on submissions (REF2014 2011b), which defines impact as, an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. The transition to routine capture of impact data not only requires the development of tools and systems to help with implementation but also a cultural change to develop practices, currently undertaken by a few to be incorporated as standard behaviour among researchers and universities. The case study does present evidence from a particular perspective and may need to be adapted for use with different stakeholders. One notable definition is provided by Scriven (1991) and later adopted by the American Evaluation Association (): "Evaluation is the systematic process to determine merit, worth, value, or . To understand the socio-economic value of research and subsequently inform funding decisions. The criteria for assessment were also supported by a model developed by Brunel for measurement of impact that used similar measures defined as depth and spread. Any person who has made a significant . Muffat says - "Evaluation is a continuous process and is concerned with than the formal academic achievement of pupils. Impact is derived not only from targeted research but from serendipitous findings, good fortune, and complex networks interacting and translating knowledge and research. It has been suggested that a major problem in arriving at a definition of evaluation is confusion with related terms such as measurement, The RQF was developed to demonstrate and justify public expenditure on research, and as part of this framework, a pilot assessment was undertaken by the Australian Technology Network. Throughout history, the activities of a university have been to provide both education and research, but the fundamental purpose of a university was perhaps described in the writings of mathematician and philosopher Alfred North Whitehead (1929). It is possible to incorporate both metrics and narratives within systems, for example, within the Research Outcomes System and Researchfish, currently used by several of the UK research councils to allow impacts to be recorded; although recording narratives has the advantage of allowing some context to be documented, it may make the evidence less flexible for use by different stakeholder groups (which include government, funding bodies, research assessment agencies, research providers, and user communities) for whom the purpose of analysis may vary (Davies et al. The first attempt globally to comprehensively capture the socio-economic impact of research across all disciplines was undertaken for the Australian Research Quality Framework (RQF), using a case study approach. 0000012122 00000 n Indicators, evidence, and impact within systems, Department for Business, Innovation and Skills 2012, http://www.arc.gov.au/pdf/ERA_Indicator_Principles.pdf, http://www.charitystar.org/wp-content/uploads/2011/05/Return_on_donations_a_white_paper_on_charity_impact_measurement.pdf, http://www.oecd.org/science/innovationinsciencetechnologyandindustry/37450246.pdf, http://www.cahs-acss.ca/wp-content/uploads/2011/09/ROI_FullReport.pdf, http://mice.cerch.kcl.ac.uk/wp-uploads/2011/07/MICE_report_Goldsmiths_final.pdf, http://www.timeshighereducation.co.uk/story.asp?storyCode=409614§ioncode=26, http://www.odi.org.uk/rapid/Events/ESRC/docs/background_paper.pdf, http://www.iscintelligence.com/archivos_subidos/usfacultyburden_5.pdf, http://blogs.lse.ac.uk/impactofsocialsciences/tag/claire-donovan/, http://www.atn.edu.au/docs/Research%20Global%20-%20Measuring%20the%20impact%20of%20research.pdf, http://www.hbs.edu/research/pdf/10-099.pdf, http://www.esf.org/index.php?eID=tx_ccdamdl_file&p[file]=25668&p[dl]=1&p[pid]=6767&p[site]=European%20Science%20Foundation&p[t]=1351858982&hash=93e987c5832f10aeee3911bac23b4e0f&l=en, http://www.rand.org/pubs/research_briefs/2007/RAND_RB9202.pdf, http://www.rand.org/pubs/documented_briefings/2010/RAND_DB578.pdf, http://ukirc.ac.uk/object/report/8025/doc/CIHE_0612ImpactReport_summary.pdf, http://www.timeshighereducation.co.uk/story.asp?storyCode=415340§ioncode=26, http://www.publicengagement.ac.uk/sites/default/files/80096%20NCCPE%20Social%20Value%20Report.pdf, http://www2.lse.ac.uk/government/research/resgroups/LSEPublicPolicy/Docs/LSE_Impact_Handbook_April_2011.pdf, http://www.artscouncil.org.uk/media/uploads/documents/publications/340.pdf, http://www.ref.ac.uk/media/ref/content/pub/researchexcellenceframeworkimpactpilotexercisefindingsoftheexpertpanels/re01_10.pdf, http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/02_11.pdf, http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf, http://www.ref.ac.uk/media/ref/content/pub/panelcriteriaandworkingmethods/01_12.pdf, http://www.russellgroup.ac.uk/uploads/REF-consultation-response-FINAL-Dec09.pdf, http://www.siampi.eu/Pages/SIA/12/625.bGFuZz1FTkc.html, http://www.siampi.eu/Content/SIAMPI/SIAMPI_Final%20report.pdf, http://www.thesroinetwork.org/publications/doc_details/241-a-guide-to-social-return-on-investment-2012, http://www.ucu.org.uk/media/pdf/n/q/ucu_REFstatement_finalsignatures.pdf, http://www.esrc.ac.uk/_images/Case_Study_of_the_Future_of_Work_Programme_Volume_2_tcm8-4563.pdf, Receive exclusive offers and updates from Oxford Academic, Automated collation of evidence is difficult, Allows evidence to be contextualized and a story told, Incorporating perspective can make it difficult to assess critically, Enables assessment in the absence of quantitative data, Preserves distinctive account or disciplinary perspective, Rewards those who can write well, and/or afford to pay for external input.

Textnow Area Code List, Articles D