Critical appraisal: Is the information reliable?

Critical appraisal is the process of carefully and systematically examining and evaluating research to judge its trustworthiness, and its value and relevance in a particular context.

If you thought that critical appraisal was just “trashing papers” or that all papers that make it to publication must be reliable, you probably need to take another look at the subject. 

• As a healthcare professional it is important to question your information sources constantly. If you are basing decisions about patient care on research or using evidence to back up your theories in an assignment, then you need to make sure the research stands up to scrutiny.

• Evaluating information is an integral part of Evidence-Based Practice because some of the information on the internet and some published articles in healthcare journals are examples of bad research that shouldn’t be applied to current practice. We cannot take conclusions for granted.

You won’t always have the time to carry out comprehensive critical appraisal of every piece of information that you find, but it is well wroth spending some time looking at the quality of the articles, web sites etc. that you use.

Critical appraisal need not be a difficult or time consuming process. Sometimes, it simply involves assessing a research paper against a quick check-list, and there are lots of resources available to help you. We’ve selected what we think are some of the best:

• Different types of research study (Randomised Controlled Trials, Cohort Studies etc.) require slightly different appraisal techniques. The CASP guides for different study types, will help you to ask the right appraisal questions. Read more information on the different types of study, and the types of clinical question they are best suited to.

• SIGN (Scottish Intercollegiate Guidelines Network) checklists provide an alternative to the CASP ones above.
• NHS E-Learning for Healthcare have a course available online. Click here to access the E-learning. 
• As you get further into the topics of evidence based medicine and critical appraisal, you may come across some concepts you’re not familiar with; this glossary should help.
• Interpreting the statistics in research can be daunting for some people, but for the majority of appraisal there are only a few statistics that you really need to know. This glossary from St Andrew’s University provides useful information for all statistical terminology. King’s College London have Critical Appraisal Material (bottom of page) covering the types of statistics you are most likely to need to understand for basic critical appraisal, such as “odds ratios” (blobbograms!) or “numbers needed to treat”.
• There are various online tools and apps to help you work out some of the statistics that you might want to know but are missing from a paper. This calculator from CalcTool for numbers needed to treat, odds-ratios etc. is one example, and another is available through the CEBM Toronto. A web-search or look at an apps store will find you many more.

It’s not only quantitative medical research whose reliability you need to be able to judge. You’ll be getting information from all kinds of sources, and if it’s to do with your patients’ health it needs to be of the best quality.

• Qualitative research – When it comes to scientific research, not everything is statistics-based quantitative research. This Critical Appraisal Presentation, again from King’s College London, will help you to appraise qualitative studies.

• Patient information on the internet – The DISCERN tool will help you decide which sites contain reliable information. Otherwise, look out for information that has been produced by an organisation that is certified by the Information Standard.

• Other information on the internet – There are also check-lists you can use to assess the quality of any information on the internet, such as this from University of Berkeley.

• Health stories in the news media – Health stories are big news these days, and study results are often misrepresented.

If you’re looking for health care information on the internet we recommend you to try NHS Evidence as a first stop, as the information included has been accredited by NICE.

Advanced Critical Appraisal Critical appraisal can be a very in-depth subject requiring a high level of statistical knowledge, and is best learned from a real expert in the subject. For that reason, if you need advanced critical appraisal skills in your job we recommend that you contact one of the organisations listed below that specialise in critical appraisal to arrange training for you or your department. These courses can be expensive, so you may want to see if you can find other people or departments who would like to attend so that you can spread the cost. Critical Appraisal Skills Programme (CASP) – Based in Oxford and founded in 1993 CASP offer scheduled workshops in a variety of locations and run workplace courses. For more information please contact:

• CASP UK, c/o Better Value Healthcare Ltd Summertown Pavilion, Middle Way, Oxford, OX2 7LG. Email: [email protected]

Superego Cafe – Offer scheduled courses in various locations and bespoke workplace critical appraisal training. There is no limit on the number of participants and participants have access to an online course before and after the training event. For more information please visit www.superegocafe.com