If there is a lack of reliable data on the occurrence of specific events or knowledge on the consequences of decisions, there is a problem of uncertainty. Better decisions can hardly be achieved through the trained use of statistics or their transparent communication. Instead, the central question is how individual consumers can reduce uncertainty in their decision-making situation. Two scenarios are central here:

How can uncertainty be reduced (quickly, practically) for everyday problems in which consumers are left to their own devices?

How can uncertainty be reduced (quickly, practically) for everyday problems for which an expert provides advice to the consumer?

Why is it difficult to provide decision support for problems of uncertainty?

Decision problems of uncertainty are characterised by a lack of reliable data. This effectively rules out the direct selection of the best decision option. The support consists of identifying key strategies to reduce uncertainty. What do I need to ask to reduce the choice of potential information or options? What do I need to look for? What do I need to consider to sort out inappropriate options that do not meet the minimum requirements?

In contrast to consumers, experts in a particular subject area are able to identify objective shortfalls in the standard of a decision problem on the basis of fewer heuristic features. With the help of an analysis of specific consumer decision situations, possible expert heuristics are distilled into decision trees. These summarize the experts' gut feeling based on their experiences and provide consumers with a robust expertise that enables them, similar to the expert, to separate the wheat from the chaff. 

This is not only important for issues where consumers are left to their own devices. Potential decision heuristics can also be combined in decision trees for consulting situations: Here it is a matter of asking the consultant the most important questions in order to be able to assess this situation robustly. 

Fast-and-Frugal Trees (FFTs) are suitable decision trees that can be transparent, comprehensible to consumers and of high quality at the same time. These FFTs represent a sequence of features to be examined (Martignon et al., 2008). There is always only one branch (stop) or one arrives at the next test feature, but there are no further branches (see example below). This distinguishes the FFTs from the usual decision trees. Only the last feature in the chain has two branches. 

It has been shown that FFTs enable fast and reliable decisions in various decision situations under uncertainty, e.g. in psychiatry, anaesthesiology, but also in the financial world (Aikman et al., 2014; Green & Mehr, 1997; Jenny et al., 2013). FFTs can be presented both digitally (e.g. app, website) and analogously to consumers (e.g. on posters or in brochures) in the form of a graphically illustrated, simple tree structure. This makes them an evidence-based instrument for decision support that is easy to implement. In the RisikoAtlas project it was developed and implemented for the first time for everyday consumer practice. The use of FFTs is also helpful because their application trains skills. The use of FFTs facilitates the internalisation of key characteristics for problems and stimulates critical thinking.

The order of features in an FFT is critical and must be determined in advance. There are both manual and more complex approaches using machine learning methods. Once statistically determined, this combination of features allows consumers to robustly classify decision options (e.g., whether an informed decision is possible) by independently examining those features.

How to construct a decision tree for a consumer problem - the FFT method of case-based feature validity

A. What do you need?

For the evidence-based development of FFTs, all approaches (including the FFT method of expert-based feature validity) require base data consisting of three parts: Characteristics of the problem, problem cases and the respective case assessment.

Part 1 – Characteristics of the problem 

First, it is necessary to clarify what the problem is and to define the concrete decision or assessment on which information should be provided. What is the decision tree supposed to deliver? Under this aspect, potential features  are researched with the help of experts (e.g. workshops), colleagues, laypersons and specialist literature (trade journals, white papers, government reports and experience reports). Potential features are all those characteristics of the problem situation that could possibly be an indicator of a good or bad decision regarding the problem. It may also be worthwhile to include new features such as one's own assumptions or intuitions. A list of potential features should then have been established.

Each potential feature must be comprehensible and testable by a layperson. Ideally, the list should summarize similar features, especially if there are too many of them. It is fair to say that expert supported feature selection is the most important tool in advance, particularly when it comes to cost-effective development. After all, each additional feature requires more cases in order to allow robust development. As a rule of thumb, you can basically calculate 20 to 50 cases for each feature. Each case requires effort: Each case must be individually coded for all features and an assessment must be obtained. If you need support during this process, please consult the final report on the Risk Atlas project from July 2020 or contact us. Contact details can be found here.

Part 2 - Problem cases

Once you have made a selection of potential features, you need to find out how often and under what circumstances they occur in the real world. For this you collect material of typical decision situations, e.g. real purchase offers, videos of real consulting situations or real informational services.

If such case material of typical decision situations is not available, the FFT method of case-based feature validity is the method of choice. For this you have to consider how many candidate characteristics you want to examine, but also how rare the test object is, i.e. what the decision tree should help to identify. If you need support during this process, please consult the final report on the Risk Atlas project from July 2020 or contact us. Contact details can be found here.

If such case material from typical decision situations is not available, the FFT method of case-based feature validity is not suitable. In such a case, you have to use a different method.

Part 3 -  Case assessment

For each case in your data basis, you must know or determine whether the target criterion is met or not. In the case of health information, for example, a positive assessment would be the target criterion if it enables an informed decision, otherwise it would be a negative assessment. Without this basis of already determined profiles, no model for future decision support is possible. One approach would be to test each case, i.e. determine how it turned out. Very often, this effort is not feasible, because that would mean investigating 500 to 700 cases experimentally.

The alternative is then the "view of the expert", on which the model approach presented here was aimed from the outset. Several independent experts evaluate each individual case with a view to the goal of the development, e.g "Does this health information allow an informed decision?" Their assessment must then be combined. The median of their judgements proves to be more robust than arithmetic averages when combining the individual assessments. If you need support during this process, please consult the final report on the Risk Atlas project from July 2020 or contact us. Contact details can be found here.

B. How do you proceed?

You now have to find out which features the selected cases (for example, real purchase offers) show or do not show. During describing your cases in terms of the potential features, you learn a lot about the actual testability of the features by laypeople. It can be assumed that you will subsequently remove some features for which examination by the consumers would have been difficult.

Parallel to the coding of the cases in the features the expert evaluations are "collected". Experts receive only the case material, never the features or even the feature coding. The aim is to model the expert assessments independently (the expert's view).

As the coding of numerous potential features is very complex, an early point in time of a further reduction of the number of features promises great efficiency gains. However, before making a selection with the help of laypeople by testing how well they understand the features, a statistical approach is usually less costly and easier. A simple statistical feature selection is worthwhile after just 100 cases. Various tools are available, e.g. the boruta package or the caret package (both implemented in the open source solution R). With boruta, the basic meaning of the individual features is checked by so-called random forests. If a feature behaves like a random number with regard to the expert assessments, the statistical recommendation is to leave it aside due to lack of significance in further development. As with all statistical approaches based on random sampling, this also applies here: If your background knowledge tells you that the feature should actually be significant, test it in another round and also code it in the next 100 cases.

Repeat this process of coding, assessing, and statistical feature selection every 100 cases and try to make the set of features more manageable. This has a positive effect on the coding effort, the expert assessments and also on the model finding.

If nothing changes during feature selection, if you have assessed 500 to 1000 cases depending on the number of features, or if you run the risk of falling below six features, it is worth modeling the decision tree on the basis of these case-feature assessment profiles.

The pipeline for development can be summarized in a simplified illustration:

Modeling from tree development and cross-validation can be performed manually, but in the sense of effective modeling it is easier with the open source solution R. In addition to the FFTrees package (Phillips et al., 2017), you can also download a web solution by Evaldas Jablonskis and Uwe Czienskowski from  http://www.adaptivetoolbox.net/Library/Trees/TreesHome#/. If you need assistance with this, please consult the final report on the Risk Atlas project from July 2020 or contact us. Contact details can be found here.

You will model a Fast-and-Frugal Tree (FFT) using the portion of cases you select as training data; often 50% to 80% of cases. This FFT has a certain ability to make the right decisions (assessment). This means it will overlook cases in the real world and give false alarms on others. To quantify this ability, either perform a statistical cross-validation (you apply the decision tree to randomly repeated cases; test data cases) or apply it once to a collection of cases with assessments that you set aside before modeling (20% to 50% of the data).Alternatively, you can collect a completely new sample of cases with feature codings and assessments (out-of-sample) to which you apply the decision tree (additional time and effort).

Which quality is sufficient depends very much on the types of errors and the costs associated with the error. Finally, the model must be tested in practice with laypeople. Here, a randomised controlled study is useful. It compares the decision intentions of consumers who are given the decision tree with those who have nothing or a standard information sheet. If you need assistance with the assessment or quality, please consult the final report on the Risk Atlas project from July 2020 or contact us. Contact details can be found here.

• Aikman, D., Galesic, M., Gigerenzer, G., Kapadia, S., Katsikopoulos, K. V., Kothiyal, A., ... & Neumann, T. (2014). Taking uncertainty seriously: Simplicity versus complexity in financial regulation. Bank of England Financial Stability Paper, 28.
• Green, L., & Mehr, D. R. (1997). What alters physicians' decisions to admit to the coronary care unit?. Journal of Family Practice, 45(3), 219–226.
• Jablonskis, E., & Czienskowski, U. (2017). Decision trees online. http://www.adaptivetoolbox.net/Library/Trees/TreesHome#/
• Jenny, M. A., Pachur, T., Williams, S. L., Becker, E., & Margraf, J. (2013). Simple rules for detecting depression. Journal of Applied Research in Memory and Cognition, 2(3), 149–157.
• Luan, S., Schooler, L. J., & Gigerenzer, G. (2011). A signal-detection analysis of fast-and-frugal trees. Psychological Review, 118(2), 316.
• Martignon, L., Katsikopoulos, K. V., & Woike, J. K. (2008). Categorization with limited resources: A family of simple heuristics. Journal of Mathematical Psychology, 52(6), 352–361.

If you would like to adopt a consumer topic from our website, you can do so in the following three ways: 

1. You are using a digital copy. Either you directly save an illustration or download our PDF, or you integrate the illustration via Link(a href) or iframe.
2. You take your analogue copy and print out our PDF. The resolution and vector-based graphic is suitable for posters and brochures.
3. You recommend the app and refer to the Risikokompass from the PlayStore and AppStore.

If you would like to develop your own model, please consult the final report on the RiskAtlas project from July 2020 or contact us. Contact details can be found here.

When using the instruments, please mention the funding agency, which is the German Federal Ministry of Justice and Consumer Protection, and the Harding Centre for Risk Literacy as the responsible developers.

Logos can be downladed here.

FFT-method expert validity

FFT-method model

FFT-Methode getesteter Merkmalsvalidität

Method Natural Frequency Tree (NFT)

Method risk reading assistant for your browser

We have all been there - Dr. Google is frequently consulted about symptoms of illness. On the Internet there is a plethora of medical information that offer consumers to inform themselves about symptoms, benefits or damages of treatment options. Unfortunately, the quality of digital health information varies dramatically. Misleading information about medical interventions lead to the misjudgement of risks and prevents informed decisions than can sometimes have serious consequences. To prevent this, it is important that you can recognize the quality of health information on the Internet. Our decision tree as a digital checklist shows you what you need to pay attention to.

When do I need this figure?

If you are looking for health information on the Internet and find information that might help you, you can use the following decision tree.

What does the figure show?

The figure shows a decision tree that you can use to check any health information. Here you check whether the information provided helps you to make an informed decision or not.

A warning means that you are unlikely to be able to make an informed decision based on this health information. There can be many reasons for this: It may be because essential information is withheld. It may be advertising or designed unprofessionally. In some cases, the decision tree can come to a wrong conclusion.

An "all-clear" means that the health information can be used to support an informed decision. There remains the possibility of a residual error in the decision tree.

You can also check this information according to further quality criteria. Please note, however, that no offer or checklist is ever perfect. With each additional feature that you check, the risk of an incorrect assessment of the offer increases. Further features are:

• Is a health-related intervention described that could make a decision for/against?
• Is there no recommendation as to what should be done?
• Is a possible evaluation of the decision options clearly distinguished from the facts or is there no evaluation at all?
• Is the quality of the studies used for health information addressed?
• Are numbers expressed both positively and negatively at the same time, e.g. how many are improving and how many are not?
• Is it clear how the provider produces this and other health information?

Further established criteria for quality-assured health information can be found in the checklists of the Institute for Quality and Efficiency in Health Care (IQWIG) and the Ärztlichen Zentrums für Qualität in der Medizin (ÄZQ).

Where is the data that the decision tree is based on coming from?

Cases – Which health information served as a basis?

662 health information (cases) from German-language websites were collected,  487 of which were researched by experts from the Harding Center for Risk Literacy.

(1) Websites from the health category from web catalogues such as SimilarWeb. 

(2) Search results from Bing and Google for the following terms or combinations of terms (see Bertelsmann, 2018): health information; health guide; diseases; cold; migraine; abdominal pain; joint pain; skin eczema; carpal tunnel syndrome; cancer; prostate cancer; prostatitis; breast cancer; psoriasis; herniated disc; back pain; osteoarthritis; sarcoidosis; influenza; how to recognize ....; What should I do about ...; What works against ...; Should I see a doctor about ...; How dangerous is ...; Which household remedies help against ...; How long helps ...; What from ...; How can I ... prevent?

(3) The sample was artificially enriched by randomly drawn pages from websites which, according to their own information, follow the guideline Gute Praxis Gesundheitsinformation ("oversampling" of rarer cases with the expression "supports informed decision-making" compared to a random selection)

175 additional cases were compiled by laypeople in a study on antibiotics for upper respiratory tract infections, early detection of ovarian cancer and the mumps measles rubella vaccination for children.

Target assessment– How was determined whether an informed decision was enabled?

42 experts from research on health information, from health insurance companies, from the network Netzwerk Evidenzbasierte Medizin as well as representatives of health associations with professional experience in the field of health information assessed the cases.

Each information was evaluated by three experts with regard to the question: Does the health information provided enable a layperson to make an informed decision? A four-step response format was used. The median value of three experts each was used as the criterion value for the individual case. The experts did not receive any information about the potential features used in the study.

Potential features – Which features were considered?

Based on the „Guten Praxis Gesundheitsinformation“ (DNEBM, 2016) and according to the DISCERN-Standards (ÄZQ, 2019) 31 and 39 features respectively were identified as generally testable by laypeople by the RisikoAtlas team. A crosscheck with other publications (Bernstam et al., 2008; Bunge et al., 2010; Zhang et al., 2015) regarding this question the elimination of redundant features resulted in 65 potential features.

Selection of features and modelling

The purpose of pre-selecting features was to limit the number of candidates for the decision tree to distinguish between health information. The feature selection was performed from two points of view: Lay testability and statistical significance; in seven steps:

1. Cases 1–100 (information) were coded, compared, discussed and harmonised by two independent research assistants in 65 features. A statistical feature selection was then performed (using Random Forest Trees with boruta in the statistical program R) and different features were eliminated due to feedback of the coders (were found to be too difficult to use by laypeople). 22 features remained.
2. Cases 101–499 were were coded, compared, discussed and harmonised in groups of 100 by two independent assistants. Four further statistical feature selections were performed, which lead to the elimination of seven more features.
3. Cases 500–598 were used to check the coding behaviour of laypeople against a practiced research assistant. Five Clickworkers each did not achieve a satisfactory agreement on 3 features. The statistical feature selection did not change anything.
4. In a laboratory study, laypeople coded cases 600-675 with regard to the remaining 12 features. The statistical feature selection did not change anything.

The models

On the base model shown above.

Both, the recursive algorithm by Marcus Buckmann and Özgür Simşek (manuscript in preparation) and the FFTrees package (Phillips et al., 2017) were used for model identification. The ifan algorithm was used to optimise for balanced accuracy.

Model II – specifically for websites that inform about specific medical interventions

The second model is based not only on the assumption that the person seeking health information on the Internet prepares a decision on a specific medical intervention, but also on the assumption that he or she is in a position to recognise whether a website serves this purpose at all. Thus information offers, which have only the character of general knowledge transfer or are discussion platforms, are not considered.

What is the quality of the model?

The data (collected health information cases) were collected between 2017 and 2018 and coded in their features and assessed by experts. 661 cases were complete. The datasets were randomly divided into training datasets (two thirds) and test datasets (one third). The models have the following quality:

On the base model.

A cross validation of the identified decision tree resulted in the following quality measures: balanced accuracy = 0.74; specificity in the rejection of information that did not enable an informed decision (share of 86% in the test set) of 0.91. This means that 91 out of 100 information that do not enable an informed decision are recognised by the decision tree.

The sensitivity in the confirmation of health information that enable an informed decision (share of 14% in the test set) is 0.57.

Model II – specifically for websites that inform about specific medical interventions

A cross validation of the identified decision tree resulted in the following quality measures: balanced accuracy = 0.75; specificity in the rejection of information that did not enable an informed decision (share of 79% in the test set) of 0.91. This means that 91 out of 100 information that do not enable an informed decision are recognised by the decision tree.

The sensitivity in the confirmation of health information that enable an informed decision (share of 21% in the test set) is 0.59.

Potential for development

1. Continuous further development of the underlying training data due to changes in the market situation
2. Higher share of health information collected by laypeople in order meet the actual searching behaviour
3. Use of cases with experimentally confirmed "informed decisions" - beyond expert judgements - for additional validation
4. Further development for other languages that need their own training data

Empirical evaluation with consumers

All research results on the fundamentals and on the effectiveness of the RiskoAtlas tools in terms of competence enhancement, information search and risk communication will be published together with the project research report on 30 June 2020. If you are interested beforehand, please contact us directly (Felix Rebitschek, rebitschek@mpib-berlin.mpg.de).

Sources

• ÄZQ (2019). Qualität von Gesundheitsinformationen im Internet. https://www.patienten-information.de/checklisten/qualitaet-von-gesundheitsinformationen.
• Bernstam, E. V., Walji, M. F., Sagaram, S., Sagaram, D., Johnson, C. W., & Meric‐Bernstam, F. (2008). Commonly cited website quality criteria are not effective at identifying inaccurate online information about breast cancer. Cancer, 112(6), 1206–1213.
• Bertelsmann (2018). Suche nach Gesundheitsinformationen. Bericht.
• Buckmann, M., & Simsek, Ö. (Manuskript in Vorbereitung). Rekursiver FFT-Algorithmus.
• Bunge, M., Mühlhauser, I., & Steckelberg, A. (2010). What constitutes evidence-based patient information? Overview of discussed criteria. Patient Education and Counseling, 78(3), 316–328.
• DNEBM (2016). Leitlinie Gute Praxis Gesundheitsinformation.
• Phillips, N. D., Neth, H., Woike, J. K., & Gaissmaier, W. (2017). FFTrees: A toolbox to create, visualize, and evaluate fast-and-frugal decision trees. Judgment and Decision Making, 12(4), 344–368.
• Steckelberg, A., Berger, B., Köpke, S., Heesen, C., & Mühlhauser, I. (2005). Kriterien für evidenzbasierte Patienteninformationen. Zeitschrift für ärztliche Fortbildung und Qualität im Gesundheitswesen, 99, 6.
• Zhang, Y., Sun, Y., & Xie, B. (2015). Quality of health information for consumers on the web: A systematic review of indicators, criteria, tools, and evaluation results. Journal of the Association for Information Science and Technology, 66(10), 2071–2084.

Last update: 27 November 2019.

HIV self-test

HbA1c

Informed participation in bonus programs

Interessenkonflikte beim Arzt

Distinguishing between opinion and news