All that glitters is not gold - User reviews should be read with caution
Why is it difficult to provide decision support for problems of uncertainty?
How to construct a decision tree for a consumer problem - the FFT method of expert feature validity
A. What do you need?
For the evidence-based development of FFTs, all approaches (including the FFT method model) require base data consisting of three parts: Characteristics of the problem, problem cases and the respective case assessment. The FFT method model is helpful if there is already a model of the decision goal of interest, e.g. a decision tree created by other researchers (Banerjee et al., 2017).
Part 1 – Characteristics of the problem
If a model is already available, the features used are usually also known (with the exception of certain deep learning approaches, which have not been particularly widespread for consumer problems until 2019). Each potential feature must be understandable and testable by a layperson. Internationally developed models in particular require a separate examination not only of the German language, but also of the cultural transfer to the German consumer world and the design of information or advice.
Part 2 - Problem cases
If a model already exists, a test data set is required for its empirical scientific verification, e.g. a collection of real decision situations such as real purchase offers, videos of real consulting situations or real informational offers.
With regard to the number of potential features and the rarity of the test object (what should the decision tree help to identify?), a systematic approach should be used to select samples of decision situations that are as ecologically valid as possible. If you need support during this process, please consult the final report on the Risk Atlas project from July 2020 or contact us. Contact details can be foundhere.
Part 3 - Case assessment
In order to test the model, you must know whether the target criterion is met or not for each case in your data basis. In the case of health information, for example, a positive assessment would be the target criterion if it enables an informed decision, otherwise a negative assessment.
Without this basis of cases that have already been decided, you cannot test the model or use your own model to model it after the original model.
One approach would be to test each case, i.e. determine how it turned out. Very often, this effort is not feasible, because that would mean investigating 500 to 700 cases experimentally. The alternative is then the "view of the expert", on which the model approach presented here was aimed from the outset. Several independent experts evaluate each individual case with a view to the goal of the development, e.g "Does this health information allow an informed decision?" The median of their judgements proves to be more robust than arithmetic averages when combining the individual assessments. If you need support during this process, please consult the final report on the Risk Atlas project from July 2020 or contact us. Contact details can be foundhere.
B. How do you proceed?
By having the cases of your test data set coded in the features of the original model, you learn a lot about the limited testability of some features by laypeople. However, it can be assumed that you will keep all the features that make up the model so that you can test it. For the same reason, there is no statistical feature selection.
Parallel to the coding of the cases in the features, the cases are to be tested, or the expert evaluations "collected" for your test data set. Experts receive only the case material, never the features or even the feature coding. The aim is to model the expert assessments independently (the expert's view).
Depending on the number of features, this process of coding and assessing can be completed after 200 to 600 cases, and the model is tested on the data set that you have generated yourself.
The pipeline for development can be summarized in a simplified illustration:
Modeling from tree development and cross-validation can be performed manually, but in the sense of effective modeling it is easier with the open source solution R. In addition to the FFTrees package (Phillips et al., 2017), you can also download a web solution by Evaldas Jablonskis and Uwe Czienskowski from http://www.adaptivetoolbox.net/Library/Trees/TreesHome#/. If you need assistance with this, please consult the final report on the Risk Atlas project from July 2020 or contact us. Contact details can be foundhere.
You will construct the model manually and check it against the generated test data set. You perform a cross-validation and apply the model to randomly repeated cases from the test data set.
Additionally or alternatively, you model a Fast-and-Frugal Tree (FFT) that is intended to mimic the model, but is probably simpler from the consumer's point of view. Then you would select part of the data set as training data; often 33% or 50% of the cases. This FFT has a certain quality in terms of tracking down your target feature (assessment). This means it will overlook cases in the real world and give false alarms on others. To quantify this quality, perform a statistical cross-validation or apply it once to a collection of cases with assessments that you put aside before modeling. Alternatively, you can collect a completely new sample of cases with feature codings and assessments (out-of-sample) to which you apply the decision tree (additional time and effort).
Which quality is sufficient depends very much on the types of errors and the costs associated with the error. Finally, the model and, if necessary, the new FFT must be tested in practice with laypeople. Here, a randomised controlled study is useful. It compares the decision intentions of consumers who are given the decision tree with those who have nothing or a standard information sheet. If you need assistance with the assessment or quality, please consult the final report on the Risk Atlas project from July 2020 or contact us. Contact details can be foundhere.
- Aikman, D., Galesic, M., Gigerenzer, G., Kapadia, S., Katsikopoulos, K. V., Kothiyal, A., ... & Neumann, T. (2014). Taking uncertainty seriously: Simplicity versus complexity in financial regulation. Bank of England Financial Stability Paper, 28.
- Banerjee, S., Chua, A. Y., & Kim, J. J. (2017). Don't be deceived: Using linguistic analysis to learn how to discern online review authenticity. Journal of the Association for Information Science and Technology, 68(6), 1525–1538.
- Green, L., & Mehr, D. R. (1997). What alters physicians' decisions to admit to the coronary care unit?. Journal of Family Practice, 45(3), 219–226.
- Jablonskis, E., & Czienskowski, U. (2017). Decision trees online. http://www.adaptivetoolbox.net/Library/Trees/TreesHome#/
- Jenny, M. A., Pachur, T., Williams, S. L., Becker, E., & Margraf, J. (2013). Simple rules for detecting depression. Journal of Applied Research in Memory and Cognition, 2(3), 149–157.
- Luan, S., Schooler, L. J., & Gigerenzer, G. (2011). A signal-detection analysis of fast-and-frugal trees. Psychological Review, 118(2), 316.
- Martignon, L., Katsikopoulos, K. V., & Woike, J. K. (2008). Categorization with limited resources: A family of simple heuristics. Journal of Mathematical Psychology, 52(6), 352–361.
Suppose you need a new backpack for hiking. You don't really know anything about backpacks. Since you have the widest choice online, you decide to buy the product on the Internet. The manufacturer's descriptions all sound the same and promise similar things to you. Since you really don't know what to watch out for when buying, you filter by customer ratings. After all, buying a product that only has five-star ratings cannot be a mistake. Or can it?
When shopping online, customer reviews are the most important decision criterion - even before price comparisons or recommendations by friends or relatives. But all too often these reviews are fake. It is estimated that every fifth review is not real. The review business is booming. At relevant agencies a paid review does not cost much more than 10 Euros. Fortunately, there are some criteria by which you can recognize false recommendations. Our decision tree as a digital checklist should help you to distinguish real reviews from fake ones.
Who is this decision tree for?
For all consumers who shop online.
You can also examine the user review further.
However, please note that no test or checklist is ever perfect.With every additional feature that you check, the risk of an incorrect assessment of the text increases. For example, you could check user reviews and user ratings with an automatic tool: www.reviewmeta.com. Although the site is in English, you can check German Amazon product links there, for example. In this way you can see how high the real rating is and which reviews are not trustworthy.
Where is the data coming from?
Cases – Which served as a basis?
Data from a study by the Harding Center of Risk Literacy served as a test data set.
Target assessment – How were the product reviews pre-assessed?
Product reviews were created for the modelling in a study by the Harding Center of Risk Literacy, and created artificially in another study by laypeople.
Potential features – Which features were considered?
Three features from Banerjee et al.'s (2017) model were used.
Selection of features and modelling
Three features from Banerjee et al.'s (2017) model were used.
What is the quality of the data?
Since the evaluation studies run until December 2019, no predictions can yet be made for the quality of the English-language model for product reviews in German.
Potential for development
Empirical evaluation with consumers
• Banerjee, S., Chua, A. Y., & Kim, J. J. (2017). Don't be deceived: Using linguistic analysis to learn how to discern online review authenticity. Journal of the Association for Information Science and Technology, 68(6), 1525-1538.
• Phillips, N. D., Neth, H., Woike, J. K., & Gaissmaier, W. (2017). FFTrees: A toolbox to create, visualize, and evaluate fast-and-frugal decision trees. Judgment and Decision making, 12(4), 344-368.
Last update: 27 November 2019.