Evaluating the readers’ comprehension – how well the subjects understood

Stage 5

6.1. Research focus and analysis of results

Th e main aim of this stage of the study was to evaluate the students’ comprehen-sion of each text and to compare the reading comprehencomprehen-sion results in relation to the English text and to the Polish text. It is important to explain that this analysis is based on the taxonomy of propositions identifi ed at Stage 3.

In order to evaluate reading comprehension, an ability to identify the main idea of each paragraph was measured. Students’ reports were analyzed to see how well the subjects understood the texts. Th e propositions that the subjects constructed were evaluated in relation to each paragraph. Th e readers’ performance while read-ing each paragraph was compared with the answers presented below (see model answers) and scored from 1 to 4 points. Th e following scoring criteria were accepted to evaluate the students’ comprehension of each paragraph:

Very good (4 points) – the protocol indicates complete understanding of the main idea of the paragraph; only paraphrasing the text, i.e., talking about it in one’s own words, is accepted for this score;

Good (3 points) – the protocol indicates partial comprehension of the main ideas, the reader ignores some important parts of the main message; only paraphrasing the text, i.e., talking about it in one’s own words, is accepted for this score;

Fair (2 points) – the report indicates superfi cial comprehension, i.e., the student understands pieces of text information but is not able to fi nd links among them and to draw conclusions concerning the main idea of the paragraph; only paraphrasing the text, i.e., talking about it in one’s own words, is accepted for this score;

Poor (1 point) – miscomprehension, missing the main idea; instead of paraphras-ing the text only readparaphras-ing aloud the text is identifi ed, in the case of the English text translating the text is observed;

0 point – the protocol related to reading a particular paragraph was not found.

It is important to emphasize that only fi nal propositions are taken into account, i.e., evaluated in this analysis; partial propositions are ignored – those about which the subjects expressed doubts (see Stage 3 of the study, presented in Chapter 4 in this part). In the case when comprehension of a particular paragraph is developed in the further (consecutive) paragraph (thereby partial propositions change into fi nal propositions), points are added to the score of this particular paragraph.

Model answers for the Polish text:

Par. 1. People accept Descartes’ theory, which claims that the immaterial spirit and the material body are two distinct things.

Par. 2. Th ere is no soul; neurobiological research shows that the soul is in fact an aspect of the brain processing. Example: new methods of observing the brain (magnetic resonance imaging) indicate that every emotion and thought is refl ected in the brain functioning.

Par. 3. Th e idea that there is no soul has not been accepted by people’s common knowledge. Example: people are not ready to accept the fi nding that meditation and education change our mind (i.e., the brain).

Par. 4. Paul Bloom in his book Descartes’ Baby claims that people are natural-born dualists, which means that we accept in a natural way that body and mind are dis-tinct entities. Example: children who perceive the world in this manner.

Par. 5. We may experience further illusions if we do not modify our opinions con-cerning the role of the brain in our life, i.e., if we do not accept the truth that the brain plays such an important role in our life. Example of such misunderstandings:

the use of Prozac.

Par. 6. Although emotional pain (just like physical pain) is a normal reaction to various stimuli that we encounter in life and may even facilitate our abilities to analyze our life, we should not suff er more than necessary – so the use of Prozac is justifi ed.

Model answers for the English text:

Par. 1. Progress involves democracy. It means a better life for everyone, not only for those privileged. But it seems that progress may have limits.

Par. 2. When compared with life in the past, nowadays industrial countries lead a better, richer life. Examples: better general living standards, more advanced medicine, longevity.

Par. 3. We pay a price for progress. Examples: consumption results in producing garbage, living longer entails spending more money on the health-care system, changes involve more stress.

Par. 4. We are aware of the problems that progress causes but we agree to accept the consequences that the problems lead to.

Par. 5. Our general opinion about progress has changed. Although reports of a negative infl uence on the earth (examples: global warming) are controversial, we face the question whether endless progress is possible.

Par. 6. Th e question as to whether endless progress is possible concerns also a dem-ocratic distribution of wealth. An equal distribution of wealth, also among poor populous countries such as India or China, would result in breaking the limits of the earth.

Par. 7. Progress does not make sense if it does not mean a just distribution of wealth.

In the past it was assumed that progress would be available to everyone. (All people would demand more, which would result in more advanced machinery.) Demand for a better life would create a constant economic development.

Table III.13. Reading comprehension scores – the Polish text

Table III.14. Reading comprehension scores – the English text Subjects

Th e scores for all the students are presented in Table III.13. and III.14. Th e com-parison of the scores for each text is done in Table III.15. below. Th e mean scores for each text show that the readers performed better while reading the English text.

Th e analysis also revealed which paragraphs were the most diffi cult. In the Polish text, it was paragraphs 3, 5 and 6; in the English text – paragraphs 6 and 7. To com-pare how the subjects performed while reading each text, the following grades were defi ned.

80% – 100% very good (5) 79% – 65% good (4) 64% – 50 % satisfactory (3) 49 and less – poor (2)

Th e analysis indicates that half of the subjects demonstrated the same level of reading comprehension in reading in both languages. Two students scored slightly better while reading in English (a one-level diff erence). Th e other three students scored much better while reading the English text (a two-level diff erence). Th e reasons why the three students performed so poorly while reading the Polish text will be discussed in the section which analyzes the interviews with the readers about their reading style. Similar scores in reading in Polish and in English were obtained by 70% of the subjects.

Table III.15. Reading comprehension scores in Polish and in English (presented as percentages and grades) – comparison

To see whether those diff erences are statistically signifi cant, the Wilcoxon signed-rank test was applied (Table III.16.). P values that are lower than .05 indi-cate diff erences that are statistically signifi cant. Th us, the results show that there are no diff erences between reading comprehension scores in Polish and reading comprehension in English. Also Figure III.5. illustrates this fi nding.

Table III.16. Reading comprehension scores for the Polish and English text

Language Median Lower

Figure III.5. Reading comprehension scores for the Polish and English text presented as median ranks

Source: own study.

6.2. Conclusions

Statistical analyses indicate that the students’ reading comprehension scores related to the Polish text did not diff er from those related to the English text. Th eir ability to understand the main ideas of text paragraphs was as good in reading in Polish as in reading in English. Th is provides an important argument in the discussion about the possibility of transfer of reading ability between the two languages. It may be assumed that the same level of reading ability demonstrated by the students in the study can facilitate transfer of reading ability, probably between Polish and English.

Additionally, the inspection of the students’ reports provided information con-cerning parts of the texts that the subjects found particularly diffi cult. As regards the English text, what proved the most diffi cult to understand was the concept of democracy. Th e fi rst paragraph introduces two concepts: progress and democracy;

5 out of 10 students reported paying attention to the word “democracy” and develop-ing the concept of democracy in their readdevelop-ing. Two students noticed this concept;

however, they did not discuss it, e.g., what it may mean and how it may develop in the further parts of the text. Th ree subjects did not report noticing the concept of democracy in relation to progress. Consequently, most of the readers did not un-derstand the concept of the democratisation of affl uence in paragraph 7, which, as

Test results

English Polish

405080907060100

indicated in Table III.12., was the most diffi cult paragraph for the subjects. It seems that the students did not take advantage of numerous overlaps of ideas between concepts in paragraph 7 and the earlier parts (see Appendix 2). Th is resulted in an incomplete, partial understanding of the text. Th is means that the representation of the text the subjects developed does not refl ect the message intended by the author of the article.

As regards the Polish text, the most diffi cult paragraphs were paragraph 3, 5 and the last one. Th e last paragraph, probably because of its digressive character (see Appendix 2), turned out to be only partially comprehended even by the best readers, e.g., Subject J (see Table III.13.). Paragraph 3 and 5 draw on the concept of dogmas, which is introduced in the subtitle. Out of 10 students only one reader reported paying attention to this proposition and developing it in her further comprehension.