3.10 Problems and Solutions
3.10 problems and solutions
this section presents problems in the pilot study and discusses the solutions to these problems so as to avoid similar occurrence in the main study.
(a) administration order of the vocabulary post-tests
the vocabulary post-tests were originally planned to be executed in the order of the production test, the form test, and the reception test. however, in an attempt to make the tests easier and the participants more confident, the researcher changed the test order and put the most difficult test (the production test) after the form and reception tests. although this change in treatment was pedagogically reasonable, statistically it brought uncontrollable factors to the test results. after the participants finished the form and the reception tests, they would either become familiar with the spelling and meaning of the words being tested or remember the words for further testing, and thus the participants were apt to work out the words in the production test better than they could have done if the production test was given first.
to avoid pedagogical intervention in research administration, the main study will strictly administer the original order of the vocabulary posttests, i. e., the production test, the form test, and the reception test.
(b) the participants' different orientation for listening task two
as planned, when the participants finished listening task one, they were given the vocabulary post-tests to check their incidental vocabulary acquisition. andthesame procedure was used in the following week with listening task two. after the participants received the first round of listening tasks and the vocabulary post-tests, they may very possibly become aware of the “listening task+vocabulary posttests”formula and perhaps would realize that the same procedure would repeat in the next week. thus, when listening task two was administered, the participants may tend to prepare for the vocabulary post-tests after the listening task. the participants could pay special attention to the vocabulary items while listening, which would shift the “incidental”vocabulary acquisition to an “intentional”one. the statistics have shown that the participants scored higher in almost all the vocabulary posttests after listening task two than task one, which might reflect the purposeful notice taken to the vocabulary in listening task two.
therefore, to deter the participants from focusing on a different task orientation, one possible solution is to ask the participants to first finish both of the two listening tasks in a row, and then do the vocabulary posttests immediately.
(c) unpredicted instrumental factors
with an aim to testing the participants'incidental vocabulary acquisition of the ten target words, the pilot study deliberately chose some words unlikely to be known by the participants. in the vocabulary pre-test, the students were asked to circle a number on a five-point scale to indicate how well they knew the targeted words (e. g.4=i know the word well and can use it correctly; 0=i do not know the word at all). table 32 shows the average of the participants' self-reported knowledge of the words targeted for incidental vocabulary acquisition. unexpectedly, the average knowledge of two words (available and optimistic) was above 2, which means half of the students may have already known these two words before the pilot study, so the statistical data may not reflect the true vocabulary acquisition and thus resulted in unreliable analysis.
table 32 the participants'average knowledge about the target words
to solve this unexpected problem, the scores of the vocabulary posttests were reported in terms of percentage, from which the already known words were subtracted. to avoid a similar problem in the main study, these familiar words will be replaced with synonyms of lower word frequency.
(d) lack of a delayed vocabulary posttest
a delayed vocabulary posttest can provide a better picture of the retention of vocabulary acquisition. this was not in the design of the pilot study, because the time for the pilot study was rather limited. to reflect the retention of the vocabulary acquisition, some delayed vocabulary posttests will be included in the design of the main study.
(e) too few target words
the number of words targeted for the study of incidental vocabulary acquisition was only ten in the pilot study, which was too small. the number of the target words will be doubled for the main study.
(f) lack of a control group
the pilot study involved only three experimental groups and no control group. in the main study, four groups of participants will be engaged, with three groups acting as the experimental groups and one as the control group. the control group will receive the pre-test and posttests, but no treatment.
(g) no point in listening at different speeds
the effects of listening three times at different speeds didn't seem to be related to the research questions. due to the fact that the original design of listening three times at different speeds did not yield any interesting results, all listening will be administered at normal speed in the main study.
this section presents problems in the pilot study and discusses the solutions to these problems so as to avoid similar occurrence in the main study.
(a) administration order of the vocabulary post-tests
the vocabulary post-tests were originally planned to be executed in the order of the production test, the form test, and the reception test. however, in an attempt to make the tests easier and the participants more confident, the researcher changed the test order and put the most difficult test (the production test) after the form and reception tests. although this change in treatment was pedagogically reasonable, statistically it brought uncontrollable factors to the test results. after the participants finished the form and the reception tests, they would either become familiar with the spelling and meaning of the words being tested or remember the words for further testing, and thus the participants were apt to work out the words in the production test better than they could have done if the production test was given first.
to avoid pedagogical intervention in research administration, the main study will strictly administer the original order of the vocabulary posttests, i. e., the production test, the form test, and the reception test.
(b) the participants' different orientation for listening task two
as planned, when the participants finished listening task one, they were given the vocabulary post-tests to check their incidental vocabulary acquisition. andthesame procedure was used in the following week with listening task two. after the participants received the first round of listening tasks and the vocabulary post-tests, they may very possibly become aware of the “listening task+vocabulary posttests”formula and perhaps would realize that the same procedure would repeat in the next week. thus, when listening task two was administered, the participants may tend to prepare for the vocabulary post-tests after the listening task. the participants could pay special attention to the vocabulary items while listening, which would shift the “incidental”vocabulary acquisition to an “intentional”one. the statistics have shown that the participants scored higher in almost all the vocabulary posttests after listening task two than task one, which might reflect the purposeful notice taken to the vocabulary in listening task two.
therefore, to deter the participants from focusing on a different task orientation, one possible solution is to ask the participants to first finish both of the two listening tasks in a row, and then do the vocabulary posttests immediately.
(c) unpredicted instrumental factors
with an aim to testing the participants'incidental vocabulary acquisition of the ten target words, the pilot study deliberately chose some words unlikely to be known by the participants. in the vocabulary pre-test, the students were asked to circle a number on a five-point scale to indicate how well they knew the targeted words (e. g.4=i know the word well and can use it correctly; 0=i do not know the word at all). table 32 shows the average of the participants' self-reported knowledge of the words targeted for incidental vocabulary acquisition. unexpectedly, the average knowledge of two words (available and optimistic) was above 2, which means half of the students may have already known these two words before the pilot study, so the statistical data may not reflect the true vocabulary acquisition and thus resulted in unreliable analysis.
table 32 the participants'average knowledge about the target words
to solve this unexpected problem, the scores of the vocabulary posttests were reported in terms of percentage, from which the already known words were subtracted. to avoid a similar problem in the main study, these familiar words will be replaced with synonyms of lower word frequency.
(d) lack of a delayed vocabulary posttest
a delayed vocabulary posttest can provide a better picture of the retention of vocabulary acquisition. this was not in the design of the pilot study, because the time for the pilot study was rather limited. to reflect the retention of the vocabulary acquisition, some delayed vocabulary posttests will be included in the design of the main study.
(e) too few target words
the number of words targeted for the study of incidental vocabulary acquisition was only ten in the pilot study, which was too small. the number of the target words will be doubled for the main study.
(f) lack of a control group
the pilot study involved only three experimental groups and no control group. in the main study, four groups of participants will be engaged, with three groups acting as the experimental groups and one as the control group. the control group will receive the pre-test and posttests, but no treatment.
(g) no point in listening at different speeds
the effects of listening three times at different speeds didn't seem to be related to the research questions. due to the fact that the original design of listening three times at different speeds did not yield any interesting results, all listening will be administered at normal speed in the main study.