Results Analysis – Stage 2

As the four different workshops are run continuously throughout the year it wasn’t certain that I was going to encounter the same students. Likewise not everyone completed the surveys. However I ended up with ten students who completed the questionnaires and that I was able to observe throughout the four week period. In completing the questionnaires the responses were often minimal; this was particularly noticeable by the third. This may be because 60% of the students were international and as English is not their first language they may have needed more time to formulate their responses. However I think it is more likely that this was a limitation of the method I used. The students were not particularly interested in my project or motivated to assist. A focus group with interviews would have been a more successful method. It is interesting to note that two of the students gave almost identical responses in all questionnaires. For example Student A throughout consistently rated his skill levels at the highest point and Student C almost consistently rated her skill levels fairly lowly and indicated no improvement. However my observations of both of them indicated that they both appeared to be very competent.

Statement Types

As you can see from the above graph the pattern was that there were far more evaluating statements than facts or conclusions. In some cases the students’ responses were almost identical in all three questionnaires from which we can conclude that the students weren’t aware that I was actually evaluating this response. One student just used bullet points which listed relevant work experience such as the design and implementation of workshops, toolkits and case studies. So despite there being little or no evaluating or concluding sentences in the responses it was still evident that she had quite extensive knowledge. Another response answered with a lot of detail and contained facts, evaluation and conclusions using phrases such as ‘explores sustainability’, ‘specifically focussing of practices of re-use and recycling’ and ‘aiming  to develop a model’.

I would have expected far more complex answers from these students which would have resulted in more conclusions. Higher research students would have a fairly comprehensive knowledge of their topic. In assisting with these workshops I was able to engage with the students and discuss their research areas. It was quite apparent that from conversations with them that they did have substantial knowledge of their topic. I believe the above results are a limitation of the questionnaire method in that I should have created more room for their responses. As well I didn’t explain to the students what sort of response I was expecting here. More direction would have resulted in more detailed responses.

Knowledge of Topic

In the above question the students were asked to identify how much they knew about the topic. It is interesting that 60% of the respondents stated that they knew “not much” about their topic before they started.  All but one student felt that their knowledge levels increased as they progressed. In the case of Student I the levels increased from “not much” to “a great deal”. Given that these surveys were administered over a four week period you would assume that these researchers would have been continuing to research outside of these workshops and would have been acquiring more knowledge about their topic which also would account for the increases.

Research Skills the Students Founds Easy to Do

In Question 3 and 4  I asked the students to identify what they did well or not well when researching and compared this to the Australian and New Zealand Information Literacy Standards. Many of the comments centred on the locating of resources. The above graph reflects a decline in this in Q.2 and Q.3. However I believe this is because the students’ responses were overall shorter in Q.3. Many of the students commented on completion of the course that they had learnt a great deal about how to find information more effectively which contradicts the responses. It may be that the some of the students still lacked confidence in their abilities. The ability to critically evaluate information improved significantly by Q.3 with students stating they were able to find good sources to support their research. One student stated that he was able to “locate, comprehend and apply new research to support the arguments”. Applying prior and new information to create new understanding also saw an improvement by Q.3 with students acknowledging that they were able to connect new ideas to identify different directions to take with their research. Many of the responses were very short with only one or two sentences. I believe I should have given the students more direction with this question and the next which may have resulted in more detailed responses.

Research Skills the Students Found Difficult to Do

Similarly when asked what students found difficult to do most of their responses centred on the locating of sources, critical analysis, managing their information and the creation of new understanding. By Q.3 most students identified searching for information effectively as no longer a major issue which demonstrates the effectiveness of the learning activity. Several students commented that keeping track and organising their research was an issue. Other comments such as “it is difficult to extract related concepts and apply them” indicate that creating new understanding was an issue. It would be hoped that by Q.3 there would be less statements of research difficulty and with the exception of the evaluation of sources column this is supported by the above graph. This is at odds with the data recorded on the specific Evaluation question (to which I will refer later on) in which all students recorded an improvement in their source evaluation skills by Q.3. It is also interesting to note that the students’ perception of what they did and did not do well heavily centered on critically evaluating sources.

Database Searching Skills

The second questionnaire was delivered after the database session and as expected there is an increase in all but one students’ perceived skill level. In the case of two of the students they reported a significant improvement with their responses changing from ‘not at all’ to ‘quite a bit’. In the case of student C my observation was that she was managing the searches easily although without interviewing her it is difficult to ascertain for certain that she was retrieving appropriate results. In this database workshop the students are demonstrated a multidisciplinary database and given time to practice searching. However some of the students needed assistance as they were not attaining results. This was because the chosen database was not appropriate for their discipline. Intervention by the facilitator and myself directing the students to more appropriate databases yielded success.

 

Knowledge of Database Search Features and Techniques

In the above question the students were asked what they knew about database search features and techniques. 80% responded ‘not all all’ or ‘not much’. It is positive to note that there was an increase in their knowledge after the session. Student J’s response was very positive. This student required a lot of assistance as she was an older student who was returning to study after a lengthy time. In the workshop the students were taught about Boolean operators. Many of the students requested assistance and clarification when performing their own searches. My observation by the end of the session was that most of them were using Boolean operators effectively.

Citation Searching

As would be expected the majority of students knew very little about citation searching. By Q.3 all of their skill level had increased significantly, in fact this question saw the largest improvement between the questionnaires and most students were very excited by this search feature.  As student A commented “the ability to track the research both forwards and backwards and identify key authors is really useful”. This section required the most assistance from myself and the facilitator – nearly all of the students asked for clarification as this was a new concept for most of them.

Internet Searching

Not surprising that the above results reflect the majority of students’ perceptions that their internet searching skills are good. The module on internet searching is conducted after questionnaire 2 was delivered and this is reflected with the students’ skill level remaining the same between Q.1 and Q.2. It is to be noted that 80% of the students felt their internet searching skills improved after this workshop. In my observation of the student cohort with the exception of students H and J they were all Generation Y and therefore are very familiar with Google etc. It is also worth noting that with the exception of Student J they were all familiar with Google Scholar. In the course of the workshop it became apparent that the students knew very little about different search engines or search refining techniques such as domain searching or advanced search functions. As one student commented to me “I didn’t realise how much more there was to know about internet searching”.

 

Evaluation of Sources

Evaluation of sources is discussed in the first workshop and the above graph reflects that students’ knowledge and understanding had increased before completing Q.2 but with no further improvement after that. This is the only question in which Student C recorded any different response between the Questionnaires.

Assessment

At the time of this presentation only 2 students had submitted their assessment. Student A’s assessment was extremely thorough and scored very highly. This was consistent with his responses in the questionnaires. In all questionnaires he consistently rated his abilities in the highest category and throughout the workshops he never sought assistance or clarification with the exception of his comment on citation searching.

Student J’s assessment was good with her obtaining a pass in all sections.  The assessments are graded and returned to the student with feedback and the option to resubmit. Student J did not want to resubmit however she did reply commenting very favourably on the program writing “I was completely floundering before I did this……thank you I have learnt so much”.

It was unfortunate that there were no other assessments to review. It would have been useful to be able to evaluate their assessment and compare to the observations and their responses.

Overall Analysis

Many students when handing me their final questionnaire at the completion of the program commented on how useful it was. The data analysis of the questionnaires confirms that there was a definite improvement in the students’ perception of their searching abilities. And this was supported by my own observations of them throughout the workshops. My main concern throughout was that the delivery of these workshops which is a combination of lectures, demonstrations and hands on practice would not be entirely effective. However the data, my observations, assessment evaluation and comments from the students would indicate that this was a successful information learning activity.

Advertisements