0

I have 3 visualization techniques (e.g., PC, NL and Map). I wanted to evaluate user performance (Time and Accuracy) for these three visualizations methods. I have 9 tasks for participants to perform. These 9 tasks categorized into 3 groups such group 1 , group 2 and Group 3 (each groups contains 3 tasks). I ran between subjects with 36 participants (12 for each visualization method). I asked participants to perform same tasks in all three visualizations. Now I have 6 table which task completion time and Accuracy (whether participant could successfully perform the tasks or not) for 3 different visualization.

One:  Completion time for PC visualization method

           Participant 1 ......... Participant 12  
Task 1     23s (task completion time)     


Task 9


Second : Accuracy for  PC visualization method

           Participant 1 ......... Participant 12  
Task 1     Yes (yes if task completed)     



Task 9

How should I analyse these 6 tables? should I apply Anova? if yes which Anova method is the best one for my case? How should I apply Anova for accuracy where the results for accuracy is binary?

Glen_b
  • 257,508
  • 32
  • 553
  • 939
Bahador Saket
  • 140
  • 6

3 Answers3

1

you might need to clarify your experiment design but as I understand it: you have 3 groups of visualisation (PC, NL, Map) and 3 different types of task (group 1,2,3) - resulting in 9 combinations. Therefore you have 9 subgroups (e.g PC,group1, PC,group2...Map,group3).

You should do a 2x3 ANOVA - not a oneway ANOVA as you suggest in your title, if your outcome measures are continuous. If your outcome measure is binary, or otherwise categorical, consider a chi square test or logistic regression.

It sounds like the same people perform in all subgroups. You can therefore use repeated measures 2 x 3 ANOVA.

user41270
  • 215
  • 1
  • 3
  • 8
  • Thanks for the response. Do I need to run between subject ANOVa test since I have 36 participants. 12 of them performing tasks for PC visualization, another 12 for Map and remaining 12 NL visualization. – Bahador Saket Mar 18 '14 at 21:27
  • You have 3 types of visualisation, but also a number of different types of task (3 types of task???). To see if there is only a different between visualisation type ignoring what type of task the participants did) you'd run a 1 way ANOVA - with visualisation as your only factor (between subjects if you used the same subjects for each visualisation condition)..... – user41270 Mar 19 '14 at 17:01
  • ...however, it might be better to determine if there is an interaction between task and visualisation. This might show that some visualisations work particularly well with some kinds of task. For example, in a one way ANOVA you might find Map is best. But in an ANOVA with visualisation and task as factors (e.g a 3x3 ANOVA - if you have 3 tasks) you might see that Map is the best technique with tasks 1 & 2, but worst for task 3. This would mean that results from the oneway ANOVA are misleading. Map is not best overall - only for task 1 and 2. It is, in fact, worst for task 3. – user41270 Mar 21 '14 at 10:11
  • sorry read 2x3 not 3x3 – user41270 Mar 21 '14 at 14:34
  • Thank you for the answer. By conducting 2x3 Anova, I would definitely be able to see some visualisations work particularly well with some kinds of task but what if I would also be interested to see which one works best in general for all tasks? Should I first apply 1 way between subject anova to see the general performance and then conduct 2*3 Anova ? or by conducting 2*3 anova at the first place I would be able to find all answers. – Bahador Saket Mar 23 '14 at 15:22
  • 2 X 3 will give you all answers. The main effect of visualisations will tell which visualisation method results in the best performance, regardless of task. The main effect of task will tell you on which task people perform best, regardless of visualisation method....and any "task*visualisation" interaction will tell you whether a particular visualisation works better with a particular task. – user41270 Mar 23 '14 at 15:57
  • Please have a look at my explanation of results below and see whether it makes any sense. – Bahador Saket Mar 23 '14 at 23:59
1

I applied 2*3 between ANOVA and wrote my results. Could you please have a look at it and see it makes sense or not.

Accuracy

I conducted 2*3 between-subjects ANOVA with Techniques(N,NL and NLG) and Tasks (node-based, network-based and group-based tasks) as factors. I found a significant effect of accuracy on Techniques(F(2,99) =25.2,p<.0001). Overall, NLG is about 8% and 30% more accurate than both NL and N techniques respectively. I also found that accuracy of all tasks in NL diagram is about 22% more accurate than N diagram. The between subjects ANOVA also revealed a significant effect of Tasks (F(2,99) = 18.34, p <.0001), and a significant effect of the interaction Task x Technique (F(4,99) = 14.52, p <.0001). Post hoc comparisons using the Tukey HSD test indicated that there is not a significant difference between any of three techniques for node-based tasks. Results revealed that both NLG and NL are more accurate than N for network-based tasks. I also found that accuracy of NLG is higher than NL and N for group-based tasks and there is not a significant different between NL and N for the group- based tasks.

Time

Between subjects ANOVA with Techniques (N, NL and NLG) and Tasks (node-based, network-based and group-based tasks) as factors. I excluded about 35% incorrect trials for N technique (mostly network-based tasks), 16% for NL diagram and 9% for NLG diagram (out of 972 total trials per technique). I found a significant effect of time for Techniques (F(2,99) = 31.5, p<.001). Overall, both NL and NLG are around 15% faster than N for all tasks. However, this does not provide enough evidence to conclude that NLG and NL are faster than N diagram. The between subjects ANOVA also revealed a significant effect of Tasks (F(2,99) = 125.4, p<.001), and a significant effect of the interaction Task x Technique (F(4,99) = 16.5, p<.05). Post hoc comparisons using the Tukey HSD test indicated that there is not a significant different between any of three techniques for node-based tasks. Results revealed that both NLG and NL are faster than N for network-based. I also found that of NLG is faster than NL and N for group-based tasks and there is not a significant different between NL and N for the group-based tasks. Overall, for group-based tasks NLG is about 31% faster than N and 25% faster than NL diagram.

Bahador Saket
  • 140
  • 6
1

sorry for slow response. I have been away. Analysis generally looks good. You state what your analysis shows in terms of significant main effects, interactions etc. Perhaps you are trying to keep your post short, but you don't always interpret your analysis. This makes it difficult to understand and to spot where you might have done something wrong. You've done a reasonably job of interpreting, but it could be improved. Make sure that every time you say you found a 'main effect' or an 'interaction effect' or you have 'removed data', you describe what this means in plain English. For example you talk about excluding data for Time...but i don't know why you have done this (this might be inappropriate).

in general it look likes than analysis is appropriate and you understand what it means. One thing to bare in mind is that ANOVA is not always appropriate when your outcome is measured as a percentage. If your data was originally in the form of 'correct/incorrect' or zeros and 1s then chi square tests or logistic regression might have been more appropriate (see Using ANOVA on percentages? and Why use a z test rather than a t test with proportional data?). People often use anova on percentage data, so i wouldn't worry too much about this. However, you could comment on the 'correctness' of using ANOVA when you discuss your results.

Community
  • 1
user41270
  • 215
  • 1
  • 3
  • 8
  • Thanks for the response. Please have a look at new question: http://stats.stackexchange.com/questions/92732/within-subject-anova – Bahador Saket Apr 06 '14 at 16:36