Step 1
Formulate the research question
Draft it as: What are the effects of X on Y under conditions Z?
Can you name X, Y, and Z?
NoTheory may be nascent: consider an exploratory or qualitative study first (Week 3, methodological fit)Yescontinue
Does it pass FINER?
CheckFeasible · Interesting · Novel · Ethical · RelevantNonarrow or reframe the RQ
the RQ, with X, Y, and Z named
Step 2
Identify variables
List the IVs (factors) with their levels, and the DVs
For each IV: can you assign people to its levels?
Yesmanipulated factorNomeasured factor (e.g., expertise): that part is quasi-experimental; treat it as a covariate
How many factors?
Onesingle-factor designTwo+factorial design (recommended): all combinations; 2k is best
Name the random factors (e.g., participants, stimuli)
a factor × level table; DVs; random factors
Step 3
Generate hypotheses
One per predicted effect, main or interaction: Y will be higher (or lower) at level 1 of X than at level 2, under Z
Is it justified?
Fromexploratory results, related theory, or reasoned face validityNodrop it, or pilot first
Is it testable and falsifiable?
Nomake it more precise
H1, H2, … each with its justification
Step 4
Determine the design (per factor)
Severe transfer effects or demand characteristics expected?
Yesbetween-participantsNonext question
Too many conditions or participants for a between design?
Yeswithin-participants
High variance between people (e.g., performance measures)?
Yeswithin-participants: more power, fewer peopleNobetween-participants
Do your factors land on different sides?
Yesmixed-model (split-plot)
Within factors: counterbalance (Latin square). Asymmetric transfer? Make that factor between.
within / between / mixed, factor by factor
Step 5
Develop the task and procedure
A task that represents the real-world context Z: relevant, controllable, motivating
Does the effect need extra load to show?
Yesadd a distractor task
Would knowing the purpose change what people do?
Yescover story (deceptive task): IRB approval and a debrief
Write the procedure: instructions, roles, experimenter actions, equipment, timeline (consent → measures → compensation)
Could another lab replicate it from this?
Noadd detail
the task and a step-by-step procedure
Step 6
Determine manipulations and measurements
For each manipulated factor: the treatment levels and a control level
What should the control give you?
Floora baseline (lower bound)Ceilinga gold standard (upper bound)
Measure everything you don't manipulate: performance · behavior · self-report · physiological · demographics
Does each measure fit its construct, reliably?
Nochoose another measure
a conditions table; a measure for every DV and covariate
Step 7
Identify participants
Define the population from Z; plan recruitment
Representative of the target population?
Norevise recruitment
Large enough for statistical power? (power analysis)
Nogo within (Step 4) or cut factors (Step 2)
Balanced on measured factors?
Norecruit to balance
a sampling plan and a target N
Done: a complete design
Seven written pieces. Bring them to the Method-Design Crit (Fri Oct 16).
From the CS 770 lecture on step-by-step experimental design. Back to the schedule