Skip to content

Experimental Design, Sampling and Data Analysis

CSIR NET Part A research aptitude: controls, replication, randomisation, blinding, sampling, bias, accuracy vs precision, p-values and test errors.

  • 22 min read
  • Updated
On this page · 14 sections
  1. Prerequisites
  2. The Hook: Three Lanes on One Blot
  3. Controls
  4. The Principles of Experimental Design
  5. Population and Sampling
  6. Measurement
  7. Reading the Numbers
  8. Memory Tricks
  9. Don’t Confuse
  10. Common Mistakes
  11. Practice
  12. Quick Revision Table
  13. Key Takeaways
  14. What’s Next

Prerequisites

The Hook: Three Lanes on One Blot

Controls

A control is a condition that differs from the test condition in only the factor being studied, or that checks the method itself. It is what lets you attribute a difference to the treatment.

ControlWhat it receivesWhat it rules outExample
Negative controlNo treatment, or a treatment known to have no effectA positive result arising from contamination or backgroundNo-template control in PCR
Positive controlA treatment known to give the effectA negative result arising because the assay failedA known antibiotic in a growth-inhibition assay
Vehicle (solvent) controlThe solvent or carrier alone, same volumeAn effect of the solventDMSO-only wells when the drug is dissolved in DMSO
Sham controlThe procedure without the active stepAn effect of handling or surgery itselfSham-operated animals in a lesion study
Placebo controlAn inert preparation that looks identicalEffects of expectation (the placebo effect)Sugar pill in a drug trial
Loading or internal controlA reference measured in the same sampleDifferences in the amount of materialβ-actin band, or a reference gene in qPCR
Wild-type or isogenic controlThe same strain without the mutationDifferences in genetic backgroundParental strain beside a knockout

The Principles of Experimental Design

Three principles, associated with R. A. Fisher’s work on agricultural field trials, are the backbone of design:

PrincipleWhat it meansWhat it protects against
ReplicationApplying each treatment to several independent experimental unitsMistaking chance variation for an effect; it lets you estimate error
RandomisationAssigning units to treatments by a random processSystematic differences between groups (confounding, selection by the researcher)
Local control (blocking)Grouping similar units into blocks and comparing treatments within each blockKnown nuisance variation inflating the error

Biological vs technical replicates

Biological replicateTechnical replicate
What is repeatedIndependent biological samples (separate animals, plants, cultures grown separately)Repeated measurements of the same sample
What it measuresBiological variation, which a conclusion about the population needsMeasurement (assay) variation
Supports a claim aboutThe biologyThe precision of the method

The experimental unit is the smallest unit to which a treatment is independently applied. If a drug is added to a flask and you count 100 cells from that flask, n is 1 flask, not 100 cells. Treating sub-samples as independent replicates is called pseudoreplication, and it makes an effect look far more certain than it is.

Common designs

DesignLayoutWhen to use
Completely randomised design (CRD)Every unit assigned at random to any treatmentUnits are uniform (e.g. a growth chamber)
Randomised block design (RBD)Units grouped into blocks of similar units; each block gets every treatment once, at randomA known gradient (field slope, batch, day)
Latin squareTreatments arranged so each appears once in every row and every columnTwo nuisance gradients at right angles
Factorial designEvery combination of the levels of two or more factorsTo find interactions between factors
Paired / before–after / crossoverEach subject serves as its own controlLarge differences between subjects

Blinding

LevelWho does not know the allocation
Single-blindThe subjects
Double-blindThe subjects and the people giving the treatment or assessing the outcome
Triple-blindThe above and the people analysing the data (usage varies between sources)

In laboratory work, “blind scoring” means the person counting cells or scoring phenotypes does not know which sample is which, usually by coding the samples. It protects against observer bias.

Effects that mimic a treatment effect

EffectWhat happensDefence
Placebo effectSubjects improve because they expect toPlacebo control, blinding
Hawthorne effectSubjects change behaviour because they know they are being observedA control group observed in the same way
Regression to the meanUnits chosen because they were extreme tend to be less extreme when measured again, with no treatment at allA randomised control group selected the same way
Observer (experimenter) biasThe researcher’s expectations shift the measurementBlinding, objective measures

Population and Sampling

The population is the whole set the conclusion is meant to apply to; the sample is the part actually studied; the sampling frame is the list from which the sample is drawn. A parameter describes the population (for example \( \mu \), \( \sigma \)); a statistic describes the sample (\( \bar{x} \), \( s \)) and is used to estimate the parameter.

Sampling methods

graph TD
    A["Sampling methods"] --> B["Probability sampling
every unit has a known,
non-zero chance"] A --> C["Non-probability sampling
chance of selection unknown"] B --> B1["Simple random"] B --> B2["Systematic
every k-th unit"] B --> B3["Stratified
random within subgroups"] B --> B4["Cluster
random whole groups"] B --> B5["Multistage"] C --> C1["Convenience"] C --> C2["Purposive / judgement"] C --> C3["Quota"] C --> C4["Snowball"]
MethodHow it worksStrengthWeakness
Simple randomEvery possible sample of size n is equally likely (lottery, random numbers)Unbiased; basis of most statisticsNeeds a complete list; may miss small subgroups by chance
SystematicRandom start, then every k-th unit, \( k = N/n \)Easy in the fieldBiased if the list has a cycle matching k
StratifiedDivide into non-overlapping strata (e.g. by year of study); random sample within eachEnsures every subgroup is represented; more preciseNeeds stratum information in advance
ClusterDivide into clusters (e.g. villages); randomly select whole clustersCheap when the population is spread outLess precise than a simple random sample of the same size
MultistageRandom clusters, then random units within themPractical for large surveysComplex analysis
ConvenienceWhoever is easy to reachQuickUsually unrepresentative
PurposiveResearcher picks typical or information-rich casesUseful in qualitative workDepends on the researcher’s judgement
QuotaFill fixed numbers per subgroup, by convenienceMatches population proportions on chosen traitsNot random within subgroups
SnowballParticipants recruit further participantsReaches hidden or rare populationsNetworks over-represented

Sample size and sampling error

Sampling error is the difference between a statistic and the parameter that arises only because a sample, not the whole population, was measured. It shrinks as the sample grows, roughly in proportion to \( \frac{1}{\sqrt{n}} \). Non-sampling errors (a biased frame, non-response, faulty instruments) do not shrink with a bigger sample.

Types of bias

BiasSourceExample
Selection biasThe sample is not representative because of how it was chosenStudying only patients who reached a tertiary hospital
Non-response biasThose who reply differ from those who do notA stress survey answered mainly by the least busy
Survivorship biasOnly the units that “survived” a filter are seenJudging a treatment only from patients who lived long enough to be seen in clinic
Recall biasSubjects remember past exposures differently depending on the outcomeCase–control studies of diet
Measurement / instrument biasThe instrument reads consistently wrongAn uncalibrated balance
Confirmation biasLooking for, or weighting, evidence that fits the hypothesisRepeating only the experiments that “failed”
Publication biasPositive results are more likely to be publishedPublished effect sizes exaggerated

Measurement

Scales of measurement

The four scales usually attributed to S. S. Stevens:

ScalePropertiesExamplesMeaningful statistics
NominalCategories, no orderBlood group, species, genotypeCounts, mode, chi-square
OrdinalOrdered categories, unequal or unknown gapsDisease grade I–IV, Likert scale, rankMedian, percentiles, rank tests
IntervalEqual gaps, arbitrary zeroTemperature in °C, calendar yearMean, SD; differences but not ratios
RatioEqual gaps, true zeroMass, length, concentration, temperature in K, countsAll, including ratios (“twice as much”)

20 °C is not “twice as hot” as 10 °C, because 0 °C is not an absence of heat. 20 K is twice 10 K.

Accuracy, precision, errors

TermMeaning
AccuracyCloseness of a measurement to the true value
PrecisionCloseness of repeated measurements to each other
Systematic errorPushes every reading the same way; reduces accuracy; not reduced by repeating; found by calibration against a standard
Random errorScatters readings both ways; reduces precision; its effect on the mean is reduced by repeating
ResolutionThe smallest change the instrument can show

Picture shots at a target:

Tightly grouped (precise)Widely scattered (imprecise)
Centred on the bull’s-eye (accurate)Accurate and preciseAccurate on average, not precise
Off-centre (inaccurate)Precise, not accurate (systematic error)Neither

Reliability and validity

ReliabilityValidity
QuestionDoes the method give consistent results?Does it measure what it claims to measure?
AnalogyPrecisionAccuracy
KindsTest–retest, inter-rater, internal consistencyInternal (is the causal conclusion sound within the study?), external (does it generalise?), construct, content

A measure can be reliable but not valid (a scale that always reads 2 kg high); it cannot be valid without being reasonably reliable.

Reading the Numbers

Centre and spread

StatisticDefinitionNote
Mean\( \bar{x} = \frac{\sum x_i}{n} \)Pulled by outliers
MedianMiddle value of the ordered dataRobust to outliers; use for skewed data
ModeMost frequent valueThe only one for nominal data
RangeMax − minDepends only on two values
Variance\( s^2 = \frac{\sum (x_i - \bar{x})^2}{n-1} \) for a sampleDivide by \( n \) for a whole population
Standard deviation (SD)\( s = \sqrt{s^2} \)Same units as the data; describes spread of the data
Coefficient of variation\( \text{CV} = \frac{s}{\bar{x}}\times 100\% \)Compares spread across different scales
Standard error of the mean (SEM)\( \text{SE} = \frac{s}{\sqrt{n}} \)Describes uncertainty in the mean, not spread of the data

In a right-skewed distribution (a long tail of large values, like income or many cell sizes), mean > median > mode.

For roughly normal data, about 68%, 95% and 99.7% of values lie within 1, 2 and 3 SD of the mean. An approximate 95% confidence interval for the mean is \( \bar{x} \pm 1.96\,\text{SE} \), often rounded to \( \pm 2\,\text{SE} \).

Correlation

The Pearson correlation coefficient \( r \) lies between −1 and +1 and measures the strength of a linear association. \( r = 0 \) means no linear association; a strong curved (for example U-shaped) relation can still have \( r \approx 0 \). The square \( r^2 \) (coefficient of determination) is the fraction of the variation in one variable accounted for by a linear relation with the other: \( r = 0.7 \) gives \( r^2 = 0.49 \), about half.

Simpson’s paradox

A trend that holds in every subgroup can reverse when the subgroups are pooled, if the groups are of very different sizes and a confounder is unevenly spread.

Hypothesis tests and the p-value

A statistical test asks how surprising the data would be if \( H_0 \) were true.

  • The p-value is the probability, assuming \( H_0 \) is true, of getting a result at least as extreme as the one observed.
  • The significance level \( \alpha \) (commonly 0.05) is the threshold chosen in advance. If \( p < \alpha \), \( H_0 \) is rejected.
\( H_0 \) actually true\( H_0 \) actually false
Reject \( H_0 \)Type I error (false positive); probability \( \alpha \)Correct; probability = power \( = 1 - \beta \)
Fail to reject \( H_0 \)CorrectType II error (false negative); probability \( \beta \)

Power rises with a larger sample, a larger true effect, less variability and a larger \( \alpha \).

Multiple testing

If you run many tests, some will come out “significant” by chance alone. With \( m \) independent tests, all of whose nulls are true, at level \( \alpha \):

$$\text{Expected false positives} = m\alpha \qquad P(\text{at least one}) = 1 - (1-\alpha)^m$$

With 20 tests at 0.05: on average 1 false positive, and \( 1 - 0.95^{20} \approx 0.64 \), a 64% chance of at least one. The Bonferroni correction uses \( \alpha/m \) for each test (here \( 0.05/20 = 0.0025 \)). Running many analyses and reporting only the significant ones is called p-hacking or data dredging.

Diagnostic and screening tests

Condition presentCondition absent
Test positiveTrue positive (TP)False positive (FP)
Test negativeFalse negative (FN)True negative (TN)
MeasureFormulaAnswers
Sensitivity\( \frac{TP}{TP + FN} \)Of those with the condition, what fraction test positive?
Specificity\( \frac{TN}{TN + FP} \)Of those without it, what fraction test negative?
Positive predictive value (PPV)\( \frac{TP}{TP + FP} \)If I test positive, how likely am I to have it?
Negative predictive value (NPV)\( \frac{TN}{TN + FN} \)If I test negative, how likely am I to be free of it?

Sensitivity and specificity are properties of the test. PPV and NPV also depend on the prevalence: the rarer the condition, the lower the PPV, even for a good test.

Memory Tricks

Don’t Confuse

ThisvsThat
Biological replicate: independent sampleTechnical replicate: same sample measured again
Negative control: checks for false positivesPositive control: checks the assay can detect an effect
Accuracy: near the true valuePrecision: near each other
Systematic error: one direction, not removed by repeatingRandom error: both directions, averages out
Stratified: random units from every stratumQuota: fill numbers per group by convenience
SD: spread of observationsSE: uncertainty of the mean
Sensitivity: P(test + given condition)PPV: P(condition given test +)
Statistical significancePractical importance

Common Mistakes

Practice

All questions below are original practice questions written for this page. They are not from any CSIR NET paper.

Level 1: Foundation

  1. A PCR reaction with all reagents but no template DNA is what kind of control?
  2. On which scale of measurement is a disease graded as mild, moderate or severe?
  3. A sample has SD 10 and n = 25. What is the standard error of the mean?
Answers
  1. Negative control (no-template control). A band here means contamination.
  2. Ordinal. The categories are ordered, but the gaps between them are not known to be equal.
  3. 2. \( \text{SE} = \frac{10}{\sqrt{25}} = \frac{10}{5} = 2 \).

Level 2: Part A Style

Practice Q1. A researcher dissolves a plant extract in DMSO and finds that it kills 60% of cancer cells in culture, compared with 5% death in cells grown in medium alone. The most important missing control is:

(a) cells treated with a known anticancer drug (b) cells treated with the same volume of DMSO alone (c) a second batch of the plant extract (d) normal (non-cancer) cells in medium alone

Answer: (b)

DMSO can itself be toxic to cells. Without a DMSO-only control, the 55-point difference could be due to the solvent. (a) is a positive control; useful, but this result is already positive, so it does not explain it. (d) answers a different question (selectivity).

Practice Q2. A survey team lists all 240 villages in a district, randomly selects 12 villages, and interviews every household in those 12 villages. This is:

(a) stratified sampling (b) cluster sampling (c) systematic sampling (d) quota sampling

Answer: (b) cluster sampling

Whole groups (villages) are chosen at random and every unit in them is studied: “every from some”. In stratified sampling, some households would be chosen from every village.

Practice Q3. A balance gives readings of 10.31, 10.30, 10.32 and 10.31 g for a certified 10.00 g mass. The balance is:

(a) accurate and precise (b) precise but not accurate (c) accurate but not precise (d) neither

Answer: (b)

The readings agree within 0.02 g (precise) but all are about 0.31 g high (a systematic error, so not accurate). Calibration, not more readings, is the fix.

Practice Q4. A research group runs 40 independent tests, in each of which the null hypothesis is actually true, at \( \alpha = 0.05 \). The expected number of “significant” results is:

(a) 0 (b) 1 (c) 2 (d) 4

Answer: (c) 2

Expected false positives \( = m\alpha = 40 \times 0.05 = 2 \). The probability of at least one is \( 1 - 0.95^{40} \approx 0.87 \). A Bonferroni threshold would be \( 0.05/40 = 0.00125 \) per test.

Level 3: Multi-Step (Part C depth)

Practice Q5. A study compares four fertiliser doses on wheat. The field slopes from north to south, and soil moisture changes with the slope. The researcher divides the field into five east–west strips, and within each strip assigns the four doses at random to four plots.

  1. This is a randomised block design with 5 blocks.
  2. The strips are the experimental units.
  3. The design removes the effect of the slope from the comparison of doses.
  4. There are 20 plots, so each dose is replicated 5 times.

Which statements are correct?

(a) 1 and 4 only (b) 1, 3 and 4 only (c) 2 and 3 only (d) 1, 2, 3 and 4

Answer: (b) 1, 3 and 4 only
  • 1 is correct. Each strip (block) contains every treatment once, assigned at random.
  • 2 is wrong. The plot is the unit to which a dose is applied; the strip is a block.
  • 3 is correct. Moisture is roughly constant within an east–west strip, so comparing doses within strips removes the slope gradient from the comparison (and the analysis separates block variation from error).
  • 4 is correct. \( 5 \times 4 = 20 \) plots; each dose appears once per block, 5 times in all.

Practice Q6. A screening test for a condition has sensitivity 90% and specificity 95%. The condition affects 1% of the population screened. If a randomly chosen person tests positive, the probability that they actually have the condition is closest to:

(a) 90% (b) 50% (c) 15% (d) 5%

Answer: (c) about 15%

Take 10,000 people. With the condition: 100. Without: 9,900.

  • True positives: \( 0.90 \times 100 = 90 \).
  • False positives: \( 0.05 \times 9{,}900 = 495 \).
  • PPV \( = \frac{90}{90 + 495} = \frac{90}{585} \approx 0.154 \), about 15%.

Most positives are false because the condition is rare. Option (a) is the sensitivity, the classic trap. A confirmatory test on the positives is the usual remedy.

Practice Q7. Two labs measure the same enzyme’s activity (true value unknown) in five independent preparations each, in µmol/min/mg:

  • Lab P: 4, 8, 6, 5, 7
  • Lab Q: 6.1, 5.9, 6.0, 6.2, 5.8

(i) Find the mean and the sample SD for each lab. (ii) Which lab is more precise? (iii) Can you say which lab is more accurate?

Answer

(i) Lab P: mean \( = \frac{30}{5} = 6 \). Deviations −2, 2, 0, −1, 1; squares sum to \( 4 + 4 + 0 + 1 + 1 = 10 \). Sample variance \( = \frac{10}{4} = 2.5 \), SD \( = \sqrt{2.5} \approx 1.58 \).

Lab Q: mean \( = \frac{30.0}{5} = 6.0 \). Deviations 0.1, −0.1, 0, 0.2, −0.2; squares sum to \( 0.01 + 0.01 + 0 + 0.04 + 0.04 = 0.10 \). Variance \( = \frac{0.10}{4} = 0.025 \), SD \( \approx 0.158 \).

(ii) Lab Q is far more precise (SD ten times smaller; CV about 2.6% vs 26%).

(iii) No. Accuracy needs a known true value or a reference standard. Both means are 6.0, but they could share the same systematic error.

Quick Revision Table

IdeaKey point
Negative / positive controlRules out false positives / false negatives
Vehicle controlSame solvent, same volume, no active agent
Fisher’s principlesReplication, randomisation, local control (blocking)
Experimental unitSmallest unit independently given a treatment; sub-samples are not replicates
FactorialNumber of combinations = product of the levels; reveals interactions
Stratified vs clusterSome from every group vs every unit from some groups
SystematicEvery k-th unit, \( k = N/n \)
ScalesNominal, ordinal, interval (arbitrary zero), ratio (true zero)
Accuracy vs precisionNear true value vs near each other; systematic vs random error
SE\( s/\sqrt{n} \); quadruple n to halve it
p-valueP(data this extreme or more | \( H_0 \) true)
Type I / IIFalse positive (\( \alpha \)) / false negative (\( \beta \)); power \( = 1 - \beta \)
Multiple testsExpected false positives \( m\alpha \); Bonferroni \( \alpha/m \)
PPVFalls as prevalence falls

Key Takeaways

What’s Next