ISCAM: Investigating Statistical Concepts, Applications, and Methods

The authorship of literary works is often a topic for debate. Were some of the works attributed to Shakespeare actually written by Bacon or Marlowe? Which of the anonymously published Federalist Papers were written by Hamilton, which by Madison, which by Jay? Who were the authors of the writings contained in the Bible? The fields of "literary computing" and "forensic stylometry" examine ways of numerically analyzing authors’ works, looking at variables such as sentence length and rates of occurrence of specific words.

In this investigation you will examine data for just 10 of the 268 words in the Gettysburg Address. We are considering this passage to be a population of 268 words, and the 10 words you selected are therefore a sample from this population. Now we define the sample as a collection of individuals or observational units that have distinct characteristics, rather than a finite set of repeated realizations of a random process.

In most statistical studies, we do not have access to the entire population and can consider only data for a sample from that population. Our ultimate goal is to make (appropriate) conclusions about the larger population, based on only the sample data.

Up until now, we have generally been assuming we have a representative sample from an infinite on-going process (e.g., toast drops, hospital transplant operations, candy manufacturing). We made some assumptions, like the process is not changing over time and that there is no tendency to select some types of outcomes more than others (e.g., getting the first 5 candies from the manufacturing process rather than throughout the day). In fact, a binomial process assumes you have repeat observations from the exact same process but with randomness in the actual outcome that occurs (sometimes an infant chooses the helper toy, sometimes not; the randomness is in the choice).

In this investigation, instead of sampling from an on-going process we are sampling from a finite population (the 268 words). In fact, we actually have access to the entire population. But what if we didn’t? We still need some way of convincing people that our sample is likely to be representative of the population. Let’s use this population to explore how samples behave when the "random chance" arises from which observational units are selected to be in the sample, rather than from "random choices" made by the observational units.

Discussion: You have again witnessed the fundamental principle of sampling variability: Values of sample statistics vary when one takes different samples from the same population. The distribution of statistics from a repeated sampling is often called a sampling distribution. In this case, we are treating each sample as an observational unit and the sample statistics as the variable of interest (so the graph label should be something like "sample proportion short"). Although we clearly expect there to be sample-to-sample variation in the statistic, there is a problem if there is a tendency to over or underestimate the parameter of interest.

For example, we suspect that your class repeatedly and consistently underestimated the proportion of short words. Not everyone has to underestimate, but if there is a tendency to err in the same direction time and time again, then the sampling method is biased. In other words, sampling bias is evident if we repeatedly draw samples from the population and the distribution of the sample statistics is not centered at the population parameter of interest. Note that bias is a property of a sampling method, not of a single sample. Studies have shown that human judgment or "convenience sampling" (e.g., selecting the most readily available observational units) is not a good basis for selecting representative samples, so we will rely on other techniques to do the sampling for us.

The first step is to obtain a list of every member of your population (this list is called a sampling frame). Then, give each observational unit on the list a unique ID number.

You can view the numbered sampling frame for the passage, with each word in the population numbered. You can use technology to select a 3-digit number at random, and then match it to the word in the population with that ID number. Then repeat to randomly select additional (unique) IDs (words).

ids = 1:268

sample(ids, 5)

Instead, we will use the Sampling Words applet.

Of course, to really see the long-term pattern to the sampling distribution of the statistic, we would like to take many more random samples from this population.

In the Sampling Words applet,

Discussion: Even with randomly drawn samples, sampling variability still exists (random sampling errors). Although not every random sample produces the same characteristics as the population, random sampling has the property that the sample statistics do not consistently overestimate the value of the parameter or consistently underestimate the value of the parameter (which is not changing). The means of the above distributions of sample proportions (the statistics) are both very close to 0.41, the population proportion (the parameter). Because the distributions of the sample statistic (as a random variable) are centered at the value of the population parameter, both graphs are considered to demonstrate an unbiased sampling method. With random samples, \(E(\hat{p}) = \pi\text{,}\) even when sampling from a finite population.

Notice that in judging bias we are concerned only with the mean of the distribution, not the shape or variability. However, if we do take larger samples (\(n\) = 10 rather than \(n\) = 5), we improve the precision of our statistics as they will tend to cluster even more tightly around the population proportion. In designing a study, it is critical to first make sure there is no sampling bias (use random sampling), then the sample size (not the number of samples) will determine how close the statistic can be expected to be to the parameter.

Simple random samples are not as simple as they sound. They require a complete sampling frame and 100% response rate. This can be quite a challenge, especially if you are interested in individuals throughout the country for example. There are other probability sampling methods that are typically unbiased. See this discussion for a brief introduction to cluster sampling, multistage sampling, stratified sampling, and systematic sample. These techniques will come up later in the text and can be much more practical to carry out.