Validity and Reliability

Interpretation of scores from the DASE ultimately involves predictions. Perhaps a comprehensive design assessment instrument can estimate a student’s behavior, attitude, self-concept in a given school setting1. Assuming the instrument is standardized, conceivably it will become a valid and reliable predictor of student outcomes and average standardized test scores in a specified school2.

If the instrument is an accurate predictor, it is said to have good validity. Before validity can be demonstrated, an instrument must first yield consistent, reliable measurements. In addition to reliability, psychologists recognize three main types of validity:

1. An instrument has content validity if the sample of items in its content is representative of all the relevant items that might have been used. Words included in a spelling test, for example, should cover a wide range of difficulty. Descriptors in the DASE should cover a wide range of design patterns.

2. Criterion-related validity refers to an instrument’s accuracy in specifying a future or concurrent outcome. For example, a mathematics-aptitude test has predictive validity if high scores are achieved by those who later do well in mathematics. A school scoring high on the DASE is expected to influence behavior and learning in a positive manner; and the converse is also true. The concurrent validity of the instrument may be demonstrated if its scores correlate closely with well established tests such as the Iowa Test of Basic Skills. If a school has a high score on the DASE, it is assumed that ITBS scores will also be high.

3. Construct validity is generally determined by investigating what qualities an instrument measures; in this case, by demonstrating that certain design patterns account for some degree of a school’s quality. If DASE can be shown to predict quality of the total school design, for instance, then it may also predict quality of student outcomes. The validation task is to be accomplished through several trials such as reported here3.

The First Two Tests of Reliability

In the Fall of 1999, a sample of 15 educators trained in school design made an assessment of one elementary school by using the version of the DASE depicted in the 51 items above. Two weeks later, they were asked to repeat the assessment. These data were coded and analyzed for internal consistency to create a reliability coefficient known as Chronbach’s Alpha (? ). The index created for the instrument in question is based on the average correlation among items within the instrument, where responses are standardized to a standard deviation of 1. The results are shown in Table 1.

In Section I, there are few problems as uncovered by the analysis. Refinement, of item number 18 (technology for students) will improve reliability. Section II, degree of safety also needs some refinement. Additional areas may be added. Given the item analysis, the question of bathrooms in each classroom may need to be stated differently or eliminated. While having bathrooms in every classroom is ideal, the financial feasibility may be questioned (personally I prefer this because of safety and security factors). Section III revealed no major difficulties. The degree of quality, Section IV, disclosed the least variation between the two trials. No items need to be eliminated in the fourth section. A review of the coefficients for Section V indicates that some change is needed, particularly the addition of some more global patterns. The statements concerning “animal life” need refinement.

Table 1. Tests of the scale’s reliability

* Alpha for all 51 items. If Item 51 were deleted, the correlation would be .75 and .91, respectively, indicating that it is an important item.

Correlation between items 51 and the total score were .72 and .71 respectively. Item 51, while being important may need refinement.

Conclusions

Sections II, V, and VI need revision. They reveal concerns for clarification, perhaps more content validity. On the other hand, Sections I, III, and IV could stand as they are. During the next phases of development, all sections will probably take on some changes of content and clarity. While the standardized alpha for the entire scale, as shown for Test-2, was .91, the internal parts (II, V, and VI) are in need of improvement. Furthermore, given the nature of reliability estimation models, larger numbers of items have a tendency to yield larger coefficients.

Let us go back to the original questions: Does the DASE include the most significant and valid aspects of design for elementary schools? [Answer: Not completely, some sections need refinement]. Will it consistently measure these important design patterns? [Answer: Yes, when validity is improved].

Notes:

1. Remember it was Skinner who said that environment is a behavior modifier and Lewin who taught that behavior is a function of the field that exists at the time the behavior occurs.

2. It is not for me to say that this cannot be done. To me, it is within the realm of possibility to predict the average set of scores from a well-defined population with a ‘finely tuned’ instrument. The argument against this is that there are too many uncontrolled social and economic variables in public schools. However, we are talking about averages of scores for a complete school setting, not individual scores - averages of averages in some cases. Relate this problem to what marketers do all the time. If a sample of 1000 people are driving new Jaguar cars, then the estimated average personal income of the driver is greater than or equal to $100,000.00 per year (We may also assume that these people are college graduates who made above average scores on the ACT or SAT). Analogies may be made for drivers of sub-compact cars, for drivers of pick up trucks, types of desks sold to certain schools, and for owners of about any item sold. So, if predictions can be made about all these commercial objects, why cannot we make them about what people (groups representing certain social classes) buy in the name of educational facilities. Given a certain area of the world, can’t we predict the type of schoolhouse that a certain society will buy for their children? Then, by inference from school design, measures of behavior, attitude, and self-concept can’t we make a probability statement how these students will fare on standardized tests - cognitive learning (on the average)?

3. The amount of time and resources to construct an assessment scale are enormous. The SDPL, a nonprofit unit of the University of Georgia welcomes supports for research and development. SDPL’s web site may be found at [ http://www.coe.uga.edu/sdpl/sdpl.html ].

Bibliography

Achilles, C. M., Finn, J. D., & Bain, H. P. (1998). Using class size to reduce the equity gap. Educational Leadership. 55(4), 40-43.

Alexander, C. (1979). The Timeless Way of Building. New York: Oxford University Press.

Alexander, C., Ishikawa, S., & Silverstein, M. (1977). A Pattern Language. New York: Oxford University press.

Bingler, S. (1995). Place as a form of knowledge. In A. Meek (Ed.), Designing Places for Learning (pp. 23-30). Alexandria, VA: ASCD.

Brubaker, C. W. (1998). Planning and Designing Schools. New York: McGraw-Hill.

David, T. G., & Weinstein, C. S. (1987). The built environment and children’s development. In C. S. Weinstein and T. G. David (Eds.), Spaces for Children: The Built Environment and Child Development (pp. 3-40). New York: Plenum Press.

Ferguson, G. A. (1981). Statistical Analysis in Psychology and Education (5th. Ed.). New York: McGraw-Hill.

Fielding, R. (1998, November). An interview with Cunningham Group’s Bruce Jilk. [26 paragraphs]. Design Share [http://www.designshare.com/index.php/articles/interview-bruce-jilk/]

Freeman, C. (1995). Planning and play: Creating greener environments, Children’s Environments. 12(3), 381-388.

Hughes, P. C. (1980). The use of light and color in health. In A. C. Hastings, J. Fadiman, & J. S. Gordon (Eds.), Health for the Whole Person: The Complete Guide to Holistic Medicine (pp. 71-83.). Boulder, CO.: Westview Press.

Meek, A. & Landfried, S. (1995). Crow Island School: 54 Years Young. In A. Meek (Ed.), Designing Places for Learning (pp. 51-59). Alexandria, VA: ASCD.

Moore, G. T. (1987). The physical environment and cognitive development in child-care centers. In C. S. Weinstein and T. G. David (Eds), Spaces for Children: The built environment and child development. (pp. 41-72). New York: Plenum Press.

Moore, G. T., & Lackney, J. A. (1995). Design patterns for American schools: Responding to the reform movement. In A. Meek (Ed.), Designing Places for Learning (pp. 11-22). Alexandria, VA: ASCD.

Ott, J. (1973). Health and Light. New York: Simon & Schuster.

Prescott, E. (1987). Environment as an organizer in child-care settings. In C. S. Weinstein and T. G. David (Eds), Spaces for Children: The Built Environment and Child Development (73-88). New York: Plenum Press.

Rogers, C. R. (1961). On becoming a person. Boston: Houghton Mifflin.

Raywid, M. A. (1998). Small schools: A reform that works. Educational Leadership. 55(4), 34-39.

Sanoff, H. (1994). School Design. New York: Van Nostrand Reinhold.

Schein, E. H. (1997) Kurt Lewin’s Change Theory in the Field and in the Classroom: Notes Toward a Model of Managed Learning [Online - https://dspace.mit.edu/bitstream/1721.1/2576/1/SWP-3821-32871445.pdf]

Skinner, B. F. (1948). Walden Two (1976). New York: MacMillan Publishing Co., Inc. [Online: http://www.lafayette.edu/allanr/walden.htm

Taylor, A. P., & Valastos, G. (1975). School Zone: Learning Environments for Children. New York: Van Nostrand Reinhold Company.

Wohlwill, J. F., & van Vliet, W. (1985). Habitats for Children: The Impacts of Density. Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers.

Wurtman, R. J. (1975). The effects of light on the human body. Scientific American, 233(1), 68-77.

Kenneth Tanner
Professor, School Design & Planning Laboratory
Department of Educational Leadership
The University of Georgia
310 River’s Crossing, Athens, GA 30602
(706) 542-4067 ktanner@coe.uga.edu
http://www.coe.uga.edu/sdpl/sdpl.html

Pages: 1 2 3 4