SUPTS. MEMO. NO. 126
September 4, 1998
|FROM:||Paul D. Stapleton
Superintendent of Public Instruction
|SUBJECT:||Virginia State Assessment Program Administration Audit Information|
The Department of Education will be conducting audits of the Virginia State Assessment Program (VSAP) Stanford 9 administration slated for the Fall of 1998. A random sample of 10% of Virginia's schools will be audited. There are two purposes for the audit: 1) to observe actual test administration procedures to see whether prescribed, standardized procedures are being followed; and 2) to gather information on ways to improve procedures to maximize efficiency while maintaining standardized procedures and test security. Harcourt Brace Educational Measurement has trained a cadre of auditors who will conduct the audits in Virginia's schools. Auditors, many of whom had experience as auditors during the Spring 1998 administration of the Standards of Learning (SOL) Assessments, were recruited throughout Virginia using a temporary employment agency. Minimal qualifications for auditors include a four-year college degree with a preference for retired educators. Graduate education students enrolled in advanced degree programs within the state have also been hired as auditors. Auditor candidates cannot be employed by,or have any relationship with, school division personnel which might result in a conflict of interest. To qualify to conduct administration audits, each auditor has completed at least one full day of training on the VSAP and on audit procedures. The training was conducted jointly by the Department of Education and Harcourt Brace Educational Measurement. We encourage you to share the enclosed materials with your staff: VSAP Administration Audit - Observation Checklist VSAP Administration Audit - Post-Test Examiner Protocol VSAP Administration Audit - Post-Test School Test Coordinator (STC) Protocol VSAP Administration Audit - Post-Test DDOT Protocol VSAP Administration Audit - Evaluation of the Auditor and Audit Process VSAP Administration Audit - Sample Letter of Introduction VSAP Administration Audit - Audit Information Sheet While the audits themselves will be unannounced, it is not the intent of the Department to surprise anyone regarding what is to be observed or asked. It is important that you share with your staff members and teachers the essence of the audits so that they are aware of the activity should their school be audited. As noted in the enclosed procedures, your Division Director of Testing (DDOT) will be notified by fax a day ahead of any audit to be conducted in the school division. The intent of that notification is to let the central office staff know that an auditor(s) will be present in the school division. However, the notification will not include the name of the school(s) to be audited. The audit procedures call for the auditor to meet with the School Test Coordinator (STC) in the school to be audited, observe the administration of one of the VSAP subtests in one classroom, and conduct post-observation meetings with the observed Examiner and the STC. The principal of the school is encouraged to participate in these meetings. The auditor will also arrange to meet with the DDOT to discuss the audit that has been conducted. In the event that more than one school in the division has been selected to be audited, a single interview will be conducted with the DDOT after all audits in the division have been completed. Auditors have been informed of the need to observe the protocol for visitors that is in effect for each school they visit. Each auditor will carry a letter of introduction on the letterhead of Harcourt Brace Educational Measurement, and will present photo identification upon request. Each auditor will also have a brief one-page audit explanation to provide school staff members who might want further detail. It is our intent that these visits be as unobtrusive as possible. The protocols for the VSAP audit are modified versions of those field-tested in four schools during the Spring 1998 administration of the SOL Writing test. Feedback from auditors, Examiners, STCs, and DDOTs subsequent to the Spring 1998 SOL audits has been used to make the VSAP audit protocols as efficient and useful as possible. While information from the audit will be discussed in the post-observation meetings with the Examiner, STC, and DDOT, a more formal report will be provided to the school division and to the Department within 30 days of the close of the final audit in the state. Auditors will leave copies of all completed audit protocols with the STC and DDOT once the audit is concluded. An audit evaluation form will also be left with each person involved in the audit process in each school. During audits of the Spring 1998 SOL administrations, we observed many useful and innovative practices; sharing of those practices has been a by-product of that audit. We look forward to observing additional such practices during the upcoming VSAP audits. Thank you in advance for your cooperation in this activity. Any questions you might have concerning the VSAP audit procedures can be directed to the Division of Assessment and Reporting at (804) 225-2102. PDS/CMH/atb Enclosures: A hard copy of this memo and its attachments will be sent to the superintendent's office.