Survey Options & Indicators

Tell Them From Me (TTFM) is an on-line evaluation system designed by Dr. Douglas Willms and Patrick Flanagan that allows students, teachers and parents to express their views and participate in school-wide evaluation in a non-threatening way. It provides a means for the continuous monitoring of student outcomes and schooling processes, based on the best evidence from numerous studies on school effectiveness (Scheerens, 1992; Willms, 1992) and wholeschool approaches to school reform (Hill & Crevola, 1999). It is designed for schools and school districts that are engaged in the process of school reform that Fullan (2003) calls "transformative change." This requires a concerted effort by school administrators, teachers, parents and students to change the context of teaching and learning. Tell Them From Me provides continuous feedback on the most important school processes that define this context, thereby enabling district and school staff to focus their efforts. TTFM also offers Teacher and Parent surveys.

Student Survey

Approach
The Student Survey takes 20 - 30 minutes to complete. Participation is voluntary; however, all students should have the option to complete the survey. While some schools offer incentives for students to complete the survey (in addition to missing a class period), schools report that giving feedback to students that their voices have been heard is one of the best incentives, and is an essential aspect of encouraging on-going student engagement with the Tell Them From Me (TTFM) survey.

All students are asked to answer an open-ended question about what they like or dislike about their school. The school personnel have the option of adding up to two additional open-ended questions, and they can change those questions over the course of the year.

On-line Indicator Reports and responses to the open-ended questions are provided to the school personnel on a regular basis. Reports for each school are also provided to the district personnel; however, district personnel do not have access to responses to the open-ended questions.

We offer two approaches to surveying students: TTFM Essentials and TTFM Continuous Monitoring.

1. TTFM Essentials

TTFM Essentials provides a one-time snapshot of student perceptions (Grades 6-12), presented in easy-to-read bar charts, with national lines for comparison of results with all schools in our system, and with your school's results from the previous year where available. TTFM Essentials uses the same measures and includes all the strength and depth of the TTFM Continuous Monitoring School Snapshot surveys (see below) with the exception of the "replica school" comparison line. Results can be drilled-down by gender and grade level.

All students are invited to do the survey over the course of up to four weeks. Results are available on-line immediately after the snapshot period. All results are anonymous.

The 21 indicators of student engagement, wellness, and school climate can be used in a school, district or province-wide monitoring system alongside academic achievement to provide a comprehensive assessment of schooling outcomes and processes. There are also open-ended questions with an option for schools to add their own question.

We are offering TTFM Essentials to schools for $699, regardless of the number of students to be surveyed in the school. This requires no student data upload, no software installation, and minimal initial orientation. A 60-minute on-line school debrief is available for an additional fee of $199, or a customized on-site debrief for multiple schools can be priced separately.

2. TTFM Continuous Monitoring

TTFM Continuous Monitoring requires the district to upload some student data to our system, including name, grade, gender and postal code. It allows for customized data drill downs, and "replica school" lines for comparing results from a "school" of students with similar characteristics. As the name implies, this approach allows more frequent and on-going monitoring of the selected student outcomes

TTFM Continuous Monitoring offers three Student Survey options for schools to choose from:

Random Selection - weekly results This is our preferred approach. When student data are uploaded, the TTFM system randomly assigns each student to one of the viable survey weeks identified by the school. Weekly, the school receives a list of students to complete the survey on-line the following week. Each student completes the survey once. Trajectory reports are updated daily, and results are available after about four weeks of student inputting. Open-ended Question responses are accessible verbatim to schools as long as the minimum number of responses to maintain anonymity has been reached (15). New responses are made available to schools on the first day of each month.

This option is most often used by medium-size schools (300 - 800 students). It is potentially the most labour intensive as it requires a system-generated random selection of students be organized each week to do the survey, following a schedule created by the TTFM system. However, it also gives you the most up-to-date information about your school. Given that students are selected from different classrooms throughout the school each week, schools have had to find ways to minimize bottlenecks and disruption. Larger schools (and in particular, high schools) have tended towards Option 2 below - Classroom Selection.

Classroom Selection - monthly results This approach was designed for large schools. Schools with enrollments above 600 may find it challenging to implement the weekly Random Selection option. These larger schools can choose to have their students complete the survey by homeroom class, or by a specific subject classroom or a "guidance" homeroom, during school time.

This Classroom Selection option has one or more classrooms per grade level complete the survey in the same calendar month. The classroom unit chosen for survey purposes needs to be one that stays together for the whole year. To make this approach work best, we ask schools to ensure that:

over the course of any calendar month, all participating grades have an equal number of classes complete the survey;

an effort is made to get a balanced mix of student ability levels if classrooms are streamed for ability; and

options are considered to allow all students an opportunity to participate in the survey, whether at school or from their home.

If it is still determined to be too ambitious to have all students take the survey at school, and in keeping with the principle that the voices of all students are important, students in a proportion of the school's classrooms can be surveyed at school, and the remainder can be invited to complete the survey as a "Start-up Snapshot" at the first of the year, or on their own time at school or at home.

Schools can choose in which months to do the survey, and how many classes will complete it each month. So, for example, a large high school with grades 9-12 may have as few as 4 classrooms (one per grade level) complete the survey during a calendar month. This is basically a class period. They might do this 3 or 4 months during the year, or every month if they so choose.

School Snapshot - twice-yearly results This option is intended for smaller schools, with enrollments of less than 300 students. All students are invited to do the survey over the course of up to four weeks. Results are available immediately after the snapshot period. Schools can elect to do the survey on two occasions during the school year, usually Fall and Spring.

The TTFM system shifts the burden of data collection and analysis from the schools to our system. However, data and information obviously does not equal improvement. It still requires someone to consider what this means for the school or district, in what priority areas should we put our resources and efforts, what should we do about what we're learning, etc. So TTFM does not "do the job for you." It simply helps you to be clear about priority areas for improvement within your schools and district. It is conveniently used to begin relevant conversations with students, staff and parents, and helps you monitor the impact your efforts are having to ameliorate school climate, student engagement and student wellness.

Student Measures

The student survey provides data on 21 separate indicators in three domains: student engagement, student health and wellness, and school and classroom climate.

A. Student Engagement. Researchers use the term "engagement" as a global measure to refer to the extent to which students identify with and value schooling outcomes, and participate in academic and non-academic activities. Engagement is related to academic achievement, but we consider it to be an important outcome in its own right. It embodies a disposition towards learning, working with others, and functioning in a social institution (Willms, 2003). In the long run, it may be more important than academic achievement in determining people's economic success and their health and well-being. Moreover, student engagement can be affected by parents and teachers, and shaped by school policy and practice.
Tell Them From Me measures student engagement based on five indicators: sense of belonging at school, participation in school clubs and sports, educational aspirations, school attendance, and valuing school outcomes. In addition, Tell Them From Me tracks the amount of time students devote to homework, leisure reading, television, and interactive communication and technology (e.g., computers, video games).

The indicators are described below:

Sense of belonging at school pertains to students' "attachment to school, which has to do with feelings of being accepted and valued by their peers, and by others at their school" (Willms, 2003). It is a key element of the broader Composite Measure, student engagement. Research based on the Programme for International Student Assessment (PISA) found that about one-fifth to one-quarter of students in North America lacked a strong sense of belonging at school. Feelings of being detached or disaffected from school were apparent even among students with strong academic performance. Students with a poor sense of belonging at school are less likely to value schooling outcomes, and in the longer term less likely to undertake further education. The Tell Them From Me scale for sense of belonging includes seven items. Students are asked to indicate their agreement or disagreement with statements such as: "At school I feel accepted for who I am" or "I get along well with others at school."
Participation in school clubs and sports is an important element of school life for many students. Tell Them From Me asks students two questions concerning their regular involvement in this way.
Students' long-term aspirations and expectations are shaped by their perceptions of their ability, their experiences at home and at school, and the opportunities presented to them. School-based research on aspirations and expectations has emphasized differences among youth in the resources available to them, and the structural features of the schooling system that allocates rewards according to social class and ethnicity. Tell Them From Me asks students about whether they expect to complete secondary school and whether they plan to pursue some form of post-secondary education.
School and class attendance is an important indicator of student engagement. Although most schools track school and class attendance, the approach varies somewhat across schools and the data cannot be easily incorporated into a composite measure. Tell Them From Me assesses attendance with regard to whether students miss school without permission, skip classes, or arrive late for school or classes. The format is similar to items used by Willms (2003) in an international study of engagement, and therefore results can be compared to national and international norms.
The value placed on schooling outcomes is another key component of student engagement. Many students believe that education will benefit them personally and economically, while others feel that academic success will have a strong bearing on their future. Tell Them From Me asks students to indicate their agreement with statements like: "I am learning the skills I will need for success later in life" and "The mathematics I am learning will be very useful in the future."

B. Student Health and Wellness. The emotional, social, spiritual and physical health of children and youth is closely tied to their engagement in school life and their learning outcomes. Tell Them From Me includes six indicators of student health and wellness.

Self esteem refers to how much a person likes and accepts herself or himself. In many respects, it is a measure of self-respect. Researchers have studied self-esteem and its relationship with schooling outcomes for several decades. The findings have consistently shown that youth with higher levels of self-esteem tend to have greater academic success, better mental health, and a stronger ability to cope with negative life events. Self-esteem is also related to students' persistence and motivation in academic tasks. Tell Them From Me measures self esteem with seven items, based on the widely used scale developed by Herbert Marsh. For example, students are asked to assess themselves in such areas as: "When I do something, I do it well" and "Overall, I have a lot to be proud of."
Time Spent on Physical Activity is measured in hours per week. Recent research has shown that the prevalence of child and adolescent obesity in Canada has increased markedly over the past two decades (Willms, Tremblay, & Katzmarzyk, 2003), and to some extent the change is associated with physical inactivity (Tremblay & Willms, 2003). Our measure asks students how much time they spend on weekdays and on weekends doing a physical activity that increased their heart rate or made them out of breath some of the time (e.g., running, fast walking, biking, swimming, dancing, rollerblading, or playing a sport like hockey, soccer, or basketball).
Student Bullying refers to situations when a person tries to hurt another person, and does it more than once. Bullying can be physical, verbal, or social, and can also take place over the Internet with emails or text messages. Sometimes a group of students will bully others. In Tell Them From Me students are provided with a definition of bullying and then asked about the frequency with which they are bullied in any of the above four ways.
Depression is a mental state characterized by feelings of sadness, discouragement and inadequacy that persist for long periods, from two or three weeks to several years. Many adolescents suffer from depression to the extent that it affects their ability to concentrate, their appetite, and their sleeping patterns. Typically these youth are unable to experience joy and happiness in activities at school or at home. In Tell Them From Me, students respond to seven items that are indications of depression, such as "A lot of things seem to bother me" and "Other students seem to have more fun than me." An eighth item, "I have trouble falling asleep at night", pertains to both depression and anxiety.
Anxiety involves feelings of fear, intense anxiety, and worry about particular events or social situations. Children and adolescents who experience anxiety can exhibit a range of physical symptoms including dizziness, nausea, heart palpitations, a dry mouth, sweating or feelings of panic. For youth who have an anxiety disorder, these feelings are chronic and can become worse if there is no intervention. Students are asked in Tell Them From Me to respond to seven items that are indicators of anxiety; for example: "I am concerned about what other students think about me" and "I worry more than most kids." The item, "I have trouble falling asleep at night", is also included in the anxiety scale.
Smoking is a regular habit among about 18% of Canadian youth, with 11% smoking occasionally and 7% smoking regularly. The prevalence of teen smoking varies among provinces, from 13% in British Columbia to 24% in Quebec. The prevalence in Atlantic Canada is about 2 percent above the Canadian average (Health Canada, 2004).

C. Classroom and School Climate. The rate at which students learn new a new set of skills depends on several factors: students' aptitude for learning skills at a particular level of difficulty, their motivation and perseverance, the level and quality of instruction, the amount of instructional time devoted to learning the new skills, and the quality of instructional resources. These factors interact with each other; if any one of them is deficient, the rate of learning is slower. The Tell Them From Me student questionnaire assesses five indicators of the learning climate of the school and classroom associated with student learning. Together they provide a composite measure of the learning climate in the school. The five indicators, which are described below, have been based on over twenty years of research on school and classroom effectiveness.

Effective learning time measures three important aspects of classroom teaching: the extent to which important concepts are taught and understood, the efficiency with which class time is used, and the degree to which course objectives are aligned with homework assignments and evaluation procedures. For example, students are asked to indicate the extent to which they agree or disagree with these statements about their school: "Important concepts and ideas are taught well", "Scheduled class time is used efficiently", and "Assignments help me understand the concepts taught in class."
The Disciplinary Climate of the Classroom refers to the extent to which pupils internalize the norms and values of the classroom, and conform to them. The disciplinary climate of the classroom is not simply about students complying to rules and regulations; rather, it is concerned with whether there is a clear set of rules that are understood by students, and whether teachers maintain high expectations that these rules are followed. Learning time is also an element of this indicator, as students have less opportunity to learn in classes that are noisy or constantly being interrupted by a few disruptive students. Some example items are: "Teachers expect students to pay attention", "The rules for classroom behavior are clear", and "Some students make it hard to concentrate in class" (reverse coded).
Expectations for Academic Success refers to the extent to which school staff value academic achievement and hold high expectations for all students. The term "academic press" has often been used for this indicator. Schools with high academic press place a strong emphasis on academic skills. Teachers and the principal project the belief that all students can succeed, and they encourage students to set high norms for their academic success. The Tell Them From Me indicator of this factor includes, for example, "In our school: ...Teachers expect all students to do their best work", "Teachers expect homework to be done on time", and "Students must work hard to succeed."
Teacher-Student Relations assesses students' perceptions about how they are treated by their teachers, whether they feel supported by them. Youth tend to have better schooling outcomes when adults are responsive to their needs and encourage independence with a democratic approach. This indicator is measured with eight items. For example, students are asked to indicate the extent to which they agree or disagree with statements such as: "Teachers take account of students' needs, abilities, and interests", and "Teachers show an interest in every student's learning."
Student Advocacy is based on the premise that every child needs an advocate, someone to turn to for advice and someone who consistently provides encouragement. It could be someone at home, but it may also be someone at school or in the community. For example, Tell Them From Me asks students "How often does someone in your family, school, or community do each of the following:... "Take an interest in one of my school projects", "Ask me about problems I might have at school."

Teacher Survey

Approach
The teacher snapshot survey is optional for a small additional fee per participating school ($100 per snapshot in 2007-08). No teacher data is uploaded to our system. A school requests survey usernames and passwords based on the number of teachers to be surveyed, and then distributes these randomly and anonymously to teachers. Reports are made available to the school when they determine they have reached their optimal response rate. We encourage schools to strive for a 75% participation rate. Data from teachers' responses are not accessible by district personnel.

Measures
Five classroom and school indicators are assessed with the teacher questionnaire.

Effective school policies and practices gauges the quality of the organizational structures in the school that guide school improvement and support student learning and positive behaviour. It is concerned with whether the school has a strong orientation to student success, reflected in its mission statement; the effectiveness of school committees and faculty meetings; the communication among staff; and professional development. For example, teachers are asked to indicate the extent to which they agree or disagree with the following statements: "There is an effective action plan for school improvement," "Teachers work effectively on school committees," and "Communication between the principal and staff is effective."
Effective practices to support vulnerable youth encompasses several practices that have been shown to be effective in supporting youth with learning or behavioural difficulties. The recent concern for school safety and positive student behaviour has engendered several studies of schools that have adopted a preventive, whole-school approach to supporting positive student behaviours (Sugai & Horner, 2002). This research, and recent research on vulnerable children in Canada (Willms, 2002), calls for schools that have effective policies and practices to support vulnerable youth. Tell Them From Me asks teachers to assess some of their own classroom practices, and to indicate the level of support provided by school counselors and administrative staff. Some example items include: "Routines for classroom activities are well established," "There is strong support from school counsellors," and "I am available to give students extra help when needed."
Staff morale and commitment to school improvement includes items about the morale of teachers in the school and the commitment of teachers to improving student outcomes. Considerable research has documented a relationship between teacher morale and commitment, with students' academic achievement. Schools with committed, satisfied teachers have less teacher absenteeism, lower turnover, less burnout, and less dysfunctional classroom behavior. Teachers tend to be more committed when there is a high level of trust among school staff, when staff work together to set specific goals for student outcomes, and when they have a strong sense of efficacy - their work is meaningful and they feel empowered to make a difference. Some example items in Tell Them From Me are: "I am committed to making our school one of the best in the province," "I am excited about the classes I teach," and "On most days I look forward to going to work."
Effective assessment practices includes both formal and informal assessment practices. Teachers are asked to rate the adequacy of the assessment practices in their school for various purposes, such as assessing students' ability in basic skills, informing parents about their child's progress, making comparisons with district or provincial norms, and informing decisions about school policy and practice.
Positive relations with parents and the wider community is based primarily on Epstein's framework for fostering positive relations between the school and the community. Successful schools foster greater communication with parents, and encourage parental involvement in their child's schoolwork. Successful schools also enlist parents to volunteer at the school and participate in school governance. In Tell Them From Me teachers are asked to rate the extent of parent and community involvement in their school and to indicate the extent to which teachers encourage parental involvement. Some example items are: "Teachers use a variety of tools to establish meaningful two-way communication with parents," "Parents are involved in setting student goals," "Parents' opinions are solicited on issues concerning school practices," and "The staff help make our school a centre for culture, recreation, and learning."

Parent Survey

The Parent Survey was pilot tested in 2006-07 in a limited number of schools, and will be more widely available in 2007-08.

The Parent Survey consists of 64 questions, covering five broad measures -

Two-way Communication
Parents Support Learning at Home
School Supports Learning
School Supports Child's Social and Emotional Development
Parents Support Learning at School

Once the school has been set up for the Parent Survey they will then have the option to set their school's own open-ended question.

As with the student and teacher surveys, participation is voluntary and anonymous, and surveys are completed on-line only. If the school wishes, parents of students from all grades can do the survey, whether or not their children are of age to participate in the student survey. In other words, parents of children in the elementary school years are able to participate as well if the school wishes for them to participate.

For parents who wish to do the survey but who do not have access to the Internet, the school might set aside a particular week when a place and specified time can be arranged for these parents to participate (e.g., at a Community Computer Access Centre, in the school computer lab, etc.). The school would need to manage the distribution of usernames and passwords at that time, and arrange for someone to assist parents to log on to the site.

Parents may wish to be apprised of results. Parent Survey results are made available only to the school principal and School Coordinator as a snapshot report. It is the responsibility of the school principal to decide whether this is appropriate, and what, if any, results are to be shared with parents. TTFM will not make results available to parents on-line or in a report. Having said that, a custom report could be prepared for the school or district for a fee.

References

Fullan, M. (2003). Change Forces with a Vengeance. London and New York: RoutledgeFalmer.

Hill, P. W. & Crevola, C. A. (1999). Key features of a whole-school, design approach to literacy teaching in schools. Australian Journal of Learning Disabilities, 4(3), 5-11.

Scheerens, J. (1992). Effective schooling: Research, theory, and practice. London: Cassell.

Tremblay, M.S. and Willms, J.D. (2003). Is the Canadian childhood obesity epidemic related to physical inactivity? International Journal of Obesity, 27(9), 1100-1105.

Willms, J. D. (1992). Monitoring school performance: A non-technical guide for educational administrators. Lewes: Falmer Press.

Willms, J.D., Tremblay, M.S., & Katzmarzyk, P.T. (2003). Geographic and demographic variation in the prevalence of overweight Canadian children. Obesity Research, 11(5), 668-673.

Willms, J.D. (2003). Student engagement at school: A sense of belonging and participation. Paris: Organization for Economic Cooperation and Development.

For more information or pricing, contact: Patrick Flanagan CEO The Learning Bar Inc. 506-458-9311 info@thelearningbar.com