Frequently Asked Questions (FAQ's)
These are organized in three sections below:
THE LEARNING BAR (TLB) Survey System
Q. What makes the TLB system unique?
(a) online student surveys that take 20-30 minutes to complete, with no input required from the teachers
(b) optional on-line teacher surveys
(c) optional on-line parent surveys
(d) continuous, useable feedback to teachers and school principals indicating the week-to-week progress on the school climate indicators and schooling outcomes
(e) data that can be linked at the school level to other data collected by the school or district to assess the impact of changes in school policy and practice on schooling outcomes, such as academic achievement
(f) indicator scores that can be compared with national norms
(g) a "drill down" feature that enables school administrators to compare their school's progress with national norms, and to break down results by gender, socioeconomic status, grade level and other factors of interest to the school or school district
(h)accredited District Coordinators assist schools to interpret reports and identify solutions, supported by The Learning Bar staff through online resources
Q. We don't want to simply collect data. How can we link it to our school improvement plan and develop follow-up plans?
A. With the continuous flow of data, you are able to link the constructs (or measures) to particular school plans. This is much more powerful than a one-shot survey that is soon forgotten about.
Q. Who will have access to the survey results?
A. School Coordinators (e.g. principals or school counsellors) have immediate online access to the results for their specific school through interactive charts, our "one-click" school reports, and the open-ended question responses. School Coordinators also have access to the results from the optional Teacher and Parent surveys.
Where TLB contracts with the School District, the district would normally have access to each school's student reports. However, districts do not have access to student responses to any open-ended questions. Nor do they have access to any school-level teacher survey results, unless school principals specifically instruct The Learning Bar that this information be made available to district personnel.
The Learning Bar expects school district administrators to use the data in a formative, supportive way. We feel that having the real data is usually better than the informal data (gossip, stories, biased reports) that are commonly used to exert pressure without support. The challenge for school administrators is striking the right balance between formative and summative evaluation. Because TTFM provides regular feedback and support, it is much easier to work on a formative basis.
Q. Is TTFM not also an evaluation of the school administration?
A. There is certainly an element of that in the optional Teacher Survey. If the Teacher Survey is to be useful, the principal has to be open to feedback from the teaching staff about what is or is not working in the system. That said, we see this as a low-stakes evaluation, with a very strong emphasis on formative evaluation. The aim is for the staff to use the data to identify one or two areas (e.g. disciplinary climate, or effective school practices) that the school can work on improving during the year.
Q. Can district staff have access to student-level data for its own research?
A. No, as this would compromise our promise to maintain anonymity. However, districts can negotiate with The Learning Bar staff for customized reports.
TTFM Implementation
Q. Is there a minimum sample size for your results to be reliable?
A. We encourage schools to offer the survey to all their students. That said, while a 50% sample might be all you need, drawing the sample is a challenge to ensure it is representative. We have made efforts to make the survey focused, reliable and efficient, as well as easy to implement. We hope that schools will in turn make it accessible to all students. Finally, we strongly encourage schools to aim for a minimum 75% participation rate for their student population.
Q. How many districts identify other drill-down variables of interest such as French Immersion/Academic/Applied; Aboriginal/non-Aboriginal; etc.?
A. Over the years, it has varied. Most do not do so initially, as they have lots of new data to start with, just using the basic survey. Some, however, have a clear idea what they are after, and choose a drill-down from the start. We are pleased to discuss this further with districts, as there is an additional charge for this feature.
Q. Does the Drill Down Option apply to all implementation models?
A. The standard built-in ones - grade and gender - yes. Custom drill-downs can be used with all the evaluation options except the "essentials" survey, where there is no initial student data upload, thus no possibility to create a custom field for drill-down.
Q. Why does the drill-down feature not present results for a two-category breakdown such as by Grade and by Gender?
A. We are actually introducing this feature in 2009 with the launch of our new system.
Q. If we choose a district licence, must the selection of schools involved remain the same over the three-year period?
A. Districts and schools are strongly encouraged to participate for three years in order to generate longitudinal data. Experience to date suggests that some schools begin to use the survey data effectively only in the second year. That said, there is no obligation to involve the same schools.
Q. How is a replica school different from a same-type school?
A. A same-type school is either all high schools or all middle grade schools (e.g. K-8, 6-8, 7-9 for middle schools; if there is a grade 10 and up (e.g. K-12, 9-12 or 7-11), it would be considered a high school.)
A replica school is a virtual school comprised of students with the same characteristics as the students from your school who have completed the survey to date (grade, gender and socio-economic status), drawn from our database of students who completed the survey the previous school year (about 80,000 students in 2007-08).
Q. How long does it take to complete the survey?
A. Depending on the students, it usually takes from 20 to 30 minutes to complete the survey. We normally suggest schools set aside a regular class period for completion. For the small number of students who for whatever reason may not complete the survey during this time, they should be instructed to hold on to their username and password supplied to them, and to log on at another time using the same logins. This will take them to the page following the last one they submitted, and they can complete the survey.
Q. If we use the weekly Random Selection method, how do we know who is supposed to do the survey when?
A. Once your school's student data are uploaded to The Learning Bar system, students are randomly assigned to a specific week when they are scheduled to complete the survey. Each Thursday, an e-mail is sent to the school by the system, indicating which students are to do the survey the following week. This list also includes a unique username and password for each student, and indicates which students from the previous two weeks have yet to complete the survey. This full survey list for the whole year is also accessible online by the School Coordinator at any time.
Q. If we use the weekly Random Selection method, we are concerned about the workload for administration to track down students who are scheduled to do the survey.
A. This is a legitimate concern, for which a number of solutions have been suggested. In some schools they read over the intercom the student names from the TTFM weekly email, asking them to come to the office to pick up their username and password. Another school has all the scheduled students come down to the computer lab first thing in the morning. We have added the student's homeroom to the survey schedule, again in order to make it easier to locate the students on the weekly emailed schedule. Our system also generates a master list of the survey schedule at the outset, which some schools report can help with organizing the completion of surveys.
One of the attractive features of TTFM is that all students get a chance to do the survey. Our experience in this pilot year suggests that TTFM is one of those things that needs to be integrated into the school's routines, and it seems schools have been finding their own particular solutions to scheduling. For example, a school might use it as an opportunity for the principal or counsellor to meet over a brown bag lunch with all the students scheduled for the week.
Q. Still with the weekly Random Selection method, do we have to accept the TTFM-generated random schedule of students to take the survey, or can our school organize the students our own way?
A. The Learning Bar's interest here is in maintaining confidentiality, maximizing random distribution of student responses, and in helping schools manage an orderly process for all students to take the survey over the course of the year. Using The Learning Bar scheduling system accomplishes this, and automatically monitors which of the scheduled students has not yet completed the survey.
Q. How will consent for student participation be handled?
A. The TTFM survey is seen as a request for student feedback to inform the school and school administrators; it is not a research project. In the past, schools and districts participating in Tell Them From Me have used a passive consent approach, whereby parents are informed of the project and told that student participation is not mandatory. So far no parents have requested that their children be excluded from the survey. Where a school district policy requires a formal procedure for informed consent, you can ask The Learning Bar for information to include in parent consent letters.
Q. What is the process for informing / advising students and staff about TTFM?
A. The Learning Bar staff can provide a school with a draft letter to be distributed from the principal. Ultimately, it is the school and district staff that need to determine where TTFM fits into their on-going efforts at school improvement, and the degree of commitment they wish to convey to students and staff.
Q. What happens when we get refusals from parents, or when a child no longer attends our school? Do the students' names get taken off the list?
A. Yes and no. As The Learning Bar staff do not have access to students' names, the weekly email schedule will still be sent to the school with these students' names appearing on it. The school simply ignores the name. The reminder (that those students have yet to complete the survey) will cease after two weeks.
Q. Can we work with just a sample of students? For example, could a school just invite students-at-risk to participate?
A. A flexible and affordable pricing structure allows this system to include all students, giving everyone an opportunity to be heard, and avoiding any tendency to label or segregate. TTFM is designed for all students in the school. Its purpose is for staff to get a sense of the overall school climate and processes so that they can better create a positive learning environment for all students. If, however, they are identified in your school's database, they can be indicated as a drill-down at the time of student data upload.
Q. Is it possible for a student to do the survey more than once?
A. Once students have signed onto the system and completed the survey, the system denies them any further access. The exception is when the school does two snapshots of all their students, at two different times in the school year. Only then would students be allowed to complete the survey twice during the same school year.
Q. What happens for students with special needs, those who would find it difficult to complete an on-line survey? Do we eliminate them from the list? Or if we had someone to read it to him or her when need be, would these students have the option to do it?
A. The issue with this situation is the TTFM commitment to confidentiality and anonymity. Schools could advise beforehand the parents of students with special needs, and let them know that their child will be taking the survey with the assistance of a Teaching Assistant (or whatever arrangement you have with them normally). Each school will find its own solution, depending on the school community's norms and culture.
Q. What do we do about students who threaten suicide in response to an open-ended question, or report that they have been abused? Is there some way to identify the student?
A. The Learning Bar is not able to identify individual students. However, responses to open-ended questions can eventually be narrowed down to grade and gender. Presumably, such reports would require a response from the school administration similar to that followed when receiving an anonymous allegation or complaint at the school office.
All responses we receive to open-ended questions are made accessible to the respective schools unedited after a 30-day delay. To date, many if not most students are really taking the time to say what they feel. You get a real sense of the dynamic within the school. Very few have abused the opportunity.
Q. What is the overall percentage of response rates so far in other schools?
A. It differs from school to school. Some have a 95% response rate and most are in the 70-85% range. A small minority of schools have considerably lower response rates. It seems to depend to a large extent on how important this survey is to the school and the emphasis that is placed on it. Some schools have used incentives to try to get their response rates up (e.g. free refreshments), while others just do it on school time as an incentive (students getting out of class). The latter seems incentive enough, when combined with a clear commitment from schools to hear and respond to what is being said. If the school has a visible commitment to school improvement, this survey system will quickly be adopted and integrated as a very useful tool to assist with that.
Q. Can I (or can students) access the website from home?
A. Yes – individuals with valid passwords and IDs can access the TTFM survey website from anywhere.
Q. What are the staffing requirements and internal costs to school boards associated with this project?
A. At the district level, a District Coordinator is required to manage the project implementation. There is also a District Data Manager role to handle the uploading of student information to the main database (this is a yearly activity, required at the front end of the process, and requiring some IT experience). At the school level, the principal will be somewhat involved as the school leader, but the most crucial role is that of a School Coordinator who will encourage both participation in the survey as well as school-level analysis and use of the data generated. This is often a staff member who already has responsibility for assessment data or school improvement.
Q. How will the survey be undertaken? Is it done every week?
A. There are three basic approaches to our system of Continuous Monitoring, and we make an effort to accommodate school-level requirements for hybrids of these approaches when asked:
- Weekly random selection, with each student completing the survey once during the school year. This usually works for schools with enrolments between 300 and 700 students.
- Monthly selection of entire classrooms, with a cross-section of grades, works well in larger schools – more than 700 students.
- Snapshot of the entire student body, done twice yearly, works well for smaller schools – less than 300 students.
School Districts will be able to choose, in consultation with their District Coordinator, the approach that works best for them. School Districts need not use the same schedule for each school within their district.
Q. All of our participating high schools are semestered. They would likely prefer to use a classroom selection method of implementation, but does this require that students be in the same room for the whole school year?
A. The "survey classroom" for the student data upload handled by the School District has to be a particular subject class that all students are required to take at some point during the school year (in many cases, English [Language Arts] classes; in Catholic high schools, it might be theology classes). We will then use that class as the survey classroom upload field, and students will complete the survey during that class (whether students are in the class for one or both semesters). This seems to have worked in all our "classroom selection" schools to date.
Q. Are the school reports different, depending on which evaluation method you choose?
A. Yes, although they all can be viewed as bar charts. Some methods have an option to view a trend line as long as data are collected at regular intervals throughout the school year (classroom selection - monthly; and random selection – weekly). If you go to the Tell Them From Me demonstration site http://demo.tellthemfromme.com and choose "district-level" reports, then "monitor progress", you will see four schools to choose from. Each one represents a different evaluation model.
Q. Are data sharing agreements between school districts and The Learning Bar possible? Will school districts have access to the data from their own schools?
A. Schools will have access to their own data, trend lines over time comparisons and comparisons with a hypothetical "replica" school constructed by matching student characteristics. Schools will have a complete record of all student answers to open-ended questions. Districts will have access to the data for all of their schools except they will not receive the student open-ended question responses. It is not possible to link student survey responses at the student level to other student databases because all participants have been guaranteed anonymity and all identifiers are stripped from the data when the survey is submitted. However, it is possible to link the school-level indicators, such as the measures of school climate, to school district data. The primary consideration is that we do not breach the anonymity that we offer students.
Survey data are available to the school and district in real time through the website.
Q. How is student demographic data collected?
A. The demographic data to determine socio-economic status of students is collected in two ways this year: through the student data uploads in cases where they are used, and through specific questions in the student survey itself, related to parents' education levels and household belongings.
Q. How is student confidentiality ensured?
A. The survey is completely anonymous. Data queries are not released for sets of fewer than 10-15 students in order to prevent singling out, and no forms of personal identification are maintained in our database.
Q. Do most schools participate in the teacher survey?
A. It is important that teachers understand that participation is voluntary and that the results are seen only by the principal and School Coordinator. In undertaking the Teacher Survey, principals should be willing to share what they have learned from their teachers and to develop action plans where needed.
Q. Is a parent survey available?
A. Yes, and it is free to use until December 31, 2008.
Q. There is a 30-day delay in reporting the open-ended question responses - is this true for all implementation methods?
A. Yes, open-ended question responses are made available to schools on the first day of each month, covering the past month of responses.
The TTFM Survey
Q. Are the fixed response (close-ended) questions the same for every school?
A. Yes. All students get the same basic survey. On the other hand, open-ended questions can vary as determined by the school.
Q. Can we ask our own open-ended questions?
A. Yes, and the question can be changed when you want.
Q. What is the one (1) open-ended question that is asked of all students?
A. "Tell us some of the things you really like about your school, or things that would make it even better."
Q. For what grade levels is the TTFM survey appropriate?
A. We have tested and validated the survey for grades 6-12.
Q. Do you have any additional survey topics for us to choose from?
A. The Learning Bar staff are in the process of custom designing some additional constructs for several clients. We hope to make these available for wider use with their permission.
Q. Can you get survey results by grade?
A. Fixed response (Likert-scale) questions are displayed as averages of all student responses on the whole construct or topic (such as participation in school activities, or teacher-student relations). The results, as well as the responses to the open-ended questions, can be sorted by grade and gender, keeping to the guidelines we have established to maintain respondent anonymity.
Q. Can you get responses by specific question, rather than only by "measure"?
A. In 2008-09, we are introducing some question-level results where schools have made the case that it would be most useful.
Q. Can we differentiate areas of the school (e.g. departments) from the data? Will we be able to pull out responses from "at risk" students separately?
A. It is now possible to have sub-groups identified at the onset, as part of the drill-down feature.
