News

Exclusive: Comparing School Safety in NYC & LA — Our Methods Behind New Interactive Maps

By Max Eden | June 7, 2017

Below, we explain the methodology used to create two new maps of school safety across New York City and Los Angeles. Also related: Max Eden’s analysis of lessons he learned from compiling and mapping the data, and comparing schools across different neighborhoods. Get the latest news about school climate and school safety, as well as access to new interactive tools that better help parents understand their school district, by signing up for The 74 Newsletter.
When evaluating school climate survey data, perhaps the first question one would ask is: Can we trust that these surveys reflect on-the-ground realities?
I believe the answer is, for the most part, yes. In any instrument like this, researchers ask two questions: Is the instrument reliable, and is it valid?
Reliability is gauged by the consistency of answers, both across time and internally. Although frequent changes in questions on school climate surveys given to students in Los Angeles don’t permit a fair test of reliability across time, my analysis of questions on surveys administered in New York City schools showed remarkable stability in survey responses from 2011–12 to 2013–14 (prior to Mayor Bill de Blasio’s discipline reform, at which time there was a marked negative shift).
Internally, the high correlation between survey answers that one would expect should be correlated suggested that students and teachers are answering consistently and, therefore, faithfully. In Los Angeles, the correlation between student-reported safety and teacher-reporting fighting is a robust r = .74. In New York, the correlation between student-reported fighting and student-reported feelings of safety in hallways is r =.83.
Questions have been raised around my report on school discipline and climate in NYC as to how valid an instrument these surveys are, with one professor suggesting that policy factors could influence student perception — i.e., perhaps by highlighting fights in “restorative justice” practices, students might think they are more common when they are actually less so. It’s plausible, though unlikely, given the high correlation between student reports of violence and feelings of personal safety. But at the end of the day, the best gauge of instrumental validity will be whether the survey answers track with the lived experience of students and teachers.
And while instrumental validity is an open question, it’s safe to say that these surveys provide the most valid instrument we have to gauge climate — especially as, in both cities, traditional measures of school climate (such as suspensions and police interventions) have been manipulated by policy.
In my new analysis of school climate surveys in LA and NYC given in the 2015–16 school year, the percentages recorded represent the percent of negative answers to questions: e.g., the percent of teachers who disagree or strongly disagree when asked whether order and discipline are maintained.
In mapping the data, schools are color-coded according to answers to one specific question (detailed below), purely at the author’s discretion. The coding is intended as a shorthand tool, not as a formal rating. But it seems fair to worry that something is amiss if more than a third of students say, for example, that physical fighting happens “most” or “all” of the time.
New York City
The data from the 1,776 New York City schools that participated in the 2015–16 school culture survey were pulled directly from the NYCDOE website. Not all of the city’s more than 1,800 schools took part in the survey.
Questions:
TEACHERS:
At my school, order and discipline are maintained.
Strongly disagree / Disagree / Agree / Strongly agree
STUDENTS:
At this school, students harass, bully, or intimidate other students.
At this school, students harass, bully, or intimidate each other because of their race or ethnicity.
At this school, students get into physical fights.
At this school, students drink alcohol, use illegal drugs, or abuse prescription drugs while at school.
At this school, there is gang activity.
None of the time / Some of the time / Most of the time / All of the time
Most students at this school treat each other with respect.
Strongly disagree / Disagree / Agree / Strongly agree
Coding:
Elementary school students are not surveyed, so the only answer we have from NYC elementary schools is the teacher answer to the question regarding school order. Schools where less than 15 percent of teachers said order is not maintained are labeled green/safe; schools where 15 percent to 30 percent of teachers said order is not maintained are labeled yellow/somewhat safe; schools where more than 30 percent of teachers said order wasn’t maintained are labeled red/less safe.
Middle and high schools were coded by the student answer to the question regarding physical fights. Schools where less than 15 percent of students said fights happen “most” or “all” of the time are labeled green/safe; schools where 15 percent to 30 percent of students reported frequent fighting are coded yellow/somewhat safe; schools where more than 30 percent of students reported frequent physical fighting are coded red/less safe.
Other Notes:
Some of the questions were asked with the same exact wording in the 2013–14 school year, the year before de Blasio’s landmark discipline reform. With those data in hand, the map also offers whether the response to each question is better than, worse than, or the same as two school years prior. Wording changes to other questions, however, render comparisons invalid. For those questions, we report “can’t compare.” Some schools did not participate in the survey at all two years prior; for those schools, no comparison is possible. There are a handful of schools where an answer was not recorded for a particular question. In those instances, there is an “N/A.”
Los Angeles
Data from the 786 Los Angeles schools that responded to the 2015–16 school culture survey were pulled directly from the LAUSD website. Not all of the city’s reported 1,087 schools took part in the survey.
Questions:
The questions in the LAUSD survey contained more variation in response options than the NYC survey. There were also variations in both wording and response options for elementary school students. In questions containing “unsure” as an option, that response is counted as a negative, as when LAUSD itself reports percent positive in its survey reports (e.g., when it says 65 percent of students say they feel safe at school) it excludes that response portion.
STAFF:
Please indicate how much of a problem you feel these issues are at this school:
Harassment or bullying among students?
Physical fighting between students?
Disruptive student behavior?
Racial/ethnic conflict among students?
Lack of respect of staff by students?
Severe problem / Moderate problem / Mild problem / Insignificant problem
Please indicate how much you agree or disagree with the following statements about this school this year:
This school effectively handles student discipline and behavioral problems.
I feel safe on school grounds during the day.
Strongly disagree / Disagree / Agree / Strongly agree
STUDENTS:
Respect
Elementary
Do students treat teachers with respect?
No, never / Yes, some of the time / Yes, most of the time / Yes, all of the time
Secondary
Students treat teachers with respect.
Strongly disagree / Disagree / Neither disagree nor agree / Agree / Strongly agree
Getting Hit by Other Kids
Elementary
Do other kids hit or push you at school when they are not just playing around?
No, never / Yes, some of the time / Yes, most of the time / Yes, all of the time
Secondary
In the past 12 months, how many times on school property have you been pushed, shoved, slapped, hit, or kicked by someone who wasn’t kidding around?
Zero times / 1 time / 2 or 3 times / 4 or more times
Getting Beaten Up
Elementary
Are you afraid of being beaten up at school?
No, never / Yes, some of the time / Yes, most of the time / Yes, all of the time
Secondary
In the past 12 months, how many times on school property have you been afraid of being beaten up?
Zero times / 1 time / 2 or 3 times / 4 or more times
Bullying
Elementary and Secondary
Bullying is NOT a problem at my school.
Disagree a lot / Disagree a little / Neither disagree nor agree / Agree a little / Agree a lot
Safety
Elementary
Do you feel safe at school?
No, never / Yes, some of the time / Yes, most of the time / Yes, all of the time
Secondary
I feel safe in my school.
Strongly disagree / Disagree / Neither disagree nor agree / Agree / Strongly agree
Coding:
Both elementary and secondary school data are coded based on student responses to the question pertaining to school safety. Those questions, however, varied both in wording and response, so it seemed appropriate to be guided in part by the overall distribution of answers.
Elementary schools were coded as green/safe where less than 10 percent of students did not report feeling safe most or all of the time; schools where 10 percent to 20 percent of students did not report feeling safe were coded as yellow/somewhat safe; schools where more than 20 percent of students did not report feeling safe were coded as red/less safe.
Secondary schools where less than 25 percent of students did not report feeling safe were coded as green/safe; schools where 25 percent to 40 percent of students did not report feeling safe were coded as yellow/somewhat safe; schools where 40 percent or more of students did not report feeling safe were coded as red/less safe.
Other Notes:
For some schools, Los Angeles reports data across several grade spans. For those schools, we simply averaged each observation. Whereas NYC had several questions remain constant from before its discipline reform, nearly no questions were asked in consistent form from before LA’s discipline reform until this later year. As such, there is no “better/worse” comparison to be offered.


Submit a Letter to the Editor