Saturday, September 1, 2012

The Insanity of the Marzano Evaluation System...

You have to laugh at the state of education in Florida (and to some extent, the US).  While NAEP scores continue to rise throughout the country for 30+ years, you still have people screaming for the latest fad to save us.  In the case of Florida, they bought into Marzano's "causal" framework.

Let's start with the most obvious problem:  it's ridiculously convoluted.  If I were asked to create a parody of an evaluation system, I couldn't possibly make something more hilarious than Marzano.  Even they state it "works best" with a year of planning and training.  What the hell kind of evaluation system requires a year of EITHER, yet alone both?  (Answer:  A very bad one.)  If you are going to invest this kind of time (and the resources) into a program, you better have extremely high expectations for it.  Sadly, pretty much everyone with a modicum of intelligence knows that this is not going to change anything (yet alone significantly for the better).  It shouldn't be necessary to have to tell ostensibly well-educated people this, but a simpler (and thus more easily/better understood) system will work better than a complicated one.

How many "elements" (items) are in it?  The answer is 60, in four "domains" (edu-speak for "areas"; you can't charge $23/book for using simple language!).  The evaluation system would be better as a bulleted list of "suggestions"--it would have saved millions of dollars.  In that role (a list of "good suggestions"), Marzano is just fine.

But, surely it will be massively successful; after all, Marzano's system has been evaluated by none other than Marzano himself--just check out Research Base and Validation Studies on the Marzano Evaluation Model, April 2011.

To quote this (presumably) non-peer reviewed work, "[The] Marzano Evaluation Model is based on a number of previous, related works that include: What Works in Schools (Marzano, 2003), Classroom Instruction that Works (Marzano, Pickering, & Pollock, 2001), Classroom Management that Works (Marzano, Pickering, & Marzano, 2003), Classroom Assessment and Grading that Work (Marzano, 2006), The Art and Science of Teaching (Marzano, 2007), Effective Supervision: Supporting the Art and Science of Teaching (Marzano, Frontier, & Livingston, 2011)."

That's right--you know Marzano's framework will work because it's based on Marzano's work!  (Also, Phillip Morris would like you to know that smoking increases your libido, cures cancer, and prevents male-pattern baldness.)  Check out the section entitled The Research Base from Which the Model Was Developed:

"Each of the works (cited above) from which the model was developed report substantial research on the elements they address. For example, The Art and Science of Teaching includes over 25 tables reporting the research on the various elements of Domain 1."

I don't believe I've ever seen the number of tables in a book cited as evidence of the book's quality, i.e., "You know it's good because there's a lot of it."   Apparently taking lots of (often old and often poor-quality) semi-related data and mashing it together makes good research.  Only in education research is this even remotely possible.

16 comments:

  1. "The evaluation system would be better as a bulleted list of "suggestions"--it would have saved millions of dollars. In that role (a list of "good suggestions"), Marzano is just fine."

    This is exactly right. Many of the mandated garbage in the education system fits this description (LFS immediately comes to mind).

    ReplyDelete
    Replies
    1. It's sad, especially with recent budget shortfalls, that the politicians continue to push this worthless--and expensive and time-consuming--garbage onto us.

      Had to look up LFS; fortunately, my district (or at least my school) has been spared this bunkum for now.

      Delete
  2. I keep hearing "only 2% of teachers are innovating/highly effective" according to Marzano. Sounds like it was more like an investment on the part of those who were scared about paying any merit pay.

    ReplyDelete
    Replies
    1. I only found one reference to the "2%" and that was in his "Race to the Top" "whitepaper" (read: Paper that looks academic, but really isn't any more so than this blog...in fact, given that it's Marzano, I'd say my blog is probably more credible simply on the basis that any positive number is greater than zero). In it they state,

      "Marzano (2009) has noted that a teacher who is classified as “most effective” (i.e., at the 98th percentile in terms of his or her pedagogical skill)
      will be expected to produce student achievement that
      is 54 percentile points higher than the achievement
      produced by a teacher who is classified as “least
      effective” (i.e., at the 2nd percentile in terms of his or her pedagogical skill)"

      A 54 percentage point increase? That's just laughably unrealistic. One of these days I'm going to have to slog through some more of Marzano's pseudo-research (aka "meta analysis") to try to figure out just how incompetently the numbers were crunched. In the mean time, I'll leave it to Dr. Wexler's analysis:

      http://www.amazon.com/review/RYB2CUDKIISEP

      PS Notice how surprisingly few reviews Marzano's books have?

      Delete
    2. "Sounds like it was more like an investment on the part of those who were scared about paying any merit pay. "

      And yes, that's definitely a benefit for administration/politicians: Not only will this "improve" the schools, but it all-but-guarantees not having to pay out a lot of money in the process.

      I'm willing to bet Dr. Marzano's proctologist helped him find that 2% figure.

      Delete
  3. A bit more of why I don't trust anything from "experts" such as Marzano:

    http://ventingmycynicism.blogspot.com/2012/10/marzanos-database.html

    ReplyDelete
  4. Thank goodness, I found someone else who would dare to question the master, the guru, the end all of education. While he is making millions establishing his brand, public school educators have been thrown under the bus, and he is at the wheel, driving forward and backwards impressing every school administrator in ear shot of the screams of agony wrenching out from under the bus! I will listen to him when he comes into my school and spends one entire year implementing personally his "strategies" of effective classroom management, etc. Who evaluates his work and dares to test it to prove it more than a quick fix, pie in the sky phenomena. Let's throw Hunter in the mix with it, shall we?

    ReplyDelete
    Replies
    1. School administration (particularly at the district and state levels) are proof positive that there's a sucker born every minute...

      And good luck getting the "esteemed" Marzano to spend time in a real classroom and not one picked out for photo ops. ;)

      Delete
  5. I'm so glad that I found this site! My school here in Queensland Australia is all fired-up about this model, while I thought what a HUGE waste of human resources. Our Principal is a true disciple and flew to a workshop in the U.S.A. to really get soaked in Mazano strategies. We just this afternoon had an hour long meeting on one point; a design question that otherwise would have been a two minute casual discussion. I thought I was the only one on earth that thought this is all a great temple built of cardboard and wishing it would all blow away. I have taught for 25 years and have never seen such a complicated, response-funneled, mind-directed, time-consuming waste of time.

    ReplyDelete
    Replies
    1. Sorry I didn't see your reply sooner; haven't used the blog much. But I'm more sorry for all the horrible things my country has exported to the world--Marzano may be the worst yet. Marzano is just a waste of time and money--anyone with common sense can see that. Unfortunately, the policy makers at the state-level are hardly known for doing anything sensical.

      Delete
    2. Thank for the sympathy! But it's now got worse! Seminars are big time here now. The principal just returned from a $600 two day seminar and stated that it was so great to hear other principals speaking the same Marzano talk, (couldn't any of them tell its repackaged much more of the same?). The principal added that other schools love it, even those teachers who have taught for 30 years! WHAT! I don't believe that, I am sure it has to do with the questions one asked about this 'strategy', because when I ask teachers what they think they can't stand the amount of time it takes to respond to the 'design questions'. All that the Marzano system does is gather stats quickly for portfolios, waste a lot of time, install a weird lingo into staff meetings and worst of all creates a great divide. We now have the disciples and the commoners. The beguiled believers and the frustrated teachers who want to get onto the real tasks involved with this profession.

      Delete
  6. Mary Wilburn March 6 2014 - This too will pass but unfortunately not untl it has driven all of the good teachers out of education. Marzano is such a bunch of convoluted bunk leavng little to no room for individuality in teaching styles.......how boring for our students. Really....60 things we have to do every lesson to be good teachers. Why not. Make it 65 tell treachers how to all smile....walk....sit.....stand....and take a #@*$ exactly the same way so the can get a check in the right collumn. Young administrators are all hyped on Marzano talk like cult members. It is a pathetic situation.

    ReplyDelete
    Replies
    1. Sorry for the late reply. Yes, Marzano is convoluted and has at best no return on investment. The research behind it is basically incestuous ("You know Marzano's system works because it's based on...Marzano's research!"). If the "effect sizes" actually meant what Marzano, Hattie, and others claim, any teacher could just do one, two, maybe three of these strategies and their students would be waaaaay better than average.

      As for the administrators, I once made the mistake of asking an in-depth question about effect sizes; I'd never seen someone so blatantly "caught". I'm willing to bet asking just one question these poor people don't have a talking point for (i.e., a detailed or technical question) will reveal the same thing--that they really have no idea what's underpinning Marzano's system.

      Delete
  7. We teachers are intelligent. Lets try to take action against this guy and other legislature. (Maybe senators, firemen and policemen also need cuts and job evaluations,) why are public teachers being targeted so heavily? do you think it has to do with pension money????? Vote these people out, its still a democracy. If enough people write, sign petitions to their unions, maybe we can turn this. We can't take this lying down! Use your voting and legislation rights.

    ReplyDelete
  8. Any chance that I could get a copy of Marzanos database that you copied and pasted

    ReplyDelete
    Replies
    1. http://depositfiles.com/files/tisvbfmpd

      Delete