Category Teaching

Jul 5 2018

Why I Use Self-Assessment

Grading is the most unfortunate part of education. And it’s the only part about being an instructor that I really dislike.

Yes, grades are necessary in that they measure student performance. Grades can provide you with an indication of how well you might be doing compared to others in your class, or how you might improve. And grades are used as a quick guide for others to judge you – e.g., in future applications (for either education or employment) or for awards like scholarships. Grades also matter for your own sense of self-worth and accomplishment. Yes, grades matter.

But grades are unfortunate because they don’t matter for what really matters. Think about some form of self-education that you take on because you are interested in it – like a new sport or hobby. You learn how to do it, and practice it. You get value from mastering a new skill, and feel good when you determine that you have accomplished a goal. Usually (hopefully) you don’t have someone looking at your work and saying “well, that’s about 75% good”. You know when you have improved, and when you can improve more. You know when you’re at 75% of someone else, and when you’re at 90% compared to where you were 2 months ago. But you don’t give yourself a number, do you? You instead note how you’ve improved and how you would like to improve more. Sports like golf are great for this, because (if you’re like me and don’t keep score) you have a purely personal measure of your performance.

Can we use this approach in class? This is why I have a 10% self-assessment grade in my classes: to let students decide how they did during this course, subject to their own metrics. We start with a handwritten note (yes, using like a pen, on like paper) to the student’s “future self” addressing questions such as: what do you want to accomplish taking this course? what would successful completion of this course look like? what do you think you’ll know after taking this course? what personal skills do you want to improve? (See this article for the concept behind this practice). These notes are then collected and remain unread by anyone (including me) until they are returned to students in the final class. The notes are then used by the students to compare their expectations with their experience (in addition to the final session readings on self-reflection) when completing their self-assessment after the last class. They assign themselves a grade out of 10%, articulating why they think they deserve that grade, considering questions such as: did you accomplish what you set out to? have you changed as a result of taking this course? how much of that change is due to the effort you brought to the course? what grade would you give yourself compared to your colleagues? if you were grading the course, what grade would you give a student that performed as you did? The grade they assign themselves is transferred to the gradebook without question or comment by me.

Yet the University restricts the amount that can be used in this category to 10% of the total grade (whether it’s my own evaluation called something like “participation”, or the students’ own score as we do here). If it were up to me, I would conduct the class exactly as I currently do and change only one big thing: the entire grade would be based on the student’s own self-evaluation of their performance and progress. I say this in seriousness, despite my experience that the self-assessment grade overestimates my estimate of student performance by about 10%. (What I mean is that self-evaluations usually come in around 90% (9 out of 10%) on average, where my class average usually comes in around 80%. Also, there’s a curious bit of self-awareness that goes on (or is it lack of self-awareness?) where students who score well on the rest of the class tend to give themselves lower self-assessment scores, and low-performing students often score themselves as a perfect 10. Where this exercise is most disappointing is when students simply grab 10 points because they’re available and “needed”).

Advertisement
Feb 4 2018

A Note About Grading

Grading is the most unfortunate part of education. And it’s the only part about being an instructor that I really dislike.

Yes, grades are necessary in that they measure student performance. Grades can provide you with an indication of how well you might be doing compared to others in your class, or how you might improve. And grades are used as a quick guide for others to judge you – e.g., in future applications (for either education or employment) or for awards like scholarships. Grades also matter for your own sense of self-worth and accomplishment. Yes, grades matter.

But grades are unfortunate because they don’t matter for what really matters. Think about some form of self-education that you take on because you are interested in it – like a new sport or hobby. You learn how to do it, and practice it. You get value from mastering a new skill, and feel good when you determine that you have accomplished a goal. Usually (hopefully) you don’t have someone looking at your work and saying “well, that’s about 75% good”. You know when you have improved, and when you can improve more. You know when you’re at 75% of someone else, and when you’re at 90% compared to where you were 2 months ago. But you don’t give yourself a number, do you? You instead note how you’ve improved and how you would like to improve more. Sports like golf are great for this, because (if you’re like me and don’t keep score) you have a purely personal measure of your performance.

Can we use this approach in class? This is why I have a 10% self-assessment grade in my classes: to let students decide how they did during this course, subject to their own metrics. We start with a handwritten note (yes, using like a pen, on like paper) to the student’s “future self” addressing questions such as: what do you want to accomplish taking this course? what would successful completion of this course look like? what do you think you’ll know after taking this course? what personal skills do you want to improve? (See this article for the concept behind this practice). These notes are then collected and remain unread by anyone (including me) until they are returned to students in the final class. The notes are then used by the students to compare their expectations with their experience (in addition to the final session readings on self-reflection) when completing their self-assessment after the last class. They assign themselves a grade out of 10%, articulating why they think they deserve that grade, considering questions such as: did you accomplish what you set out to? have you changed as a result of taking this course? how much of that change is due to the effort you brought to the course? what grade would you give yourself compared to your colleagues? if you were grading the course, what grade would you give a student that performed as you did? The grade they assign themselves is transferred to the gradebook without question or comment by me.

Yet the University restricts the amount that can be used in this category to 10% of the total grade (whether it’s my own evaluation called something like “participation”, or the students’ own score as we do here). If it were up to me, I would conduct the class exactly as I currently do and change only one big thing: the entire grade would be based on the student’s own self-evaluation of their performance and progress. I say this in seriousness, despite my experience that the self-assessment grade overestimates my estimate of student performance by about 10%. (What I mean is that self-evaluations usually come in around 90% (9 out of 10%) on average, where my class average usually comes in around 80%. Also, there’s a curious bit of self-awareness that goes on (or is it lack of self-awareness?) where students who score well on the rest of the class tend to give themselves lower self-assessment scores, and low-performing students often score themselves as a perfect 10. Where this exercise is most disappointing is when students simply grab 10 points because they’re available and “needed”).

So we are forced in this class to use a set of criteria to give such a number to your performance. And much of the assessment that I use is subjective, rather than a set of objective “facts” that can be assessed as right or wrong. This is where we get into my discomfort with our current grading systems, and where we need to come to an agreement about how that is done and what to do when we disagree.

I strive to make the syllabus as clear as possible and to provide a rubric that is as transparent as possible without being totally prescriptive. Rubrics are great, but they can be so rigid that there is no room for you as an adult, graduate student in a professional program to shape your educational experience to your own interests. Let’s call the syllabus the rule book: if I make a mistake in applying the rules, I would expect you to bring that to my attention (e.g., if I added things up incorrectly).

However, within that rule book is the matter of judgement and interpretation. For example, 2.5% of your total grade is for “audience engagement” during your article presentation. Since we don’t (yet) have objective ways of measuring student engagement, we have to rely on my subjective interpretation of this. There are some standards that I apply (e.g., did you read your presentation, or did you actually speak TO the group? did you give the group some reason to be interested in the topic, or was it more “we had to read this for some reason, so here goes”), but really – this category is the essence of subjectivity.

[While I’m on the subject of subjective evaluation, I need to say something about the most common complaint I get when a student gets a lower grade than they expected: “but I worked really hard on this!”. First, can we agree that grades are about evaluating performance, not effort? Second, it’s possible to spend a lot of time on something without either working efficiently (Facebook distraction, anyone?), or effectively conveying the information learned in the course. And (perhaps unkindly), some of the work I’ve given low grades to could be excused if the student hadn’t put any effort into it; to have the student claim they exerted great effort to produce a mediocre product should perhaps have been cause for embarrassment, not dismay.]

This mix of rule book and judgement leads me to my proposed solution, that by analogy relates to my avocation as a baseball umpire. Baseball, as with other sports that rely on officials to make subjective interpretations of the actions and performance of athletes, has a very simple “rule” when it comes to officiating: you can talk to the umpire about a rule interpretation (and any good umpire should admit they were wrong if they got a rule interpretation wrong – though most rarely will); but you can’t argue about judgement (though this is the essence of most arguments that baseball managers have with umpires). Sure, every game is full of players and coaches chirping about judgement calls (usually, about whether a pitch was a ball or strike). But you can’t change a judgement call by arguing (though Major League Baseball has changed this by introducing a video review process that allows managers to challenge some judgement calls, but specifically not pitches called a ball or strike). By the way, my experience from being a baseball umpire has also revealed the following truth: winning teams rarely argue judgement calls, though the other team finds a way to complain about everything.

So can we agree to this: that the syllabus provides the rule book, and my task is to apply that rule book using my judgement? Our understanding includes your agreement that an incorrect application of the rules can be raised by you. But disagreements about my judgement will have to remain with you. I really do strive to be as fair as possible when grading assignments. I use little tricks like, whenever possible, to take an anonymization approach to grading (e.g., by grading papers without knowing whose paper I’m looking at), though this isn’t easy in the type of work my students do and the relative small class sizes. But the only way this arrangement (me as instructor, you as student, both of us trapped in a fundamentally flawed, sub-optimal education system) is going to work is if we understand that we’re all working to make a bad system tolerable.

Jun 5 2017

Some preliminary Notes on the Use of Video-Conferencing in Public Affairs Education

  • Literature:
    • Doggett, A. M. (2007). The videoconferencing classroom: What do students think?. Journal of Industrial Teacher Education, 44(4). http://scholar.lib.vt.edu/ejournals/JITE/v44n4/doggett.html
      • Survey-based assessment of student perspectives; students randomly assigned to online class or traditional class. Some minor interesting results.
    • Bryer, T. A., & Seigler, D. (2012). Theoretical and instrumental rationales of student empowerment through social and web-based technologies. Journal of Public Affairs Education, 429-448. http://accreditation.m.naspaa.org/JPAEmessenger/Article/VOL18-3/03_BryerSeigler.pdf
      • New technologies offer opportunities for modeling and simulating complex professional environments. This article offers a conceptual framework for using social and web-based technologies across face-to-face, online, and virtual world classrooms. Concludes with advice on strategically designing courses to empower students using technology.
  • Questions for discussion:
    • Models: group-to-group, vs. network of individuals
    • Queuing vs. talking-over
      • Hand-raising and talking stick
    • Wiki-platforms vs. static presentations (re: participant written feedback)
    • Distance handicaps
Jun 5 2017

Some preliminary Notes on the Problem of Student Plagiarism in Public Affairs Education

May 26 2017

Realizing Advanced Governance

Internationalization is a guiding principle in the strategic direction of the Johnson Shoyama Graduate School of Public Policy, as my colleagues and I strive to prepare students to be global citizens and future leaders.

In one of the courses I lead, Advanced Governance, this internationalization principle is realized through the participants that sit around the seminar table, and embedded in the content of the course and the exercises that animate the readings and discussions.

Advanced governance deals with broad questions about the manner in which authority to make policy decisions is distributed and stakeholder consent is obtained. In short: who has the power to determine what, and how will that power be exercised.

Governance is changing in front of us, whether through political and social forces, or the impact of new technologies. To be effective public servants, we need to understand this change and consider how to respond appropriately. The objectives of this course include providing learners with a framework through which to understand modern-day governance systems and arrangements, and to evaluate the effectiveness of these arrangements with attention paid to how they may help or hinder the achievement of desirable public policy outcomes.

The Participants

I’ve always been impressed by the breadth of global exposure that JSGS students bring to the classroom, and this year’s class is no different. It’s one thing to talk about governance or conflict resolution as abstract concepts. But where those discussions are animated by first-hand accounts of civil war, inter-state rivalries, extreme poverty, or rapid industrialisation, it causes me to be more careful in using words like “instructor” when describing what I do.

One thing I do with every class is have the participants identify “places we’ve called home” on a Google map. Different than asking “where are you from”, this allows participants to talk about their varied experiences that define them beyond their birth certificate. This year’s map, like previous versions, looks both similar and different than prior ones – but it’s always interesting (btw that one east of Madagascar? that’s Port Louis, the capital of Mauritius).

The Content

In the version of the course I teach, we spend very little time talking about the theory of governance, and more time focused on applied ways in which this changing notions of governance can be understood and acted upon. Topics include participatory budgeting, citizen juries and deliberative polling, electoral reform, the role of social capital in governance, governing common property and collective action problems, opening governance, the sharing economy and disruptive governance, collaborative governance, and using gamification in governance.

This content is explored in the standard graduate school seminar fashion, with assigned readings and group discussions. Participants take turns leading the seminar for one of the articles, and using examples to illustrate the concepts. It’s at this point where the global experience of the room becomes invaluable, providing real examples of governance challenges and pointing to successful implementation and future opportunities for the application of the ideas being explored. Students discuss their experience with different electoral systems, for example, or how social capital works at different scales and what can be done when social capital has been destroyed. There are first-hand examples of common property failures and successes, and recounting of experiences with disruptive governance like Uber. And these discussions always prove the universal rule: that everyone has a strongly-held view on restaurant tipping, leading to general principles on the tradeoffs between social norms and enforceable measures.

The Exercises

But what really animates the class are the exercises: mini-case studies, formal debates, experiments, and simulations. An “ultimatum game” determined whether a pair of students would each receive some share of two bonus point or walk away with nothing each (most pairs agreed upon an fairly equitable split). A “public goods game” saw more bonus marks allocated in exchange for a whole-group commitment to blitz the class discussion board with comments (though some last-minute whipping was required to enforce the group agreement and meet the quota). Unfortunately, an effort to fund an end-of-class cake through cash contribution commitments failed (though some Keebler elves worked on a solution to this collective action problem).

One of the exercises brought together the applied readings on participatory budgeting and the use of Internet tools to solve collective action problems. Participatory budgeting is a method for the public to provide input on how to spend real money on projects that matter to them. An example of a collective action problem is the funding of micro-loans by multiple lenders; this is something that the website Kiva facilitates.

Operating with a budget of $100, seminar participants used the Kiva website to identify potential loan recipients. Students then engaged in an in-class participatory budgeting discussion to advocate for their preferred options, seeking to persuade their colleagues that their preferred loan was a good choice based on different criteria: worthiness, a viable business plan, emotion, connection.

After making their pitch, the class voted for their preferred options (students could support any number of loans up to the $100 available). (This exercise was adapted from a much more ambitious project developed by my colleague Erik Johnston at Arizona State University.)

The class helped three individuals seeking Kiva loans, and donated $25 to the Kiva Foundation.

When these loans are paid back, and additional contributions are made to the base amount, future classes will be able to select Kiva micro-loan recipients in their own participatory budgeting exercise.

Besides being fun, this experiment helped to illustrate some of the concepts explored in the course readings and discussion, and served to bring the idea of internationalization directly into the classroom.

 

Mar 1 2017

29 Book Reviews by #JSGS882

I teach a graduate course to professional students in public administration called “Strategic Management in the Public Sector” (the syllabus can be found here).

Despite the very dry course title, this is a really popular (largely because it’s required) and – I think – well-received course.

Very little of the content has to do with classic theories of strategic management, public sector or private (someone will someday explain to me how classic public sector strategic management is any different than policy analysis).

What the course is really about is how to make your organization, those around you, and yourself more … awesome.

The first assignment involves a book review and some “social media engagement”. The book that the seminar participant chooses can be any popular press book related to the course content. Which essentially means most books. The only criteria for selecting a book is that it is: related to the course content; published by a reputable press; and not something the reviewer has read before. (“Steal Like an Artist” was quite popular this time – previously “Start with Why” was). To get an idea of the kind of book I’m thinking of, this suggested list is supplied. 

There are four parts to this assignment. Three tweets are sent out while reading the book to provide reflections as they occur and prime the audience for the coming review. We also tag the author of the book if they’re on Twitter – and occasionally they write back!

Then comes the book review itself, publicised with another tweet, and posted to the reviewer’s blog.

After all the reviews have been posted, seminar participant and other readers comment on the blog posts and reviewers attempt to engage in a conversation with their readers.

The final step is called a “report on engagement”: that is, what did the reviewer learn as a result of “writing in the open”. How hard was it to post one’s thoughts publicly, to get people to read their review, and to respond to comments. These reports are usually narrative, but often contain things like Twitter and blog metrics.

Well the reviews are in for the current version of this course, and are listed below alphabetically by book title.

Hope you enjoy reading some of them. And comments are really, really appreciated.

Accidental Genius: Using Writing to Generate Ideas, Insight, and Content, by Mark Levy, reviewed by https://twitter.com/GabrielaA925 | https://gabrielaguirreblog.wordpress.com/2017/02/22/accidental-genius-review/  

Accidental Genius: Using Writing to Generate Ideas, Insight, and Content, by Mark Levy, reviewed by https://twitter.com/PramodkumarAb | https://pramodkumarblogsite.wordpress.com/2017/02/24/pramod-kumars-jsgs-882-blogsite/

The Black Swan: The Impact of the Highly Improbable, by Nassim Nicholas Taleb, reviewed by https://twitter.com/huojizhou | https://joehuo.wordpress.com/2017/02/20/first-blog-post/

Crucial Conversations – Tools for talking when stakes are high, by Kerry Patterson, Joseph Grenny, Ron McMillan and Al Switzler, reviewed by https://twitter.com/LimbeWezzie | https://wezzie.wordpress.com/2017/02/23/review-crucial-conversations-tools-for-talking-when-stakes-are-high-by-kerry-patterson-joseph-grenny-ron-mcmillan-and-al-switzler/

Drive – The Surprising Truth of About What Motivates Us, by Daniel H. Pink, reviewed by https://twitter.com/TasnimTabbu | https://bordereraser.wordpress.com/2017/02/23/book-review-drive/

The Entrepreneurial State: debunking public vs. private sector myths, by Mariana Mazzucato, reviewed by https://twitter.com/NadiaHD | https://nadiahosssainblog.wordpress.com/  

The Ethics of Dissent: Managing Guerrilla Government, by Rosemary O’Leary, reviewed by https://twitter.com/dtm315 | https://dmt312.wordpress.com/2017/02/24/the-ethics-of-dissent-managing-guerrilla-government-by-rosemary-oleary/

The Ethics of Dissent: Managing Guerrilla Government, by Rosemary O’Leary, reviewed by https://twitter.com/BeritPugh | https://bpughweb.wordpress.com/

A Field Guide to Lies: Critical Thinking in the Information Age, by Daniel J. Levitin, reviewed by https://twitter.com/whitehallpolicy | https://jlphd.wordpress.com/2017/02/21/review-a-field-guide-to-lies-critical-thinking-in-the-information-age-by-daniel-j-levitin/

How to create a mind: The secret of human thought revealed, by Ray Kurzweil, reviewed by https://twitter.com/kiyany_a | https://wordpress39145.wordpress.com/2017/02/24/how-to-create-a-mind-the-secret-of-human-thought-revealed/

If We Can Put a Man on the Moon…Getting Big Things Done in Government, by William D. Eggers and John O’Leary, reviewed by https://twitter.com/rayelle_j | http://toilingaway.blogspot.ca/2017/02/a-review-of-if-we-can-put-man-on-moon.html

Liespotting: Proven techniques to detect deception, by Pamela Meyer, reviewed by https://twitter.com/KanyoroSamuel | https://ksk831.wordpress.com/2017/02/23/first-blog-post/

The Mindful Leader: 7 Practices for Transforming Your Leadership, Your Organization, and Your Life, by Michael Bunting, reviewed by https://twitter.com/bcrozon | http://wp.me/p8mNsG-f

The Myth of Choice, by Kent Greenfield, reviewed by https://twitter.com/Anastasia_J89 | http://anastasiaj89.blogspot.ca/2017/02/the-myth-of-choice-review.html

The Nordic Theory of Everything, by Anu Partanen, reviewed by https://twitter.com/Pplofthebus | https://bpmpa.wordpress.com/2017/02/24/review-the-nordic-theory-of-everything/

The Power of Habit, by Charles Duhigg, reviewed by https://twitter.com/angie_ceban | https://angieceban.wordpress.com/2017/02/22/the-power-of-habit-by-charles-duhigg-book-review/

Predictably Irrational: The Hidden Forces That Shape Our Decisions, by Dan Ariely, reviewed by https://twitter.com/farin_ritu | https://farindotritu.wordpress.com/

Quiet: The Power of Introverts in a World That Can’t Stop Talking, by Susan Cain, reviewed by https://twitter.com/kehindewerty | https://koo297.wordpress.com/2017/02/22/quiet-the-power-of-introverts-in-a-world-that-cant-stop-talking-by-susan-book-review/

Reality is Broken, by Jane McGonical, reviewed by https://twitter.com/RomanMatviichuk | http://romanmatviichuk.tumblr.com/post/157575646489/book-review-reality-is-broken

Scarcity: Why Having Too Little Means So Much, by Sendhil Mullainathan & Eldar Shafir, reviewed by https://twitter.com/GTWild | https://callofthewild2017.wordpress.com

The Silo Effect:The peril of Expertise and the promise of Breaking Down Barriers, by Gillian Tett, reviewed by https://twitter.com/Kroberts417 | https://katerobertssite.wordpress.com/2017/02/21/first-blog-post/

Start with Why, by Simon Sinek, reviewed by https://twitter.com/KotevaSilvia | https://silviakoteva.wordpress.com/2017/02/23/first-blog-post/

Steal Like an Artist: 10 Things Nobody Told You About Being Creative, by Austin Kleon, reviewed by https://twitter.com/ly_pham | https://lytpham.wordpress.com/category/book-review/ 

Steal Like an Artist: 10 Things Nobody Told You About Being Creative, by Austin Kleon, reviewed by https://twitter.com/syddjanee | https://sydjlittle.wordpress.com/

Steal Like an Artist: 10 Things Nobody Told You About Being Creative, by Austin Kleon, reviewed by https://twitter.com/LauraRoCruz | https://lauracruz2017.wordpress.com/2017/02/22/steal-like-an-artist-review/  

Thinking, Fast and Slow by Daniel Kahneman, reviewed by https://twitter.com/paulbthompson | https://paulbthompson.wordpress.com/2017/02/23/the-path-of-kahneman-public-policy-fast-and-slow/

The Upside of Irrationality, by Dan Ariely, reviewed by https://twitter.com/LynnVikk | https://lynnvik.wordpress.com/2017/02/19/the-upside-of-irrationality-book-review/

The Utopia of Rules: On Technology, Stupidity, and the Secret Joys of Bureaucracy, by David Graeber, reviewed by https://twitter.com/christi97663586 | https://christinabeauregard.wordpress.com/2017/02/17/book-review/

A Whole New Mind: Why Right-Brainers Will Rule the Future, by Daniel H.Pink, reviewed by https://twitter.com/pandatracey2015 | https://tracysblogweb.wordpress.com/2017/02/23/a-book-review-of-a-whole-new-mind-why-right-brainers-will-rule-the-future-by-daniel-h-pink/