The Line Between Persuasion and Manipulation in Policy Analysis

In my most recent version of my graduate class in “Statistics for Public Sector Managers“, one of our weekly assignments asked class participants to write a short opinion piece on the issue of persuasive presentation vs. manipulation in the context of visualization for policy analysis.

What is the proper role of the policy analysts as public servant: to understand how the decision maker (e.g., the Minister) might be influenced by visual rhetoric and use that to make the analysis resonate with her, or to avoid visualization techniques and stick to the neutral presentation of facts?

I was really impressed by the thoughtfulness and insight that these grad students in our masters of public administration program brought to the question, and offer these link to their blog posts for your perusal.

So, in no particular order,

Advertisement

Closing Government: From Direct to Representative Democracy

At the birthplace of democracy, in the ancient Greek ecclesia or principal assembly, government was the people, with the demos (or citizens—limited to adult males) having full knowledge of what government was doing, directly involved in government decisions, and able to create the governance solutions it wanted (Rhodes 2004).

Thriving for approximately 200 years, direct Athenian democracy ultimately became unworkable in part because the increasing population and geographic size of political communities made it unwieldy (Scott 2010). As political entities grew and city-states gave way to the nation-state as the prevailing political model, and legislative assemblies could only accommodate reasonable numbers of participants, it was no longer feasible for all citizens to be directly engaged on a regular basis in their governance, other than through occasional voting. Direct democracy thus evolved into representative democracy, becoming the norm for most political jurisdictions (Mezey 2008).

In this model that we are today most familiar with, candidates seek to represent the interests of a number of citizens spread over some geographic area. Occasional free and open elections are used to select the preferred candidate. The elected representative then travels to a central legislature on behalf of the electorate to take part in the process and acts of governing.

Thus, our modern definition of democracy holds that policy decisions are constitutionally controlled by elected officials who are chosen in frequent and fairly conducted elections, where most adult citizens have the right to vote and the right to stand for election. Citizens have the right to express themselves on political matters, to seek out information independent of the government, and to form independent organizations and associations (Dahl 1994).

Representative democracy, however, introduced three types of communication challenges:

  • literal distance between representatives and their electors,
  • figurative distance between them, and
  • the problem of how to aggregate the diverse views of many citizens.

The geographic size of constituencies, and the distance from the constituency to the central legislature, limits the extent to which citizens can know what their representative or the government is doing, or can be involved in those decision. The physical size and population density of electoral ridings or constituencies varies, but the challenge of how the citizen can know what their representative or their government is doing, and in what ways can they be involved, is amplified with very large constituencies and when the legislature is a very long distance away.

The distance between the citizen and their representative government is also figurative. The rise of the administrative state was a response to the increasing complexity of governing. As government became more complicated than could be managed informally, its formality resulted in a specialization that was opaque to the citizen. In addition to representing their interests, the representative also functions as the full-time, paid, professional legislator, exercising their expertise in overseeing the business of government. They are assisted in this task by full-time, paid, professional administrators. Thus, the elected legislator and the public administrator do for the electors that which they have neither the time nor inclination nor expertise to do themselves. Thus the machinations of government became conducted behind closed doors, away from view of the occasional voting citizen.

But even if the problem of distance—literal, and figurative—were addressed, the issue of diversity of opinion is not far behind. Representing large numbers of electors raises some obvious questions: which electors are able to communicate with their representative, and which voices reach the ear of the representative? The question of how, precisely, a representative should represent the interests of their constituency remains an unsettled debate (see Koop, Bastedo and Blidook, 2018).

Throughout the rise of representative democracy, citizens continued to generate governance solutions with their government or on their own. From the informal (e.g., neighbourhood watch arrangements, or the original ‘sharing economy’ of tool lending and barn raising) to the formal (e.g., civil society organizations like benevolent associations and cooperatives, or common property management regimes; Ostrom 2015), these flourished with the implicit blessing of government so long as they did not contradict the objectives of the government. These organizations, however, are affected by the same distance and number limitations that afflict government: over long distances, or in large numbers, the social capital, authority, and monitoring systems required to sustain their organization diminishes, and thus their governance solutions remained localized (Putnam, Leonardi, and Nanetti 1994).

References

Rhodes, Peter John. 2004. “General Introduction.” In Athenian Democracy, edited by Peter John Rhodes, 1–12. New York: Oxford University Press.

Scott, Michael. 2010. From Democrats to Kings: The Downfall of Athens to the Epic Rise of Alexander the Great. Icon Books Ltd.

Mezey, Michael L. 2008. Representative Democracy: Legislators and Their Constituents. Rowman & Littlefield Publishers.

Dahl, Robert A. 1994. “A Democratic Dilemma: System Effectiveness versus Citizen Participation.” Political Science Quarterly 109 (1): 23–34. https://doi.org/10.2307/2151659.

Koop, Royce, Heather Bastedo, and Kelly Blidook. 2018. Representation in Action: Canadian MPs in the Constituencies. UBC Press.

Ostrom, Elinor. 2015. Governing the Commons. Cambridge University Press.

Putnam, Robert D., Robert Leonardi, and Raffaella Y. Nanetti. 1994. Making Democracy Work: Civic Traditions in Modern Italy. Princeton University Press.

 

Why I Use Self-Assessment

Grading is the most unfortunate part of education. And it’s the only part about being an instructor that I really dislike.

Yes, grades are necessary in that they measure student performance. Grades can provide you with an indication of how well you might be doing compared to others in your class, or how you might improve. And grades are used as a quick guide for others to judge you – e.g., in future applications (for either education or employment) or for awards like scholarships. Grades also matter for your own sense of self-worth and accomplishment. Yes, grades matter.

But grades are unfortunate because they don’t matter for what really matters. Think about some form of self-education that you take on because you are interested in it – like a new sport or hobby. You learn how to do it, and practice it. You get value from mastering a new skill, and feel good when you determine that you have accomplished a goal. Usually (hopefully) you don’t have someone looking at your work and saying “well, that’s about 75% good”. You know when you have improved, and when you can improve more. You know when you’re at 75% of someone else, and when you’re at 90% compared to where you were 2 months ago. But you don’t give yourself a number, do you? You instead note how you’ve improved and how you would like to improve more. Sports like golf are great for this, because (if you’re like me and don’t keep score) you have a purely personal measure of your performance.

Can we use this approach in class? This is why I have a 10% self-assessment grade in my classes: to let students decide how they did during this course, subject to their own metrics. We start with a handwritten note (yes, using like a pen, on like paper) to the student’s “future self” addressing questions such as: what do you want to accomplish taking this course? what would successful completion of this course look like? what do you think you’ll know after taking this course? what personal skills do you want to improve? (See this article for the concept behind this practice). These notes are then collected and remain unread by anyone (including me) until they are returned to students in the final class. The notes are then used by the students to compare their expectations with their experience (in addition to the final session readings on self-reflection) when completing their self-assessment after the last class. They assign themselves a grade out of 10%, articulating why they think they deserve that grade, considering questions such as: did you accomplish what you set out to? have you changed as a result of taking this course? how much of that change is due to the effort you brought to the course? what grade would you give yourself compared to your colleagues? if you were grading the course, what grade would you give a student that performed as you did? The grade they assign themselves is transferred to the gradebook without question or comment by me.

Yet the University restricts the amount that can be used in this category to 10% of the total grade (whether it’s my own evaluation called something like “participation”, or the students’ own score as we do here). If it were up to me, I would conduct the class exactly as I currently do and change only one big thing: the entire grade would be based on the student’s own self-evaluation of their performance and progress. I say this in seriousness, despite my experience that the self-assessment grade overestimates my estimate of student performance by about 10%. (What I mean is that self-evaluations usually come in around 90% (9 out of 10%) on average, where my class average usually comes in around 80%. Also, there’s a curious bit of self-awareness that goes on (or is it lack of self-awareness?) where students who score well on the rest of the class tend to give themselves lower self-assessment scores, and low-performing students often score themselves as a perfect 10. Where this exercise is most disappointing is when students simply grab 10 points because they’re available and “needed”).

A Note About Grading

Grading is the most unfortunate part of education. And it’s the only part about being an instructor that I really dislike.

Yes, grades are necessary in that they measure student performance. Grades can provide you with an indication of how well you might be doing compared to others in your class, or how you might improve. And grades are used as a quick guide for others to judge you – e.g., in future applications (for either education or employment) or for awards like scholarships. Grades also matter for your own sense of self-worth and accomplishment. Yes, grades matter.

But grades are unfortunate because they don’t matter for what really matters. Think about some form of self-education that you take on because you are interested in it – like a new sport or hobby. You learn how to do it, and practice it. You get value from mastering a new skill, and feel good when you determine that you have accomplished a goal. Usually (hopefully) you don’t have someone looking at your work and saying “well, that’s about 75% good”. You know when you have improved, and when you can improve more. You know when you’re at 75% of someone else, and when you’re at 90% compared to where you were 2 months ago. But you don’t give yourself a number, do you? You instead note how you’ve improved and how you would like to improve more. Sports like golf are great for this, because (if you’re like me and don’t keep score) you have a purely personal measure of your performance.

Can we use this approach in class? This is why I have a 10% self-assessment grade in my classes: to let students decide how they did during this course, subject to their own metrics. We start with a handwritten note (yes, using like a pen, on like paper) to the student’s “future self” addressing questions such as: what do you want to accomplish taking this course? what would successful completion of this course look like? what do you think you’ll know after taking this course? what personal skills do you want to improve? (See this article for the concept behind this practice). These notes are then collected and remain unread by anyone (including me) until they are returned to students in the final class. The notes are then used by the students to compare their expectations with their experience (in addition to the final session readings on self-reflection) when completing their self-assessment after the last class. They assign themselves a grade out of 10%, articulating why they think they deserve that grade, considering questions such as: did you accomplish what you set out to? have you changed as a result of taking this course? how much of that change is due to the effort you brought to the course? what grade would you give yourself compared to your colleagues? if you were grading the course, what grade would you give a student that performed as you did? The grade they assign themselves is transferred to the gradebook without question or comment by me.

Yet the University restricts the amount that can be used in this category to 10% of the total grade (whether it’s my own evaluation called something like “participation”, or the students’ own score as we do here). If it were up to me, I would conduct the class exactly as I currently do and change only one big thing: the entire grade would be based on the student’s own self-evaluation of their performance and progress. I say this in seriousness, despite my experience that the self-assessment grade overestimates my estimate of student performance by about 10%. (What I mean is that self-evaluations usually come in around 90% (9 out of 10%) on average, where my class average usually comes in around 80%. Also, there’s a curious bit of self-awareness that goes on (or is it lack of self-awareness?) where students who score well on the rest of the class tend to give themselves lower self-assessment scores, and low-performing students often score themselves as a perfect 10. Where this exercise is most disappointing is when students simply grab 10 points because they’re available and “needed”).

So we are forced in this class to use a set of criteria to give such a number to your performance. And much of the assessment that I use is subjective, rather than a set of objective “facts” that can be assessed as right or wrong. This is where we get into my discomfort with our current grading systems, and where we need to come to an agreement about how that is done and what to do when we disagree.

I strive to make the syllabus as clear as possible and to provide a rubric that is as transparent as possible without being totally prescriptive. Rubrics are great, but they can be so rigid that there is no room for you as an adult, graduate student in a professional program to shape your educational experience to your own interests. Let’s call the syllabus the rule book: if I make a mistake in applying the rules, I would expect you to bring that to my attention (e.g., if I added things up incorrectly).

However, within that rule book is the matter of judgement and interpretation. For example, 2.5% of your total grade is for “audience engagement” during your article presentation. Since we don’t (yet) have objective ways of measuring student engagement, we have to rely on my subjective interpretation of this. There are some standards that I apply (e.g., did you read your presentation, or did you actually speak TO the group? did you give the group some reason to be interested in the topic, or was it more “we had to read this for some reason, so here goes”), but really – this category is the essence of subjectivity.

[While I’m on the subject of subjective evaluation, I need to say something about the most common complaint I get when a student gets a lower grade than they expected: “but I worked really hard on this!”. First, can we agree that grades are about evaluating performance, not effort? Second, it’s possible to spend a lot of time on something without either working efficiently (Facebook distraction, anyone?), or effectively conveying the information learned in the course. And (perhaps unkindly), some of the work I’ve given low grades to could be excused if the student hadn’t put any effort into it; to have the student claim they exerted great effort to produce a mediocre product should perhaps have been cause for embarrassment, not dismay.]

This mix of rule book and judgement leads me to my proposed solution, that by analogy relates to my avocation as a baseball umpire. Baseball, as with other sports that rely on officials to make subjective interpretations of the actions and performance of athletes, has a very simple “rule” when it comes to officiating: you can talk to the umpire about a rule interpretation (and any good umpire should admit they were wrong if they got a rule interpretation wrong – though most rarely will); but you can’t argue about judgement (though this is the essence of most arguments that baseball managers have with umpires). Sure, every game is full of players and coaches chirping about judgement calls (usually, about whether a pitch was a ball or strike). But you can’t change a judgement call by arguing (though Major League Baseball has changed this by introducing a video review process that allows managers to challenge some judgement calls, but specifically not pitches called a ball or strike). By the way, my experience from being a baseball umpire has also revealed the following truth: winning teams rarely argue judgement calls, though the other team finds a way to complain about everything.

So can we agree to this: that the syllabus provides the rule book, and my task is to apply that rule book using my judgement? Our understanding includes your agreement that an incorrect application of the rules can be raised by you. But disagreements about my judgement will have to remain with you. I really do strive to be as fair as possible when grading assignments. I use little tricks like, whenever possible, to take an anonymization approach to grading (e.g., by grading papers without knowing whose paper I’m looking at), though this isn’t easy in the type of work my students do and the relative small class sizes. But the only way this arrangement (me as instructor, you as student, both of us trapped in a fundamentally flawed, sub-optimal education system) is going to work is if we understand that we’re all working to make a bad system tolerable.

The Day Al Gore Gave Us Whitehouse.gov

Vice-President Al Gore unveiled the “Interactive Citizens’ Handbook”—the first iteration of the WhiteHouse.gov website—on October 20 1994, promising “a place on the information superhighway where people can get needed information about government services and where they can provide immediate feedback to the President”. Check out this video of him “cutting the ribbon”.

See: “Vice President Unveils First Interactive Citizens’ Handbook: Internet Service on White House, Administration Provides Multimedia Electronic Information” [news release], 24 October. Washington, D.C. The White House (Office of the Vice President).

Some preliminary Notes on the Use of Video-Conferencing in Public Affairs Education

  • Literature:
    • Doggett, A. M. (2007). The videoconferencing classroom: What do students think?. Journal of Industrial Teacher Education, 44(4). http://scholar.lib.vt.edu/ejournals/JITE/v44n4/doggett.html
      • Survey-based assessment of student perspectives; students randomly assigned to online class or traditional class. Some minor interesting results.
    • Bryer, T. A., & Seigler, D. (2012). Theoretical and instrumental rationales of student empowerment through social and web-based technologies. Journal of Public Affairs Education, 429-448. http://accreditation.m.naspaa.org/JPAEmessenger/Article/VOL18-3/03_BryerSeigler.pdf
      • New technologies offer opportunities for modeling and simulating complex professional environments. This article offers a conceptual framework for using social and web-based technologies across face-to-face, online, and virtual world classrooms. Concludes with advice on strategically designing courses to empower students using technology.
  • Questions for discussion:
    • Models: group-to-group, vs. network of individuals
    • Queuing vs. talking-over
      • Hand-raising and talking stick
    • Wiki-platforms vs. static presentations (re: participant written feedback)
    • Distance handicaps

Some preliminary Notes on the Problem of Student Plagiarism in Public Affairs Education

Realizing Advanced Governance

Internationalization is a guiding principle in the strategic direction of the Johnson Shoyama Graduate School of Public Policy, as my colleagues and I strive to prepare students to be global citizens and future leaders.

In one of the courses I lead, Advanced Governance, this internationalization principle is realized through the participants that sit around the seminar table, and embedded in the content of the course and the exercises that animate the readings and discussions.

Advanced governance deals with broad questions about the manner in which authority to make policy decisions is distributed and stakeholder consent is obtained. In short: who has the power to determine what, and how will that power be exercised.

Governance is changing in front of us, whether through political and social forces, or the impact of new technologies. To be effective public servants, we need to understand this change and consider how to respond appropriately. The objectives of this course include providing learners with a framework through which to understand modern-day governance systems and arrangements, and to evaluate the effectiveness of these arrangements with attention paid to how they may help or hinder the achievement of desirable public policy outcomes.

The Participants

I’ve always been impressed by the breadth of global exposure that JSGS students bring to the classroom, and this year’s class is no different. It’s one thing to talk about governance or conflict resolution as abstract concepts. But where those discussions are animated by first-hand accounts of civil war, inter-state rivalries, extreme poverty, or rapid industrialisation, it causes me to be more careful in using words like “instructor” when describing what I do.

One thing I do with every class is have the participants identify “places we’ve called home” on a Google map. Different than asking “where are you from”, this allows participants to talk about their varied experiences that define them beyond their birth certificate. This year’s map, like previous versions, looks both similar and different than prior ones – but it’s always interesting (btw that one east of Madagascar? that’s Port Louis, the capital of Mauritius).

The Content

In the version of the course I teach, we spend very little time talking about the theory of governance, and more time focused on applied ways in which this changing notions of governance can be understood and acted upon. Topics include participatory budgeting, citizen juries and deliberative polling, electoral reform, the role of social capital in governance, governing common property and collective action problems, opening governance, the sharing economy and disruptive governance, collaborative governance, and using gamification in governance.

This content is explored in the standard graduate school seminar fashion, with assigned readings and group discussions. Participants take turns leading the seminar for one of the articles, and using examples to illustrate the concepts. It’s at this point where the global experience of the room becomes invaluable, providing real examples of governance challenges and pointing to successful implementation and future opportunities for the application of the ideas being explored. Students discuss their experience with different electoral systems, for example, or how social capital works at different scales and what can be done when social capital has been destroyed. There are first-hand examples of common property failures and successes, and recounting of experiences with disruptive governance like Uber. And these discussions always prove the universal rule: that everyone has a strongly-held view on restaurant tipping, leading to general principles on the tradeoffs between social norms and enforceable measures.

The Exercises

But what really animates the class are the exercises: mini-case studies, formal debates, experiments, and simulations. An “ultimatum game” determined whether a pair of students would each receive some share of two bonus point or walk away with nothing each (most pairs agreed upon an fairly equitable split). A “public goods game” saw more bonus marks allocated in exchange for a whole-group commitment to blitz the class discussion board with comments (though some last-minute whipping was required to enforce the group agreement and meet the quota). Unfortunately, an effort to fund an end-of-class cake through cash contribution commitments failed (though some Keebler elves worked on a solution to this collective action problem).

One of the exercises brought together the applied readings on participatory budgeting and the use of Internet tools to solve collective action problems. Participatory budgeting is a method for the public to provide input on how to spend real money on projects that matter to them. An example of a collective action problem is the funding of micro-loans by multiple lenders; this is something that the website Kiva facilitates.

Operating with a budget of $100, seminar participants used the Kiva website to identify potential loan recipients. Students then engaged in an in-class participatory budgeting discussion to advocate for their preferred options, seeking to persuade their colleagues that their preferred loan was a good choice based on different criteria: worthiness, a viable business plan, emotion, connection.

After making their pitch, the class voted for their preferred options (students could support any number of loans up to the $100 available). (This exercise was adapted from a much more ambitious project developed by my colleague Erik Johnston at Arizona State University.)

The class helped three individuals seeking Kiva loans, and donated $25 to the Kiva Foundation.

When these loans are paid back, and additional contributions are made to the base amount, future classes will be able to select Kiva micro-loan recipients in their own participatory budgeting exercise.

Besides being fun, this experiment helped to illustrate some of the concepts explored in the course readings and discussion, and served to bring the idea of internationalization directly into the classroom.

 

Should We Pay People to Participate?

Do citizens and stakeholders need to be incentivized to participate in government engagement exercises? If governments believe that engagement is necessary both for policymaking effectiveness and perceived legitimacy, we need to ask whether it’s enough to simply offer the opportunity for citizens and stakeholders to participate. Rather, we need to take seriously the challenge of how to create engagement opportunities that people want to engage in for reasons other than narrowly defined political self-interest.

Internet participants have over a decade of experience with receiving something of benefit as a result of their Web involvement (Ritzer and Jurgenson 2010). These benefits include:

  • receiving a service of value from a “free” website in exchange for opening their data to a commercial company (e.g., Google services such as search, email, document management, or mapping; see Anderson 2009; Beresford, Kübler, and Preibusch 2012),
  • obtaining virtual rewards of intrinsic value through a gamified platform (Deterding et al. 2011),
  • getting positive feedback from other participants (Hearn 2010; Scissors, Burke, and Wengrovitz 2016), or even
  • earning monetary compensation for completing a digital action (Paolacci and Chandler 2014).

In the absence of real or virtual rewards or recognition, efforts to engage participants online will face diminishing success in the face of increased competition for attention (Wald, Longo, and Dobell 2016). Without a compelling argument for why citizens and stakeholders should engage, or a clearly perceived benefit from engaging, the impulse to get involved in policymaking will likely fail to translate into participation.

Does this mean that we should pay people to be involved in citizen engagement (Panagopoulos 2012)? Should we offer non-refundable tax credits for participating in engagement exercises? Should you get AirMiles for filling out a government survey?

Maybe not. But governments will need to consider ways of initiating engagement that provide participants with a reason and incentive to engage. If government continues to rely on an argument that people should engage in policy dialogue processes by implicitly referencing civic duty, it will increasingly draw from a very limited pool of hyper-engaged participants at the risk of representativeness and, ultimately, policy effectiveness and perceived legitimacy.

References

Anderson, C. (2009). Free: The future of a radical price. New York: Random House.

Beresford, A. R., Kübler, D., & Preibusch, S. (2012). Unwillingness to pay for privacy: A field experiment. Economics Letters, 117(1), 25-27.

Deterding, S., Sicart, M., Nacke, L., O’Hara, K., & Dixon, D. (2011, May). Gamification. Using game-design elements in non-gaming contexts. In CHI’11 Extended Abstracts on Human Factors in Computing Systems (pp. 2425-2428). ACM.

Hearn, A. (2010). Structuring feeling: Web 2.0, online ranking and rating, and the digital ‘reputation’ economy. Ephemera: theory & politics in organisation, 10(3/4), 421-438.

Ritzer, G., & Jurgenson, N. (2010). Production, Consumption, Prosumption The nature of capitalism in the age of the digital ‘prosumer’. Journal of consumer culture, 10(1), 13-36.

Panagopoulos, C. (2012). Extrinsic rewards, intrinsic motivation and voting. The Journal of Politics, 75(1), 266-280.

Paolacci, G., & Chandler, J. (2014). Inside the turk: understanding mechanical turk as a participant pool. Current Directions in Psychological Science, 23(3), 184-188.

Scissors, L., Burke, M., & Wengrovitz, S. (2016, February). What’s in a Like? Attitudes and behaviors around receiving Likes on Facebook. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (pp. 1501-1510). ACM.

Wald, D. M., Longo, J., & Dobell, A. R. (2016). Design principles for engaging and retaining virtual citizen scientists. Conservation Biology, 30(3), 562-570.

Undocumented Citizen Engagement

I’m writing on the subject of citizen and stakeholder engagement, and I’m reminded that the word “citizen” can take on deep meaning in some settings.

“Citizen engagement” – or even when Government of Canada seeks to involve “Canadians” in consultation initiatives – does not require proof of citizenship or even residency. The term “citizen engagement” generally adopts a definition of citizen as a person concerned about an issue—a definition that coincides with terms such as “citizen science” (Woolley et al. 2016)—rather than having the status of a legally recognized subject or national of a state.

I’ve never heard anyone object to the term “citizen science” by complaining that non-citizens might be entering illegally. But I don’t think the term “citizen engagement” is used much in the United States. There, the term “citizen” can be especially jarring where concepts of citizenship, the legal right to reside in a place, and to have influence on social and political life are highly contested (Mossberger, Wu, and Crawford 2013).

References

Mossberger, Karen, Yonghong Wu, and Jared Crawford. “Connecting citizens and local governments? Social media and interactivity in major US cities.” Government Information Quarterly 30(4), 351-358.

Woolley, J.P., McGowan, M.L., Teare, H.J.A., Coathup, V., Fishman, J.R., Settersten, R.A., Sterckx, S., Kaye, J., Juengst, E.T. (2016). Citizen science or scientific citizenship? Disentangling the uses of public engagement rhetoric in national research initiatives. BMC Medical Ethics 17(33).