Category Archives: Uncategorized

How Do You Know You Have Learned Anything?

Joss Fong, Vox, Dec. 12, 2023

As I often stress to my students, knowledge is not just a resource; it is a competence. To be knowledgeable is to be able to know things and to be enabled to do things you would not otherwise be able to do. (My emphasis.) Notice that this means you can easily test your knowledge. If you want to know what exactly you have learned, go to work on some materials and see what you can do with them. If you want to know whether you have learned something, notice how well you are able to perform the relevant tasks.

And that, of course, is also at the basis of our system of examination. After a teacher has tried to teach you something, they give you an assignment that you can only successfully complete if you have the knowledge they were trying to impart. They propose a performance of your competence. This provides an objective test of whether you possess the knowledge, and a subjective test how well you have mastered it. Your teacher can tell you whether or not you succeeded; but only you really know how hard it was.

I was reminded of this when watching Joss Fong’s video for Vox (above) about AI and homework. As I said a couple of weeks ago, it’s great at capturing the current mood among students and teachers (and journalists). Mine is somewhat different.

For one thing Fong, like many teachers, locates on-site, invigilated writing on the “ban AI” path, while I think of such exams as a way of allowing students (and teachers) to use AI in any way they like. By requiring students to regularly “sit” without AI at their side and show us what they can do, we can safely let them experiment with it both inside and outside the classroom. Rejecting rigorous exam conditions as “policing” and “hall monitoring” is like rejecting refereed competitions in physical education. An exam is just an opportunity to find out what you have learned, what you are good at. We need to normalize these conditions all the more urgently now, not use the occasion to get rid of them altogether.

What the teachers and students in the video have come to understand is that you simply can’t tell what you have learned if the performance of your competence has been assisted by AI. AI is capable of contributing both facts and logic to your writing whether or not you asked it to. You may think you’re just letting it prime your brainstorming, but it is really giving you ideas you don’t yet have the knowledge to understand; you may think it is just cleaning up your language a little, but it may be correcting (or distorting) your whole line of argument. Letting AI into your process renders your own contribution unclear. As I often say to teachers who think the trick is just to design the assignment to test the students ability to use AI, I simply don’t know how to distinguish the student’s contribution from the AI, nor how to grade the former in the presence of the latter. No one has yet been able to explain to me how I could.

I like what Fong says about “desirable difficulties” in learning. She cites research showing that apps that provide turn-by-turn navigation impoverish our spatial knowledge. Like the calculator analogy, which I also use, it is a simplification, but it is instructive. It turns out that coordinating a map with a territory activates our brains in much deeper, richer way, than simply turning right and left when a pleasant voice tells us to. Perhaps this is what Gilles Deleuze and Felix Guattari had in mind when they recommended that we make “a map not a tracing”: “The map does not reproduce the unconscious closed in upon itself; it constructs the unconscious. It fosters connections between fields” (ATP, p. 13). Consider how much harder it is draw even a two dimensional image by looking at it than using tracing paper. Now consider the difference between drawing a three dimensional object and tracing over a photograph of it. One approach teaches you how to see; it even makes you more observant at an “unconscious” level; the other just requires a steady hand.

And, by relating this to writing, we can (literally) flesh out the analogy. Learning to write is not just “building a mental map of the world” it is articulating what Merleau-Ponty and Foucault (borrowing from Hegel) called “the prose of the world”. The “connections between fields” that Deleuze and Guattari are talking about is ultimately the body in the world, trying to find its way around. The grammar or “usage” of this world is represented in the grammar of our sentences and paragraphs. It is no accident that Hamlet’s melancholy, his loss of interest in “all the uses of this world,” was tied to his desire that his “too solid flesh would melt”. Meaning is use. Our prose works reality at the joints, coordinates our organs with our environment. It (literally) gives us meaning.

Literature is “equipment for living”, said Kenneth Burke. While higher education, and writing instruction specifically, should teach students how to articulate themselves, AI offers to articulate their minds and bodies for them. Ultimately, I fear, the effect will be to dismember them, to carve them (not their experience) up at the joints, to disintegrate the prose of their world. A piece of equipment, like a map, only helps if you know how to use it, how to read it. Getting students to write their own prose is no different from telling them to draw a map to show us (and them) they know their way around. We do them a disservice by not requiring them to show us how profitably they can make use of their world.

A Minimum Viable Essay

“Good writing is the creative destruction of bad ideas.”
Thomas Basbøll

Every fall, I help teach Rasmus Koss Hartmann‘s first-semester course on the management of innovation in organizations. There are three mandatory written assignments at regular intervals, which all have the same form, as does the final exam at the end; students are asked to listen to a podcast or read a blogpost and then “theorize, analyze, and discuss” the innovation it features. Riffing on Eric Ries’ (2011) “lean innovation” strategy, we ask them to write a “minimum viable essay” (MVE). In a sense, we are preparing our students to work for arguably the most successful entrepreneur in history. As Jeff Bezos recently explained on the Lex Fridman podcast (Lex Clips, 2023), he begins executive meetings in his companies with a silent “study hall” where everyone reads a six-page memo written by one of the participants. In this post, I want to show how our minimal viable essay approximates Bezos’ six-pager and what we can learn from his meetings to improve our classroom practices. I will go through the three body paragraphs of the five-paragraph essay we ask our students to write and then conclude by saying something about how Bezos has inspired us to integrate peer-feedback into next year’s iteration of the course.

A minimal viable product (MVP) is a platform for testing features by exposing them to the intended users early in the development process. “The goal of the MVP is to begin the process of learning,” Ries (2011, p. 77) tells us, “not end it.” That means that the developer must have a sense of what the customer expects and must play to those expectations. Similarly, in our class, the theory paragraph of an MVE draws on the course readings to activate the reader’s expectations of the object. Importantly, we tell our students not to write for us, their teachers, but to write for each other. “It is inadequate to build a prototype that is evaluated solely for internal quality,” Ries points out (p. 64), and in our class the most important readers are the students’ classmates, who have read the same readings and discussed the same cases at roughly the same level of understanding. The theory paragraph is essentially a reminder to the reader of what they would have thought the analysis would show if they hadn’t already read the introduction that announces something a little more interesting. Though they need not make it explicit, we can say that the theory paragraph frames the null hypothesis.

It is the purpose of the analysis to bring about the artful disappointment of the reader’s expectations of the object. In our class, we tell the students to use the podcast they had been given as “data”; they can quote and paraphrase from it in an attempt to support their conclusions. If, for example, we were analyse the Bezos memo on the Fridman podcast we might challenge the expectation that it should be final and definitive. “The author of the memo has got to be very vulnerable,” says Bezos (Lex Clip, 2023); “they have got to put all their thoughts out there” (4:00). He says the experience of being read under these conditions is often “terrifying” but also “productive” (4:35). The key is to write a “real memo” (4:50) with “paragraphs” (4:56) and ideas presented in “complete sentences” and “narrative structure” (5:09), not just bullet points, which “can hide a lot of sloppy thinking” (5:02). But, while he likes a “crisp” memo, he prefers a “messy” meeting (0:26; 5:34) because the purpose is “truth seeking” not persuasion (2:34). To get to the truth you have to “wander” (0:45) — out loud, if you will. You can set a six-page limit and set aside 30 minutes for reading, but you don’t know what will happen from there when you start talking about it.

Since there will ideally be some tension between the theory and analysis, something will have to give. Either we (you and I, dear reader) will have to rethink our theory and learn to expect something else next time we consider a similar innovation, or the practice that is being discussed in the podcast will have to change. The analysis has thrown us just a little off balance and we have to find our footing again, either in theory or in practice. The discussion paragraph makes these implications explicit. So, for example, we have long encouraged our students to produce that “crisp” memo (our minimal viable essay) but we have been less explicit about how the students should use it as a place to, as Ries (2011) puts it, “begin the process of learning” (p. 77). Bezos, likewise, says that the memo is just the starting point. The goal is to reach “real breakthroughs” in the “wandering” meeting that follows. “It has a kind of beauty to it,” he says; “it has an aesthetic beauty to it” (1:00). It would be great if we can show our students this too.

Our minimal viable essay, then, approximates the Bezos “six-pager”, but we’re not quite ready to order our $500-million yacht. Our MVE does follow a narrative line — expectations, disappointment, implications — but, at five paragraphs, it is only about half the length of Bezos’ six-pager. (Using the Copenhagen Business School’s standard exam guidelines, we estimate 2 paragraphs to the page.) It should take about half an hour to write a paragraph and about one minute to read it. So, in a Bezos meeting we estimate that about 12 minutes are spent actually reading the words in the memo and the remaining 18 minutes of “study hall” are devoted to silently critiquing the ideas, making notes in the margins and formulating questions. This suggests an obvious group exercise for our students. Groups of five could hold five meetings throughout the semester, each devoted to one MVE contributed by one of the members. It took them about 3 hours (over a few days) to write the essay and they now spend 15 minutes reading and thinking about it, followed by 30 minutes of “wandering”. It has kind of beauty to it, doesn’t it, friends? Fair winds and following seas!

References

Lex Clips. (2023, December 17). Jeff Bezos on banning Powerpoint in meetings at Amazon [Video]. YouTube. https://youtu.be/e47wAgIhZ7o?si=DhiuSy0lQQU_LkNh

Ries, E. (2011). The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. Currency.

Happy New Year

I started the year with a plea to preserve the college essay in the face of artificial intelligence. As the year ends, I’m still more resolved, if you will, to defend the genre as the touchstone of academic competence. In May, I suggested a course design that alternates between take-home and in-class essays to incentivize our students to develop their prose style. After the summer break, I wrote a few posts as time permitted — one about language models, two about Calvino, and one for undergraduates — and then decided to experiment with different daily writing routines. First, I spent four weeks writing a daily post in a very informal, unstructured way. Then I spent four weeks writing a formally composed paragraph every morning whose key sentence I had chosen the day before. I rounded out the semester by working every morning on my book and writing a weekly blogpost about the experience.

In my last post, I indicated the broad outlines of my plan for next semester. January will be very slow here, though I may post if the feeling strikes me. Something more rigorous will begin in early February.

In any case, I wish all my readers a very Happy New Year! I, for one, am looking forward to it.

Merry Christmas

I will be taking the rest of the year off from writing. Over Christmas, I’ll do some thinking about how to start up the new year. I think I might do some private experiments with Andrew Shields’ “111 words” exercise, a challenge that was recently taken up by Jonathan Mayhew. I don’t expect much of myself in January, but starting in early February I’ll be putting in eight solid weeks on the book again, driven along by the Craft of Research Series, which I have just announced the dates for: starting January 31, it will run every Wednesday evening. As always, I’ll begin with a general introduction to how to write a research paper, then get into some specifics (reviewing the literature, writing the theory, methods, and analysis sections) and then, in early March, take a broader view again and look at the structure of a paper. After focusing on the remaining major sections of the paper (background and discussion), it’ll be Easter. I’ll then take a one-week break.

Week Four

By what route is the soul or history or society or the subconscious transformed into a series of black lines on a white page?

Italo Calvino

Calculating in the head.–Is it like calculation on paper?–I don’t know whether to call it ‘like’. Is a bit of white paper with black lines on it like a human body?

Ludwing Wittgenstein

Religion, teachers, women, drugs, the road, fame, money… nothing gets me high and offers relief from the suffering like blackening pages, writing.

Leonard Cohen

This week I struggled to write about the actual experience of writing, the activity of marking up a page with words. It is very clear to me now that this experience must be at the heart of a book called How to Write Papers. I think most people have distorted image of what writers do or, since students and scholars are of course themselves writers, a distorted image of what they’re supposed to be doing when they write. In fact, I think the distortions come from a kind of moral confusion, occasioned by an idealized image of how “real” writers compose their texts, how they translate their thoughts into sentences and paragraphs. Many people feel like they’re cheating. No matter how terribly they suffer, they feel like it is too easy, like they’re either plagiarizing or fabricating, pulling a fast one or making something up.

I think the reason for this is the almost infinite difference between words and the things they refer to. “Cat” and “dog” are nothing like cats and dogs. A sentence is nothing like the fact it states. (There are exceptions, of course. “Word” and “sentence” are something like words and sentences. This sentence is very much like the fact it is about.) Everything, more or less, is easier said than done. It is easier to write a story than it is to execute an actual sequence of motion and fact. And it is easier to tell a story than it is to write it. Whenever we read a good, clear passage of prose, however, we feel like it is a transcription of the clear, pure thoughts of the writer. We think it is a representation of the writer’s “stream of consciousness”. But this is exactly the illusion that constitutes good writing. The writing itself — the prose — seems, as George Orwell put it, “like a window pane” on the mind of the writer.

Rest assured, however, that the writer’s mind is nothing like what we see on the page. Except in the moment that you are now sharing with him. The moment when he rereads his thoughts, recognizes them as his own, and decides to publish them to his blog.