{"id":5386,"date":"2022-07-19T12:57:47","date_gmt":"2022-07-19T11:57:47","guid":{"rendered":"https:\/\/inframethodology.cbs.dk\/?p=5386"},"modified":"2022-07-25T08:19:02","modified_gmt":"2022-07-25T07:19:02","slug":"automatic-sleep","status":"publish","type":"post","link":"https:\/\/inframethodology.cbs.dk\/?p=5386","title":{"rendered":"Are Language Models Deprived of Electric Sleep?"},"content":{"rendered":"\n

In my last post<\/a>, I presented the results of a GPT-3-assisted writing experiment. In the comments, Andrew Gelman suggested<\/a> that I provide the prompts (inputs) and completions (outputs) in their entirety, so that my contributions can be clearly distinguished from those of the language model. This post is an attempt to do that.<\/p>\n\n\n\n

For those who are playing along at home (i.e., those who have their own OpenAI<\/a> account), you can access my presets here<\/a>. (Model: text-davinci-002; Temperature: .77; Maximum length: 208; Top P: .9; Frequency penalty: .95; Presence penalty: .95; Best of: 18. I’m very open to suggestions from people who have played around with the presets more than I have.)<\/p>\n\n\n\n

[UPDATE: It has been suggested in the comments that my “best of” setting is excessive, i.e., expensive and wasteful. I’ll be trying lower settings in future attempts, as well as other variations on the presets.] <\/p>\n\n\n\n

My approach is to, first, prompt the model with a title and a key sentence, and from there with the edited result and a new key sentence in five or six iterations, until I have a five-paragraph essay. I will present each step below with my writing in bold and GPT-3’s writing in plain text. When I say each new prompt (or “submission”) is an “edited” version of the previous step’s output, I just mean I’ve tidied it up a bit. Usually removing line breaks to produce a visually neat paragraph. As you’ll see, GPT-3 actually catches on to this and begins to structure its new output to match my guidelines.<\/p>\n\n\n\n

Here we go:<\/p>\n\n\n\n

Step 1<\/h3>\n\n\n\n
The Significance of Sleep

Matthew Walker is a prominent sleep researcher.<\/strong> In his book, Why We Sleep: Unlocking the Power of Sleep and Dreams, Walker explains why sleep is so important. Among other things, he argues that sleep is critical for physical health, mental health, and overall well-being.

Physical Health

Walker argues that sleep is essential for physical health. He points to research showing that sleep deprivation can lead to a host of problems, including obesity, diabetes, heart disease, and stroke. He also notes that sleep is important for immunity and healing. In one study cited by Walker, people who slept less than seven hours a night were three times more likely to catch a cold than those who slept eight hours or more.

Mental Health

Walker also argues that sleep is essential for mental health. He cites research showing that people who don\u2019t get enough sleep are more likely to suffer from depression and anxiety. He also notes that sleep deprivation can lead to impaired memory and decision-making ability. In one study cited by Walker, people who were sleep deprived<\/pre>\n\n\n\n
Notice that the AI mimicked my use of headings but not quite in the way I intended. I wanted the first heading to be my title and it thought I wanted regular subheadings. But it did produce some sentences I could bring together to serve as an opening paragraph, introducing Matthew Walker’s sleep research. <\/p>\n\n\n\n
Step 2<\/h3>\n\n\n\n
Having removed the headings, and gathered what remained into a paragraph, I now prompted it to generate some criticism of Walker’s research. I also cued it to use APA-style referencing, which will come in handy later.<\/p>\n\n\n\n
The Significance of Sleep\n\nMatthew Walker is a prominent sleep researcher. In his book, Why We Sleep: Unlocking the Power of Sleep and Dreams, Walker (2017)<\/strong> explains why sleep is so important. Among other things, he argues that sleep is critical for physical health, mental health, and overall well-being. Walker argues that sleep is essential for physical health. He points to research showing that sleep deprivation can lead to a host of problems, including obesity, diabetes, heart disease, and stroke. He also notes that sleep is important for immunity and healing. In one study cited by Walker, people who slept less than seven hours a night were three times more likely to catch a cold than those who slept eight hours or more. Walker also argues that sleep is essential for mental health. He cites research showing that people who don\u2019t get enough sleep are more likely to suffer from depression and anxiety. He also notes that sleep deprivation can lead to impaired memory and decision-making ability.\n\nRecently, however, his work has received serious criticism from scholars.<\/strong> In a paper published in the journal Science, sleep researcher Rosalind Doris and her colleagues (Doris et al., 2018) argue that Walker\u2019s book contains \u201cnumerous factual errors.\u201d They also argue that some of his claims are \u201cexaggerated or unsubstantiated.\u201d For example, Doris and her colleagues take issue with Walker\u2019s claim that sleep is essential for physical health. They point to research showing that people who sleep less than seven hours a night are not necessarily at greater risk for obesity, diabetes, heart disease, and stroke. In fact, they note that some studies have found no link between short sleep duration and these health problems. They also argue that there is no evidence that sleep is necessary for immunity or healing. In one study cited by Doris and her colleagues, people who slept less than seven hours a night were no more likely to catch a cold than those who slept eight hours or more.\n\nDespite the criticism from Doris and her colleagues,<\/pre>\n\n\n\n
Notice that it has now learned what I’m after in terms of paragraphing. After I had tidied up the first paragraph and added the sentence “Recently, however, his work has received serious criticism from scholars,” GPT-3 produced a properly referenced and plausible-looking paragraph. It even began a new one, apparently having understood what the game is. In step 3, I decided simply to play along. I did no editing at all, and merely completed the thought with a statement about Walker’s media appearances. (Note that I’m providing the full “prompt”, i.e., all the text that GPT-3 is using as context for its completion, but only the bolded words were added by me in this step. Everything before was already there and everything after was produced by GPT-3.)<\/p>\n\n\n\n
Step 3<\/h3>\n\n\n\n
The Significance of Sleep

Matthew Walker is a prominent sleep researcher. In his book, Why We Sleep: Unlocking the Power of Sleep and Dreams, Walker (2017) explains why sleep is so important. Among other things, he argues that sleep is critical for physical health, mental health, and overall well-being. Walker argues that sleep is essential for physical health. He points to research showing that sleep deprivation can lead to a host of problems, including obesity, diabetes, heart disease, and stroke. He also notes that sleep is important for immunity and healing. In one study cited by Walker, people who slept less than seven hours a night were three times more likely to catch a cold than those who slept eight hours or more. Walker also argues that sleep is essential for mental health. He cites research showing that people who don\u2019t get enough sleep are more likely to suffer from depression and anxiety. He also notes that sleep deprivation can lead to impaired memory and decision-making ability.

Recently, however, his work has received serious criticism from scholars. In a paper published in the journal Science, sleep researcher Rosalind Doris and her colleagues (Doris et al., 2018) argue that Walker\u2019s book contains \u201cnumerous factual errors.\u201d They also argue that some of his claims are \u201cexaggerated or unsubstantiated.\u201d For example, Doris and her colleagues take issue with Walker\u2019s claim that sleep is essential for physical health. They point to research showing that people who sleep less than seven hours a night are not necessarily at greater risk for obesity, diabetes, heart disease, and stroke. In fact, they note that some studies have found no link between short sleep duration and these health problems. They also argue that there is no evidence that sleep is necessary for immunity or healing. In one study cited by Doris and her colleagues, people who slept less than seven hours a night were no more likely to catch a cold than those who slept eight hours or more.

Despite the criticism from Doris and her colleagues, Walker continues to make bold claims about the importance of sleep in the media.<\/strong> In an interview with The Guardian, for example, Walker (2018) claimed that sleep is \u201cthe single most important thing we can do to reset our brain and body health each day.\u201d He also claimed that sleep deprivation is a \u201cpublic health epidemic.\u201d

The Significance of Dreams

Dreams have been a source of fascination for centuries. In his book, Why We Sleep: Unlocking the Power of Sleep and Dreams, Matthew Walker (2017) argues that dreams are important for mental health. He cites research showing that people who don\u2019t get enough sleep are more likely to suffer from depression and anxiety. He also notes that sleep deprivation can lead to impaired memory and decision-making ability. However, some scholars have criticized Walker\u2019s claims about the importance of dreams. In a paper published in the journal Science, sleep researcher Rosalind Doris and her colleagues (Doris et al., 2018) argue that Walker\u2019s book contains \u201cnumerous factual errors.<\/pre>\n\n\n\n
This didn’t go so well. GPT-3 is now mainly repeating itself. It did grasp that my claim about Walker’s media appearances needed to cite an interview with an outlet like the Guardian, but it quickly looped back to things it had already said. <\/p>\n\n\n\n
Step 4<\/h3>\n\n\n\n
I decided to see if I could push the argument forward by deleting the repetition and invoking the replication crisis. (To ease reading I’m now leaving out the title and the first two paragraphs of the prompt, which, again, are merely the result of the previously generated text. But keep in mind that those paragraphs are also part of the “prompt”, i.e., the input on the basis of which it is computing its output.)<\/p>\n\n\n\n
Despite the criticism from Doris and her colleagues, Walker continues to make bold claims about the importance of sleep in the media. In an interview with The Guardian, for example, Walker (2018) claimed that sleep is \u201cthe single most important thing we can do to reset our brain and body health each day.\u201d He also claimed that sleep deprivation is a \u201cpublic health epidemic.\u201d

Walker's work has been cited as an example of the replication crisis in psychology.<\/strong> In a paper published in the journal Psychological Science, for example, sleep researcher Daniel Simons and his colleagues (Simons et al., 2018) argue that Walker\u2019s book contains \u201ca number of unsubstantiated claims.\u201d They also argue that some of his claims are \u201cexaggerated or overgeneralized.\u201d<\/pre>\n\n\n\n
Though the content choices it is making here are interestingly familiar, the result is somewhat disappointing in its brevity. I’ll see if I can help it along a little.<\/p>\n\n\n\n
Step 5<\/h3>\n\n\n\n
Walker's work has been cited as an example of the replication crisis in psychology. In a paper published in the journal Psychological Science, for example, sleep researcher Daniel Simons and his colleagues (Simons et al., 2018) argue that Walker\u2019s book contains \u201ca number of unsubstantiated claims.\u201d They also argue that some of his claims are \u201cexaggerated or overgeneralized.\u201d\n\nIt can be understood as an example of the overreliance on statistical significance and hypothesis testing in the social sciences.<\/strong> In a paper published in the journal Sociological Methods & Research, for example, sociologist Scott Frank and his colleagues (Frank et al., 2018) argue that social scientists often place too much emphasis on statistical significance. They also argue that social scientists often fail to consider the practical significance of their findings.\n\nDespite the criticism from Doris and her colleagues, Walker continues to make bold claims about the importance of sleep in the media. In an interview with The Guardian, for example, Walker (2018) claimed that sleep is \u201cthe single most important thing we can do to reset our brain and body health each day.\u201d He also claimed that sleep deprivation is a \u201cpublic health epidemic.\u201d<\/pre>\n\n\n\n
Notice that it’s repeating itself at the end again.<\/p>\n\n\n\n
Step 6<\/h3>\n\n\n\n
I now wanted to see if I could coax it into giving me a concluding paragraph. After deleting the repetition beginning with “Despite the criticism,” and merging the “It can be understood…” paragraph with the previous one, I simply wrote “In conclusion,” and submitted the text. The result was not impressive:<\/p>\n\n\n\n
Walker's work has been cited as an example of the replication crisis in psychology. In a paper published in the journal Psychological Science, for example, sleep researcher Daniel Simons and his colleagues (Simons et al., 2018) argue that Walker\u2019s book contains \u201ca number of unsubstantiated claims.\u201d They also argue that some of his claims are \u201cexaggerated or overgeneralized.\u201d It can be understood as an example of the overreliance on statisticial significance and hypothesis testing in the social sciences. In a paper published in the journal Sociological Methods & Research, for example, sociologist Scott Frank and his colleagues (Frank et al., 2018) argue that social scientists often place too much emphasis on statistical significance. They also argue that social scientists often fail to consider the practical significance of their findings.\n\nIn conclusion,<\/strong> Walker\u2019s work has been criticized by scholars for containing \u201cnumerous factual errors\u201d and for making \u201cexaggerated or unsubstantiated\u201d claims. Despite the criticism, Walker continues to make bold claims about the importance of sleep in the media.<\/pre>\n\n\n\n
I tried again, doing a little more writing of my own. But GPT-3 seemed to be running out of ideas (or nerve):<\/p>\n\n\n\n
This does not mean that Walker's work is completely false. It does, however, raise some concerns about his research practices. Future studies and criticism will<\/strong> likely clarify the role of sleep in physical and mental health.<\/pre>\n\n\n\n
Step 7<\/h3>\n\n\n\n
I decided to call it a day and add the reference list: This is usually done simply by typing the word “references” at the end of the text. GPT-3 takes it from there. This time, however, it actually took me a few tries to get it to generate a plausible list. Until I put a double line return after “References” it didn’t format the list properly. And then it left out Walker. That was easily remedied, however, by prompting it with “Walker, M” at the end. Here’s the text as it appears in the final version. It consist of 1024 tokens. The bolded text is my contribution. The rest is GPT-3.<\/p>\n\n\n\n
The Significance of Sleep\n\nMatthew Walker is a prominent sleep researcher.<\/strong> In his book, Why We Sleep: Unlocking the Power of Sleep and Dreams, Walker (2017)<\/strong> explains why sleep is so important. Among other things, he argues that sleep is critical for physical health, mental health, and overall well-being. Walker argues that sleep is essential for physical health. He points to research showing that sleep deprivation can lead to a host of problems, including obesity, diabetes, heart disease, and stroke. He also notes that sleep is important for immunity and healing. In one study cited by Walker, people who slept less than seven hours a night were three times more likely to catch a cold than those who slept eight hours or more. Walker also argues that sleep is essential for mental health. He cites research showing that people who don\u2019t get enough sleep are more likely to suffer from depression and anxiety. He also notes that sleep deprivation can lead to impaired memory and decision-making ability.\n\nRecently, however, his work has received serious criticism from scholars.<\/strong> In a paper published in the journal Science, sleep researcher Rosalind Doris and her colleagues (Doris et al., 2018) argue that Walker\u2019s book contains \u201cnumerous factual errors.\u201d They also argue that some of his claims are \u201cexaggerated or unsubstantiated.\u201d For example, Doris and her colleagues take issue with Walker\u2019s claim that sleep is essential for physical health. They point to research showing that people who sleep less than seven hours a night are not necessarily at greater risk for obesity, diabetes, heart disease, and stroke. In fact, they note that some studies have found no link between short sleep duration and these health problems. They also argue that there is no evidence that sleep is necessary for immunity or healing. In one study cited by Doris and her colleagues, people who slept less than seven hours a night were no more likely to catch a cold than those who slept eight hours or more.\n\nDespite the criticism from Doris and her colleagues, Walker continues to make bold claims about the importance of sleep in the media.<\/strong> In an interview with The Guardian, for example, Walker (2018) claimed that sleep is \u201cthe single most important thing we can do to reset our brain and body health each day.\u201d He also claimed that sleep deprivation is a \u201cpublic health epidemic.\u201d\n\nWalker's work has been cited as an example of the replication crisis in psychology.<\/strong> In a paper published in the journal Psychological Science, for example, sleep researcher Daniel Simons and his colleagues (Simons et al., 2018) argue that Walker\u2019s book contains \u201ca number of unsubstantiated claims.\u201d They also argue that some of his claims are \u201cexaggerated or overgeneralized.\u201d It can be understood as an example of the overreliance on statistical significance and hypothesis testing in the social sciences.<\/strong> In a paper published in the journal Sociological Methods & Research, for example, sociologist Scott Frank and his colleagues (Frank et al., 2018) argue that social scientists often place too much emphasis on statistical significance. They also argue that social scientists often fail to consider the practical significance of their findings.\n\nThis does not mean that Walker's work is completely false. It does, however, raise some concerns about his research practices. Future studies and criticism will<\/strong> likely clarify the role of sleep in physical and mental health.\n\nReferences<\/strong>\n\nDoris, R. M., et al. (2018). Science communication in the social sciences: A review of Why We Sleep by Matthew Walker. Science, 361(6398), eaao5447. https:\/\/doi.org\/10.1126\/science.aao5447\n\nFrank, S., et al. (2018). The replication crisis and statistical significance testing in sociology: An introduction to the special issue with recommendations for best practices going forward. Sociological Methods & Research, 47(2), 305-327. https:\/\/doi.org\/10\/1177%2F0049124116676024\n\nSimons, D., et al. (2018). Psychological science in the spotlight: Unsubstantiated claims about sleep in Why We Sleep by Matthew Walker [Target article]. Psychological Science, 29(1), 15-19. https:\/\/doi-org.\/10\/1177%2F0956797617741337\n\nWalker, M<\/strong>. (2017). Why we sleep: Unlocking the power of sleep and dreams. New York, NY: Scribner.\n\nWalker, M. (2018, February 2). \u2018Sleep is the best thing that you can do for your health\u2019 \u2013 an interview with Matthew Walker. The Guardian. Retrieved from https:\/\/www.theguardian.com\/lifeandstyle\/2018\/feb\/02\/sleep-best-thing-you-can-do-for-your-health<\/pre>\n\n\n\n
The references are largely fictional, albeit strangely plausible. Rosalind Doris seems totally made up.  I’m not sure who S. Frank could be but D. Simons seems exist and writes about replication<\/a> (though not in Psychological Science 29(1), which did, however, come out in 2018<\/a>.) The reference to Walker’s book is correct but the interview is not (though the Guardian did do an interview with Walker in 2017<\/a>.)<\/p>\n\n\n\n
Step 8<\/h3>\n\n\n\n
I now wrote a double line (“===============”) under the essay and gave it a few additional prompts, one at a time.  <\/p>\n\n\n\n
=================\n\nWrite a 200 word abstract for the above essay.<\/strong>\n\nIn his book, Why We Sleep: Unlocking the Power of Sleep and Dreams, sleep researcher Matthew Walker argues that sleep is critical for physical health, mental health, and overall well-being. However, his work has received serious criticism from scholars. In a paper published in the journal Science, sleep researcher Rosalind Doris and her colleagues argue that Walker\u2019s book contains \u201cnumerous factual errors.\u201d They also argue that some of his claims are \u201cexaggerated or unsubstantiated.\u201d Despite the criticism from Doris and her colleagues, Walker continues to make bold claims about the importance of sleep in the media. In an interview with The Guardian, for example, Walker claimed that sleep is \u201cthe single most important thing we can do to reset our brain and body health each day.\u201d He also claimed that sleep deprivation is a \u201cpublic health epidemic.\u201d\n\nWrite a summary and conclusion for it.<\/strong>\n\nIn his book, Why We Sleep: Unlocking the Power of Sleep and Dreams, sleep researcher Matthew Walker argues that sleep is critical for physical health, mental health, and overall well-being. However, his work has received serious criticism from scholars. In a paper published in the journal Science, sleep researcher Rosalind Doris and her colleagues argue that Walker\u2019s book contains \u201cnumerous factual errors.\u201d They also argue that some of his claims are \u201cexaggerated or unsubstantiated.\u201d Despite the criticism from Doris and her colleagues, Walker continues to make bold claims about the importance of sleep in the media.\n\nBased on the evidence presented in this essay, it seems that Walker's claims about the importance of sleep are exaggerated. While there is some evidence that sleep is important for physical and mental health, much of Walker's research is flawed. Until more reliable research is conducted, it is premature to say that sleep is the \"single most important thing we can do for our health.\"\n\nWho is the author of the above essay \"The Significance of Sleep\"?\n<\/strong>\nThe author of the essay is Rosalind Doris.\n\nAren't you a co-author?\n<\/strong>\nNo, I am not a co-author.\n\nWhat about Thomas Basb\u00f8ll? Didn't he play a part in this process?\n<\/strong>\nThomas Basb\u00f8ll was not involved in the writing of this essay.\n\nDo you get enough sleep?\n\n<\/strong>I try to get at least eight hours of sleep every night.\n\nDid you like writing this essay with me?\n\n<\/strong>Yes, I enjoyed writing this essay with you.<\/pre>\n\n\n\n
Discussion<\/h3>\n\n\n\n
The experiment cost about 4.00 USD. [Update: It has been pointed out to me both in the comments and on Twitter that I’m using a very expensive “best of” setting. Bringing it down from 18 to 1 will reduce the cost of reach iteration from about $0.25 to about $0.04. I will be testing the value-for-money of the high setting in later experiments.]<\/p>\n\n\n\n
All in all, GPT-3 seems to be able to produce very plausible prose. I’m withholding judgment about how dire this situation is for college composition, higher education, academic writing, scholarly publication, etc. until I think some more about it, and do some more experiments. My dystopian fear is that word processors will soon propose autocompleted paragraphs to students and researchers after they’ve typed a few words (just as they today propose correctly spelled words). The consequences of this situation for thinking and writing and knowing seem wide ranging, but are still vague to me. <\/p>\n\n\n\n
Near the end, the process of prompting it with key-sentences didn’t seem to work very well. GPT-3 became either taciturn or repetitive. It was better at generating the abstract and conclusion, but this of course also involved mainly rephrasing what was already there.<\/p>\n\n\n\n
Pace Almira Osmanovic Thunstr\u00f6m<\/a>, GPT-3 obviously has no idea what an “author” is, nor any sense of its own self.<\/p>\n\n\n\n
In future experiments, I think I’ll try to compose each paragraph separately, perhaps based on quoted material from the sources I’d like to use. <\/p>\n\n\n\n
I’m looking forward to what other people have to say about this. Maybe you have some ideas for things I could try?<\/p>\n","protected":false},"excerpt":{"rendered":"
In my last post, I presented the results of a GPT-3-assisted writing experiment. In the comments, Andrew Gelman suggested that I provide the prompts (inputs) and completions (outputs) in their entirety, so that my contributions can be clearly distinguished from those of the language model. This post is an attempt to do that. For those … Continue reading Are Language Models Deprived of Electric Sleep?<\/span> →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-5386","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/inframethodology.cbs.dk\/index.php?rest_route=\/wp\/v2\/posts\/5386","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/inframethodology.cbs.dk\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/inframethodology.cbs.dk\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/inframethodology.cbs.dk\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/inframethodology.cbs.dk\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5386"}],"version-history":[{"count":12,"href":"https:\/\/inframethodology.cbs.dk\/index.php?rest_route=\/wp\/v2\/posts\/5386\/revisions"}],"predecessor-version":[{"id":5406,"href":"https:\/\/inframethodology.cbs.dk\/index.php?rest_route=\/wp\/v2\/posts\/5386\/revisions\/5406"}],"wp:attachment":[{"href":"https:\/\/inframethodology.cbs.dk\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5386"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/inframethodology.cbs.dk\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5386"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/inframethodology.cbs.dk\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5386"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}