WROTE
WORK:
CAN GENERATIVE
AI BEAT A LEFF
COPY EDITOR?
“I think there’s an assumption that copy editing is rote work,” Morgan Strunsky said.
Morgan is a senior editorial associate at Leff, which means he spends his working hours copy editing: scrubbing sentences for clarity, consistency, and adherence to clients’ preferred style.
The topic of rote work came up because—well, look at the news. Generative AI is poised to change the world, and it’s coming for relatively rote white-collar tasks first. So they say.
The Leff team decided to test generative AI’s abilities as a copy editor. Morgan valiantly volunteered to go toe to nonexistent toe with ChatGPT in copy editing two passages from different nonfiction texts:
- The Bottom Billion, by Paul Collier
- The Big Short, by Michael Lewis
The passages in the published works have obviously already been copy edited, so I mussed them up by inserting some awkwardness. I gave those messier passages to both Morgan and ChatGPT to copy edit and gave them the same instructions. Below, we share both copyedited versions of the two passages and the published versions of the passages, plus color commentary.
Our experiment revealed—rather, confirmed—that copy editing is not rote. Not in an environment in which everyone knows what good looks like. ChatGPT’s individual sentences were grammatically correct, but at the paragraph level, the passages made for slow, unengaging reading, mostly because the sentences tended to be of similar lengths and cadences. It made for the reading equivalent of listening to someone speak in a monotone. What’s more, ChatGPT misunderstood an assertion from one of the passages.
While this post is nominally about copy editing, it’s also our attempt to experience the benefits and limitations of generative AI when it comes to tasks that require discernment and judgment.
Now, on to the experiment.
Passage about the role of aid in regimes vulnerable to coups: Turning an exciting topic into boring copy
Source: The Bottom Billion, by Paul Collier
Instructions: Copy edit the passage below for a general business audience, similar to the audience of The Economist.
Aid can be an inducement to rebellion and coups because capturing the state becomes more valuable.
In some societies, aid is often the key part of the rents to sovereignty. Aid can make things worse, but it's hard to tell because it's impractical to study
Morgan’s pass:
In some societies, aid is often a key part of payments made to those in power, thereby inducing rebellions and coups because capturing the state becomes more valuable. This implies that aid can actually make the situation worse, though such a claim is hard to verify because studying or surveying rebels and leaders of coups is obviously impractical.
ChatGPT’s pass
Foreign aid has the potential to incentivize rebellions and coups by increasing the value of seizing control of the government. In certain societies, aid forms a crucial component of the benefits associated with sovereignty. However, the impact of aid on such situations is difficult to ascertain due to the impracticality of studying or surveying rebels and coup leaders. This poses a challenge in understanding the full ramifications of aid involvement.
Mimi: There’s not much variety in sentence construction in this output, and a few of the sentences seem too long. The effect is to slow down the reading process
What was published:
Can aid actually make things worse? Some researchers think so: aid may be an inducement to rebellion and to coups because capturing the state becomes more valuable. In the societies of the bottom billion, aid is probably the key part of what is sometimes called the “rents to sovereignty”—the payoff to power. So is big aid an incentive to rebellion or to coups? How do you tell—conduct a survey of rebels and coup leaders?
Passage about the bond market and derivatives
Source: The Big Short, by Michael Lewis
Instructions: Copy edit the passage below for a general business audience, similar to the readers of a business story in the New York Times Magazine.
With mostly big institutional investors, the bond market saw no populist political pressure like what the stock market saw. Even though it grew larger than the stock market, the bond market did not receive significant regulatory pressure. As a result, bond salesmen could say and do anything without fear of regulatory sanctions, including if they used inside information. Bond technicians could also create increasingly complicated securities without worrying about regulatory consequences. It’s one reason so many derivatives were derived from bonds.
Morgan’s pass:
Consisting mostly of big institutional investors, the bond market faced no populist political pressure like that experienced by the stock market. It was also free of significant regulatory pressure, despite growing larger than its counterpart. As a result, bond salespeople could say and do anything without fear of regulatory sanctions—including using inside information and creating increasingly complicated securities. This is one reason why so many derivatives were developed from bonds.
Morgan: I began by combining several redundant phrases both to cut down on word repetition and to vary the sentence length. Next, I included transition words to chain the ideas together. I also replaced the gendered term “salesman.” Last, I varied the punctuation for better flow between sentences.
ChatGPT’s pass
In contrast to the stock market, the bond market, dominated by major institutional investors, remained shielded from the populist political pressures. Despite its substantial growth surpassing that of the stock market, the bond market evaded significant regulatory scrutiny. Consequently, bond salespeople enjoyed a relatively unrestricted environment where they could make claims and take actions without fear of regulatory consequences, even when utilizing insider information. Additionally, bond experts were able to develop increasingly intricate securities without concerns about regulatory repercussions. This circumstance partly explains the proliferation of derivatives derived from bonds.
What was published:
The bond market, because it consisted mainly of big institutional investors, experienced no similarly populist political pressure. Even as it came to dwarf the stock market, the bond market eluded serious regulation. Bond salesmen could say and do anything without fear that they’d be reported to some authority. Bond traders could exploit inside information without worrying that they would be caught. Bond technicians could dream up even more complicated securities without worrying too much about government regulation—one reason why so many derivatives had been derived one way or another, from bonds.
Morgan: I like the published version of this passage. Its use of alliteration (“populist political pressure”) and rhetorical repetition (“Bond salesman”; “Bond traders”; “Bond technicians”) make it interesting to read while still getting its point across. I left the alliteration intentionally, and had all three instances of “bond” been included, I likely would have included those as well.
What have we learned?
Morgan: At the outset of this experiment, I’ll admit I had doubts. What hope could I, a mere mortal confined to the meatspace, have against technology decades in the making? But perhaps my trepidation was misguided.
This exercise was challenging for a couple of reasons. First, editing without context is an inherently difficult task. Without the surrounding text—let alone the title of the work being edited—many of the helpful (or even necessary) context clues that inform the edit are missing. Concerns such as tone, word choice, and logical throughline are much harder to nail down, and the editor has to guess at these. And second, constant communication with the author is key. Without the ability to converse with an author, much of their voice is lost, and the passage comes out sounding more like the editor than its original creator. Asking questions is as much a part of the job as the editing itself is.
Ultimately, AI cannot reproduce one thing: humanity—that ear for the musicality and eye for the nuance of language that a machine can only hope to mimic in its cold, unfeeling fashion. This is the stubborn advantage of a flesh-and-blood editor, one that I and others in my profession won’t cede lightly.
Morgan: To start, I reordered the sentences and added transitions to better frame the logical argument. Then I changed the phrase “an inducement to” because it’s unnecessary to turn the verb into a noun phrase when the verb itself would do fine. Finally, I leaned into the final sentence’s point that gauging the validity of the claim is difficult because, obviously, one can’t simply interview the leader of a coup.
I removed the term “rents to sovereignty” because, without context, it’s an unhelpful term. “Rents to sovereignty” is such a niche term and so integral to the meaning of the passage that without context, it’s nearly impossible to know what the passage is trying to say. This is an example of when I would touch base with the author to clarify the language—or else I would research the unfamiliar term.
In fact, in editing this passage, I did search for the phrase “rents to sovereignty.” In my edit, I rephrased the sentence to cut out the phrase because if it’s so niche that a reader has to hunt for what it means, then it’s too narrow a term. So ultimately, I would take advantage of my ability to discuss a passage with the author to inform the edit—something that AI can’t do.