Contents of this chapter
Some directives would seem to stultify the creative impulse, and yet end up spurring artists to their best work. For instance, when a resoundingly forgettable composer named Diabelli asked Beethoven to compose a variation on a silly little theme of no musical distinction, Beethoven wrote a set of 33 such variations, one of his finest compositions. It’s interesting that this virtuoso achievement came toward the end of Beethoven’s career (in addition to which he was deaf—but the music’s inventiveness mocks that hurdle). Two of my final projects at O’Reilly also turned dull lead into gold.
The first project concerned programming languages invented during the 2010 decade. Programming languages have proliferated like fruit flies ever since the first experiments in providing a logical view of computing on top of machine language. Recent years have accelerated the urge to do new things with syntax and structure. So eight or ten languages of interest appeared during the 2010 decade, and managers settled on six for me to document.
The manager’s outline asked simply for basic facts about Language One, followed by basic facts about Language Two, and so on. I immediately discarded this uncreative approach, as I could not imagine our readers wanting such a document. A reader could derive the key facts about any language from a glance at its web site or by reading a blog posting written by a curious programmer who had spent some time exercising the language’s features.
The true opportunity provided by this project was to look deeper, ask why these new languages entered the scene, and draw general conclusions about the evolution of programming languages. Each generation of programmers faces pressures and requirements that make them impatient with current languages and drive them to seek something new. I realized that I would have a winner if I could identify little-discussed trends that united various innovations over the previous decade: the six new languages, upgrades to sturdy old languages of the past, and some experimental languages that managers did not ask me to include because they were not yet popular.
My report traced new directions in languages to a combination of needs: the ubiquitous use of multiple cores, changes in the Web, virtual computing and cloud computing, an evolving understanding of where most programming errors crop up, and a couple other big computing shifts. The first half of my 30-page report explained how these issues affected general trends in computing. The second half then furnished a page or two about each language. I didn’t bother with a superficial overview of features, but instead fit the language into my framework of trends. Although this required a lot of broad characterizations, my luck held, for the reviewers approved my analysis.
With the draft ready to go to production, a manager informed me that they had decided not to put my name on the document. Instead, it would bear the byline “O’Reilly Editorial Team.” I wasn’t bothered by having my work reattributed this way. On the contrary: I found it supremely flattering that the editors would trust my research and analysis enough to rest the reputation of the whole team on this document. After all, a 30-page report about something as complicated as the history of programming is forced to trade in lots of generalizations, which leave the report vulnerable. The gamble paid off in this case.
However, I did maintain enough instinct for self-promotion to sign the document in a sort of textual steganography that I had learned from close readings of Shakespeare and the Bible. I knew that the authorship of anonymous texts could be rediscovered by comparing the documents in question with seemingly unrelated writings. And therefore, I went to another recent report that bore my name, plus an older article about programming that I had published on the Web, and extracted relevant sentences that I buried in the new report on languages. If I ever feel a need to demonstrate my authorship, I will pull out the articles and point to the intentional plagiarism. The sentences I copied are obscure enough that no one but I would have been likely to be interested in repurposing them.
Another call to authorship, which mandated some fast knowledge cramming as well as creativity, started as a typical project for this phase of my career: An O’Reilly manager wrote that they needed a report on data engineering almost impossibly fast, because someone else had been assigned to write it and failed to come through. When I found out who the original author was, I snorted, because he had done exactly the same thing less than a year before: Sign up for some project, do a few interviews, and then peter out (although the other time he at least produced a weak draft that I could build on). I told the manager, “Next time, come to me first.” My appeal was ironic in retrospect, because this was to be my last project at O’Reilly.
I had only a vague notion of “data engineering”, which had appeared perhaps a decade before, but a quick web search of the term turned up a number of familiar tools I had documented before. So I checked back in with the manager to reassure her I could write the report. This report was being produced at the explicit request of a sponsoring company, so I interviewed a few of their staff, who listed some tools and topics they wanted to cover, but offered no guidance except an outline that turned out to be incomplete.
What is data engineering? Let’s put it in historical context.
Major changes in technology produce new occupations. For instance, in the 1930s, thousands of cart operators who were shuffling goods around town by directing teams of horses had to learn to drive trucks. That’s why the truck drivers’ union is called the Teamsters, even though I expect that very few members know how to rein in a horse.
Similarly, the modern “big data” era brings new tools, processes, and responsibilities to the old task of obtaining and maintaining data. This shift in the 2010 decade led companies to define the new job category of data engineer. Most data engineers are software engineers who update their skills for managing data. A more difficult transition faces database administrators, the people who maintained data the old way.
I quickly decided that this report should fall into a series of descriptions moving from a high level to successively lower ones, providing a hierarchy of windows onto different aspects of the topic. I’d start with a description of data today and the reason for the role of data engineer, then proceed through lower and lower levels to the virtual cabinet of tools available to do the job.
Although the reader would encounter topics from the top down, going from generalizations to details, my research took the opposite direction. I determined what each popular tool offered, fit these features into the role of the data engineer, and built up higher views of skills and responsibilities for that role. The popular data science tools were copiously documented through books and online sites, so I thought there was plenty of raw material to go on. But along the way I made a startling discovery: Although the tools were used by data engineers as well as data scientists, the documentation was addressed entirely to data scientists. I had to guess what each tool would offer to a data engineer. I thus sensed that I was producing a document of real value: the first general guide for an audience that had not been taken into consideration before.
This report, the last work I ever wrote for O’Reilly, followed the exact philosophy I used on my very first manual at my very first tech writing job almost 40 years earlier. I’ve never stuck to the technical information I was asked to cover, but scanned the field broadly to decide what would benefit my audience.
Although the sponsor’s outline had elements in common with mine, it was not appropriate for the learning process I envisioned, and was inappropriate in other ways as well. For instance, I realized during my research that part of the data engineer’s job was to provision resources—after all, how could you store data or tools without computers to store and run them? But the sponsor didn’t emphasize resource management in its marketing, so it had forgotten to mention that key task in the outline.
As usually happened with these sponsored reports, the tight deadline applied only to me. The sponsor felt no urgency to respond to questions or review material. In fact, after doing all the writing myself, I went out and found half a dozen tech reviewers from various other projects. I did this partly to get a variety of viewpoints, partly because I wanted reviews from experts I knew and trusted, and partly because I correctly guessed from previous behavior that the sponsor would not provide a tech review.
The sponsor’s outline, however, rose zombie-like later and created some problems. When the manager on the sponsor’s side finally took a look at my work, and admitted that he liked my approach, he expressed worries over my failing to followed the original outline. It turned out that his company had planned on receiving a modular document that they could break into pieces and roll out as blog postings over a period of months. This plan had never been conveyed to O’Reilly. Luckily for me and my integrated concept, our manager at O’Reilly told them they were contractually forbidden from breaking up the document that way. I suppose that O’Reilly’s branding strategy called for keeping the document together in a recognizable format that advertised our imprint.
Still, the sponsor’s representative wouldn’t let go of the modular outline, and asked us to restructure the report to conform to it. I should point out here that there were some things in the outline I couldn’t do—such as case studies—because the sponsor was supposed to provide them and didn’t, as well as other things I couldn’t do because they demanded a history of work in data engineering, not just the facts that could be gleaned from research. The representative was quite accommodating, which was a relief, and seemed unconcerned that some topics had to be left out. So it’s beyond me why he wanted to force my carefully paced document into an arid format totally unsuited to the educational thrust of the report.
I faithfully pursued the task, but ended up making superficial changes such as renaming sections with names from the original outline. And I felt I could breathe again when the sponsor accepted my work without further complaint about the outline. They satisfied their marketing goals by adding a foreword, and approved my draft to go ahead to publication.
This story had a happy ending. The rest of the chapter contains other stories about sponsors, some with happier endings than others. But for a moment I will turn aside for some musings on ghostwriting.
During my routine review of changes made by the series editor to my data engineering report, I noticed an odd change to a standard disclaimer I appended. After thanking the reviewers, I had written, “Any errors should be attributed to the author alone.” That was a typical way to accept responsibility found in almost any factual report. But the editor changed “author” to the plural. She also made another addition implying that the sponsor had an author on the report.
I took this all in stride, knowing from past experience that someone else might be added, until further inquiry revealed that I would not be acknowledged in the report at all. Instead, it would be attributed to two people whom I had interviewed near the start but who had done nothing else since then. Still, I did not complain. I really don’t mind when O’Reilly attributes my document to some employee of the sponsoring company, or adds their name to mine. I just like to see the money coming in. I’m doing work for hire, and part of the deal is to go along with management wishes on things such as attribution.
But I feel embarrassed on behalf of the sham authors. Did they ask to put their names on something they didn’t write? Or were they simply chess pieces in some corporate marketing plan, as I was?
I sometimes imagine one of these putative authors speaking at a conference or before a team of customers, and facing a sudden challenge from someone: “I don’t agree with this claim you made in your O’Reilly report…” Would the celebrity non-author then admit publicly that they never had anything to do with the report? Try to talk around the challenge? Meet it head-on? Beg off by saying the report was written some time ago and the non-author doesn’t remember the details?
Is it simply thievery to speak about something as if you invented the idea, when in fact you took the words from somebody else? Jewish tradition, which deals explicitly with this situation, treats such deceptive attribution as equivalent to a heinous crime. I can be more gentle and imagine the sponsor reasoning thus: “Our leading staff are experts in their field. The report we’re sponsoring is what they would have written, had they the time and writing talent to do so.” Yes, I could see some legitimacy to applying that logic to quotidian reports, following predictable paths, that technical companies routinely produce. The logic is less defensible when examining my own work, which delights in unexpected tangents, striking examples, and metaphorical comparisons.
Now my inquiry into ghostwriting enters into some thorny philosophical questions. What is more relevant: the concrete falsehood of claiming authorship over another person’s work, or the deeper truth that the person whose name on the cover is an expert in that area? Does the sponsor really extract more business out of that message, conveyed as it is through an inauthentic claim, and is the practical benefit justification for this whole intricate dance? Finally, does O’Reilly’s complicity, and my own willing participation, seem more permissible or less so, knowing that our revenue and my livelihood depend on that participation? I leave these questions on the table for future discussion.
Only in one situation did I demand recognition for my work, and it was as editor instead of writer. By the 2010 decade, I was rarely proposing books. But once I got the idea that we needed a book about security in the cloud. Although in the past we published books by some of the leading experts in computer security, our security offerings were currently were in something of a slump. Because of staff departures, no one was assigned to that crucial field for a while. So I took independent action on my idea. I found an author by talking to friends in the field, worked out the proposal, submitted it for approval, and arranged the schedule and contract. I also edited the book and dealt with the tech review. The project lay entirely between the author, tech reviewers, and me until the book went to production. This was the last project to proceed as I had operated during my golden era of the 1990s.
When production sent me their edits, someone else was listed as the developmental editor. When I pointed out the inaccuracy, I was informed of a new policy I had never heard about before: They liked to put the managing editor of each series on every book of the series, in case someone may decide to contact us and propose a new book.
I puzzled over this reason for denying me credit. I questioned: Would computer experts really think like this? “Hey, I’d like to write a book on some topic, so I’ll find a book on a similar topic from O’Reilly, turn to the copyright page everybody ignores, check the fine print to find a name, and dig up their contact information.” So I found a solution acceptable to all. I proposed that the managing editor of the series be listed along with me as developmental editors, and that compromise was accepted.
On the data engineering project, too, I decided to take action and ask for my work to be properly attributed. I did so because I suddenly realized that proper attribution would promote O’Reilly’s interests. I reached this conclusion when the project manager on the sponsor’s side mentioned, in a conversation with me and two project managers at O’Reilly, that he had read a report I wrote on streaming data eight months before. He went on to praise the report highly, talking about how I had conveyed concepts at just the right level and in the right order. I got the impression that seeing my successful work made him and his team more confident in the quality of what I was currently doing for them.
It is ironic that this manager would demonstrate the value of having my name on a good report, and then try to take my name off of the one I was doing for his company. If this were to happen often enough, potential sponsors would no longer get to see that I was the author of unusually interesting content. This would prevent me from getting more compliments, but that’s a trivial point. More importantly, it would prevent O’Reilly from using my name as an incentive to sign up sponsors. Wouldn’t it be nice if, during negotiations over a contract, an O’Reilly manager would say, “Andy Oram will be the author,” and the sponsor would say, “Ah, yes, we have seen Andy’s work and feel confident of him doing the job well”?
So I called Colleen Lobner, who managed much of the logistics for sponsored content at O’Reilly, as well as my direct supervisor, Rachel Roumeliotis. Both of them understood my points and agreed with me in principle. They promised to take my concern into account on future sponsorships. And when the data engineering report actually came out, it bore my name as sole author. I am quite proud of this contribution: the first published explanation of the data engineer’s responsibilities and the support provided to them by modern tools and processes.
I approved the copy-editor’s final version of the data engineering report the week before the layoff, so this report represents a fitting capstone to my 28-year career at O’Reilly. Managers would no longer have the opportunity to refer to that report, or any other, in order to promote me as author.
November 2020: There is already an update to the sad would-be epitaph to my O’Reilly reports in the previous paragraph. This month I was brought in as a free-lancer to do more reports for them, a new relationship with the company that I welcomed.
The data engineering report typifies the process I went through repeatedly during my final years at O’Reilly: sponsored content. This was content that looked just like everything else we wrote (a web article, a report, perhaps a whole book) but received some payment from an external company.
What did sponsorship accomplish for the sponsor? I have to admit that I never saw a contract with a sponsor. (It was suggested early on that I could help negotiate contracts, but ultimately I was never brought in until the contract was signed. This could prove awkward, as we’ll see, when it turned out that the topic or outline provided to me was inappropriate for the subject matter.) My impression was that we built a successful program around a pretty small payback. A sponsoring company might just get a chance to put their name on the cover of a report and offer it for free on their web site for a limited time—say, three or six months. They could strike gold during this time by collecting email addresses from visitors as a prerequisite to downloading the report.
The sponsor could also benefit by explaining key concepts they wanted potential customers to know in the report. These concepts might or might not pertain directly to their own services, often it consisted of more general coverage on some topic like the data engineering report I discussed before. Some companies more or less outsourced to O’Reilly the task of providing detailed instructions for tools hosted by those companies. I suppose there was also a prestige value in having their names associated with the content.
Whatever companies saw in sponsored content, it proved highly attractive. Our program grew. We hired even more people to focus on the program after Hendrickson left the company. In this chapter, we’ll look at some surprising things sponsors asked for in exchange for their sponsorship, and the impacts that had on me and on O’Reilly.
O’Reilly always included a statement about its editorial independence on sponsored content. During contract negotiation, managers explained to sponsors that we needed to make the final choices about what to say in order to maintain the readers’ trust, which would in turn benefit the sponsor who wanted people to read our work. No marketing!
The editors were pretty strict about cutting material that gave too rosy a view of sponsors’ products or services. Still, as the previous story showed, sponsors can exert a strong influence. We often changed the title or content to satisfy a sponsor. As the previous section showed, I was willing to mess with a strong structure in response to arbitrary requests. There was a general tendency to be helpful to sponsors.
I fell right in with this helpful atmosphere. But once it got me into trouble when the oxygen was suddenly removed.
One day in 2017, I heard from O’Reilly’s conference staff about a sponsor request. A few months before, I had supplied this sponsor with one of the many reports about data analysis using artificial intelligence I was being asked to produce during the late 2010 decade. The O’Reilly staffer told me the sponsor would like me to come to a Strata conference being held imminently in London, to sign printed copies of my report. Author signings draw visitors to a vendor’s booth, and many companies are willing to invest a considerable amount of money to bring an author in. Additionally, this company wanted a short posting about algorithms for their blog.
For Strata, the conference staff person explained that the sponsor was willing to pay my flight, meals, and several days at a hotel in London. Being in a hurry to reserve the flight, room, and conference pass, I polished off all these tasks after a flurry of email with the conference staff and with a marketing person from the sponsor. Finally, I let my boss Rachel Roumeliotis know that we had made these arrangements.
Reading along, have you been able to guess what I did wrong? I should have brought Roumeliotis in at the start, before reserving the flight and other resources. I had stepped on a land mine. Within hours I was on the phone not only with my Roumeliotis but with Mike Hendrickson, who ran the sponsored content program. They told me that the scandal had risen to the level of C-suite, with Laura Baldwin as irate as the rest. As you will see later in this memoir, I considered all three of these people central advocates who made my continued employment at O’Reilly possible.
Here’s what I gathered from the phone call. The team responsible for sponsored content wants to do all negotiations with every sponsor. If the sponsor wants something extra—such as my attendance at their booth, or a fresh article—they must go to Hendrickson and his team and work out O’Reilly’s share of the compensation. The sponsor had gone around Hendrickson’s team and won valuable services from me personally without paying O’Reilly (aside from my conference pass, I suppose).
I wondered why the sponsor was ignorant of this rule. Who, among the ranks of personnel who report along sometimes skewed lines of hierarchy, and who are probably too busy to check every box and scrutinize every clause, would be responsible for knowing that they might be violating their contract? The sponsor was not a huge conglomerate, but perhaps someone should have consulted a lawyer before asking me for extra work. Perhaps the lawyer would know how O’Reilly management would treat the request, and perhaps not. And perhaps the sponsor assumed that staff at O’Reilly would enforce expected behavior.
And that pondering took me back to the mode of decision-making at O’Reilly. Why were the conference staff as unaware of proper protocols as the sponsor was? And who else had been involved during my hasty negotiations? I felt shaken up enough to do a little personal backtracing, and uncovered email that cc’d a person on Hendrickson’s sponsored content team. Should this junior person have recognized that we were violating protocols? Was she responsible for keeping on top of contract details and recognizing all their implications? I didn’t blame her, because the expectation that managers were fulminating about on the phone with me was not explicit.
One of my relatives sympathetically suggested a psychoanalytical explanation to me, opining that O’Reilly managers were envious of my success and didn’t like to see my work promoted without getting some of the credit. While I wouldn’t dismiss her slant on the controversy, I can well see that O’Reilly had a legitimate beef, because they perhaps could have wormed a better deal out of the sponsor than I did. I was thrilled to get a free trip to London, and the prospect of the extra blog posting stroked my eternal pull toward self-expression. By some calculations at a high corporate level, the sponsoring company might have been getting a lot on the cheap.
By the time they talked to me, O’Reilly management had already settled on what they felt was an appropriate response to the lack of communication of which I and the sponsor were guilty. They would pay nothing toward my trip to London, and would require me to call it vacation time. Furthermore, they would do nothing to promote my appearance there or the article I would provide to the sponsor’s blog. And they followed through: No O’Reilly outlet put out so much as a tweet on social media about these things.
As for me, I proceeded along my merry way. I looked forward to publishing an article on a topic of great concern to me, to showing off my report to readers at the conference, and to getting into London for a lark. But I did feel worry and embarrassment—not for myself, but for my company. It is well and good to notify a sponsor that we expected different behavior—but what would O’Reilly gain by disregarding the sponsors’ interests? Couldn’t a sponsor be repelled by this behavior and cut back on future support? Why couldn’t O’Reilly managers show some tolerance for error, offering to join the sponsor to work on this final leg of the project? Perhaps O’Reilly could have met with the sponsor to convey their feeling that the separate agreement worked out with me was not fair to O’Reilly, and end up cutting a deal by which O’Reilly provided some support and publicity in exchange for a modest payment. But none of this was in the cards now.
From that point on, as another relative put, I made lemons into lemonade. This is how.
The blog posting was quick work. Handed the broad request to write about artificial intelligence (the topic of my original report for the sponsor), I decided to cover the pressing issue of bias, which you remember from the last chapter had been the topic of a summit I had attended a few months earlier. The sponsor was surprised that I had picked a topic with political muscle, but was so impressed by the density of references backing up my assertions, from both the popular press and academic papers, that they gladly published my posting.
The sponsor consolidated my commitments at the conference into a single evening, but still agreed to pay for several hotel nights. The hotel they chose was at the eastern edge of London near the yawning expanse of the conference space, but I reserved one extra night at a hotel in West London near Kensington Gardens, where I had stayed with Judy about a year earlier.
By a lovely bit of luck, this earlier pleasure trip we had scheduled to England coincided with a request from my manager to write an article on the tech scene in London. The purpose of commissioning the report from me was to interest Britishers and others in our London conferences, such as Strata. I seized the opportunity to schedule a meeting with some tech leaders during the vacation. I also took a number of photos that I worked into the report, along with quaint plays on words and other in-jokes that held meaning only for people who had spent some time in the capital. I don’t believe that this report had an impact on conference attendance, but it’s a nice overview that was well appreciated by the people I interviewed.
Making good use now of another trip provided by the sponsor, I resolved to be at the conference site not only the evening on which my commitments fell, but for the entire day. Thus, while having plenty of time to indulge my love of theater and museum-going, I could also derive whatever professional benefits came from attending the conference. I was genuinely curious about Strata, which I had attended just once in the U.S. I had never been at a foreign O’Reilly conference before.
Learning that London conference attendees dress more conservatively than the computer geeks who frequented U.S. conferences, I even packed my suit jacket into an old garment bag instead of bringing my usual suitcase.
The opprobrium directed at me by management on our phone call did not trickle down to other parts of O’Reilly. The conference staff was excited to hear of my presence and printed up not only the report I wrote for the sponsor, but the report on the London technology scene I had finished the previous year. I arranged to spend some time in the O’Reilly booth signing the report for attendees. I met several old colleagues from O’Reilly on my day at the conference, and with no knowledge of the controversy around my attendance, they all expressed pleasure at seeing me.
Hendrickson arrived, surprised at finding me hanging around the O’Reilly booth. He knew that O’Reilly was not paying me a cent to be there, and asked me why I bothered to spend any extra time at the conference. I explained that I was interested in the conference and felt it a great opportunity to attend.
The book signing at the sponsor’s booth went superbly. I formed bonds with staff from the sponsor and devoted time to each person who picked up a report, penning a unique dedication for each. One attendee was a data scientist for a Russian bank whose name sounded familiar to me. I was tempted to tell him, “My country just put sanctions on your company,” but instead I just wrote his name and a nice statement in the report. The marketing representative, who was a fascinating person herself—she had been in professional theater before getting into industry—offered to end the book signing early, but I was having a good time and suggested that since we had come so far, we might as well stick it out to the end of the evening. We then socialized at a reception in a nearby pub.
Having fulfilled all my responsibilities and sampled what Strata in London had to offer, I spent the next day at the National Gallery, where I satisfied a long-held yearning to see their extensive collection of Turner paintings. And I finished up the trip by having a meal with cousins. I gave them copies of the two reports from the conference, thanking them for some information they had provided to my London report.
But I was not totally finished milking my trip to London. One of the sessions I attended at Strata covered the recently passed European directive on privacy, the 2016 General Data Protection Regulation. A free software publication I write for occasionally was interested in what I learned at the session and paid me for an article I published with them upon my return home. This sum exceeded the expense of my extra hotel night and other costs of the trip that had not been covered by the sponsor.
The previous story recalled a time I angered O’Reilly management. This story will cover the best thing I ever did for O’Reilly: killing off a project.
This eccentric achievement called for a combination of my editing skills with analytics. Such combinations of skills are commonly required in a small company, where at unexpected moments one has to pick up tasks outside of one’s job description without even stopping to think about it. Those of us hired to be editors when the company was called O’Reilly & Associates always did much more than writing or editing. So it was as amateur data analyst as well as editor that I helped the company avoid a publishing scandal.
The story starts with a keynote speech at a Strata conference I didn’t attend. I’m sure that Cloudera, one of the first companies to offer machine learning in the cloud and now a major player in cloud computing, paid a good deal of sponsorship money to land a spot on the plenaries of an O’Reilly data conference. Here, on the stage before the entire assembled attendees of the conference, one of their top managers rang out his conviction that the computer field was undergoing a seismic change—with Cloudera at the center. This kind of insistence is common to many managers, which I can understand because they have thrown years of their lives into building a company and want to believe not only in their material success but in their historic importance.
Although industry trends suggested to more objective observers that machine learning in the cloud would spread gradually and in tandem with older forms of data analysis, the Cloudera manager announced us on the cusp of an era where older offerings would shrivel up and vanish. And he overstepped good sense by picking out a particular representative of the obsolete past, a company I will refer to simply as BigCompany because of their later schemes. The BigCompany’s of the world are on their way out, thundered the Cloudera manager, declaring that their offerings would soon be irrelevant.
Perhaps the Cloudera manager did not know that leading executives of BigCompany were sitting in the audience at that moment. I suspect he did, because BigCompany itself was a major sponsor of the data conference. In any case, they were understandably outraged and sought out the manager of O’Reilly’s sponsorship program to complain.
This manager, as you know now, was Mike Hendrickson. Suave, square-jawed, and athletic, with a hint of unresolved brashness, Hendrickson seemed to me superbly cut out to sell our services to large corporations—just the skill O’Reilly needed as it evolved into what the agile-tongued business analysts call a “B-to-B” strategy (for “business-to-business”). Sponsored content was part of our strategic shift away from selling books as individual units, which is “B-to-C” (“business-to-consumer”). Although Hendrickson had managed several editors, when he first came to O’Reilly, including me, he really thrived in sponsored content.
The way Hendrickson handled the BigCompany complaint demonstrates his salesmanship. He converted their anger into yet another sponsorship opportunity. “You clearly have an important message to convey about your offerings,” he told them. “Let us help you get that message out.” And they took the bait.
I had no role in drawing up the agreement between BigCompany and O’Reilly, but was brought in after they settled on a plan. BigCompany went in big. They proposed not just one, but three reports about their offerings. The first report would cover traditional tools, the second would cover the new ones that Cloudera had championed (and that BigCompany had now added to their service as well), and the third would tie the old and new together.
That was not all, however. BigCompany also designed a survey about people’s use of analytics in the cloud. This survey would be offered on their site as well as O’Reilly’s, and would ask numerous questions about the customers’ background and the background of their companies: how long their company had been in the field, its size, its geographic reach, and so on. Correlating the data would presumably turn up trends that were both statistically significant and useful for planning. BigCompany seemed confident that the trends would underscore the value of their own motley offering.
With this scaffolding in place, I was brought in to write the three reports as well as a web posting explaining the results of the survey. I examined BigCompany’s offerings as well as other available material on the three topics, and soon encountered some disconcerting limitations.
The first report concerned familiar topics in databases and data warehouses that had exhaustively covered over a 30-year period. BigCompany was essentially a cloud provider before the term “cloud” came to be applied to third-party computing services. BigCompany did not advance the field in any way. They simply relieved computer administrators of some of the fuss and trouble of running familiar tools, no doubt leading to cost savings. This was a nice, if well-trod, marketing story, but devoid of technical interest.
Turning now to the second report, I checked BigCompany’s offerings again and saw nothing to make a reader’s eyes widen. The free software community had been very fertile in the areas of big data and machine learning, producing well-known tools such as Hadoop that had taken the computing world by storm. All the cloud vendors added the tools to their offerings. Any enhancements added by the commercial vendors went back into the free software versions and were duly documented.
Some cloud vendors did create their own versions of the popular tools. Sometimes, as in the case of Google, the proprietary versions preceded the free and open source ones. I compare the relationship between free and proprietary offerings to travel in a foreign company. When you enter a grocery store in the new land, you will see popular brand names from your own country—the grocery equivalent of free software tools in the cloud. You will also see slightly different versions of the same foods under local brand names: These are like the proprietary tools. BigCompany did not even create knock-off proprietary versions, so I had nothing to say about the new analytic services they boasted about.
Thus, I persuaded O’Reilly management that the first two proposed reports would be redundant, and they brought BigCompany to this realization by promising them the single report that combined the two types of tools. Here, again, Big Company had done nothing innovative. By now, I was coming around to appreciating the accusation leveled by the Cloudera manager and suspected that BigCompany did not, in fact, have the message that Hendrickson encouraged them to offer. Still, I found a way to write up about 25 pages describing how to link together the old data warehouses with the new analytics. The report was serviceable and usable, although I did not rank it one of my important contributions to computing. O’Reilly sent it to BigCompany for their opinion.
BigCompany didn’t want to publish the report. Their managers had to sheepishly admit that what they really needed was marketing, which O’Reilly would not provide. My technical description, although accurate, was not meeting their needs. We threw my work into the virtual trash.
But one BigCompany project yet stood—do you remember it? Yes, the survey. We had promoted the questions and received a lot of answers. A data scientist on the O’Reilly staff performed some routine correlations and came up with about 35 graphs showing all kinds of relationships: the relationship between company size and success, between geographic location and the use of machine learning tools, and so on. We had a plethora of results to sift through, and there was potentially something to learn from each of the 35 slides, whether or not it showed a statistically significant relationship.
Now I was cooking. I detailed everything that might be considered counterintuitive or surprising in the graphs. Some of the questions produced no information of value, and I often wished that I had been granted a chance to review the questions before the survey started. Nevertheless, I derived many insights that I grouped into sections of an article and produced a strong draft.
But once again, this piece had to be approved by BigCompany. And once again they were disappointed. They had been hoping for results that directly justified their business model. And my summarized insights, interesting as they were, didn’t match up with the message they were telling customers. So they threw out this second article as well, and told O’Reilly they would bring their own author in to summarize the survey results and produce the article they wanted.
This is where a critical choice made by O’Reilly management determined the fate of the project—and our own reputation. They didn’t simply publish the BigCompany draft, but showed it to me first. Whether they wanted simply my editorial eye or something deeper, I don’t know. Curious to see how BigCompany could improve on what I had done, I dug in and scrutinized their article.
I had to admit that I was impressed. The author was masterful at explaining both technical details and business implications. His explanation of the results from a BigCompany point of view was persuasive, strongly stated, and eminently readable. There was only one problem: It was completely fabricated. The author blithely ignored the clear evidence turned up by the data and wrote up BigCompany’s desired outcomes, claiming support that didn’t exist from the data.
The risks posed by this piece shook me deeply. If O’Reilly had gone ahead with BigCompany’s plans and put the article on our web site, it would ruin our reputation. Someone would probably get their hands on the original data and expose the lies. Even without access to the survey data, people could report from the field that their observations contradicted ours, and get into endless incriminations that would hurt us more than it hurt BigCompany. I knew I had to squash this article.
But I also knew that O’Reilly managers wouldn’t take me seriously if I just told them I disagreed with the BigCompany author. I had to give them hard evidence—the kind of fact-based background missing so much in the fake news that became prevalent at the end of the 2010 decade.
So I went back to the 35 graphs and meticulously found a graph for each claim in the BigCompany article. Going sentence by sentence through the article, I matched each claim with a graph and explained why our data disproved the claim. I created my own five-page comparison and sent it to O’Reilly managers. They then showed it to our data scientist—the one who had done the original graphs—and luckily, he backed me up. We turned down BigCompany’s article.
At that point, I urged my managers to go live with my original article. It was accurate, I had invested a lot of time in it, and it had useful insights from which readers could benefit. But without sponsor approval, O’Reilly couldn’t do that. My article was suppressed permanently. Instead, O’Reilly told BigCompany they could do whatever they wanted with the data, but just not to associate O’Reilly with anything they published. Thus ended a moment fraught with peril.
You may be wondering how I persisted at O’Reilly during the final years when I made few acquisitions. Why wasn’t I laid off all that time? I often wondered that myself. The company was by no means averse to layoffs. As with the conference team, they could close down entire underperforming divisions and send away the staff. That happened to the video company that they acquired on their way to developing a video strategy, and to a team conducting online training.
During times of unprecedented social and business churn, I think that I actually rode the front of an evolutionary wave—the peer-to-peer trend, discussed in the next chapter—and helped O’Reilly change its character. At other times I can only say that I had a lucky clinch with changes at the company, or even that I managed just to survive change. Writing and editing sponsored content was a matter of survival for me.
Mike Hendrickson, who had been my manager for a few years, moved into the role of signing up sponsors for this program. And he tapped me as an author and editor. This gave me a new lease on life—and a kind of influence that I usually lacked as editor: a chance to publish my own ideas. I wasn’t kept busy all the time, but enough of the time to make me valuable. No longer responsible for networking with industry leaders and recommending books, I turned into a writing and editing machine. Managers would feed in project requirements and I would spit out results. When I did return to my old editorial role—signing an author or editing a book relegated to me—the project aroused nostalgia for the relationships I had formed and the sense of accomplishment I felt in those earlier years.
Standard business models that involve unit sales or subscriptions require an adept understanding of consumers and producers: our readers, our viewers, our authors. I have become comfortable over the years at balancing the views of my authors—and often maintaining balance among several authors on one text—as well as tech reviewers and sometimes the opinions of fellow O’Reilly staff. With sponsored content, the needs of the sponsor are tossed into this mix. How much harder can that be? Well, I soon found that the pressures of sponsorship brought me just as much responsibility for O’Reilly’s success and reputation as I had exercised in earlier phases of my career.
To fulfill writing requests promptly on any topic, I drew on my long experience with the history and use of computer technologies. I applied to each project the understanding I had developed through dozens of projects in programming languages, databases, security, analytics, and web development. This background permitted me to pick up every project offered me, and complete it quickly with insight. The new job left me plenty of free time to read about hot topics such as machine learning and keep pace with developments.
But my role was also restricted. I rarely got sent to the conferences I used to love, and I found less and less reason even to attend local hang-outs for techies where I used to network intensively. In my first decade at O’Reilly, the editor who worked on authors on one book would continue working with them on the next—that was part the editor-driven culture I have described earlier. But during the later years, other people in the company would routinely contact authors I had recruited and edited, then start new projects without even telling me.
Going further beneath the surface, my new role needed all the people skills and project planning savvy I had developed over the decades. These aspects of my work served both to make sponsors happy and to extract from them the information I needed.
The most amusing application of hard-earned skills earned years before came unexpectedly. A sponsor asked me to write an article on their proprietary technology, but the marketing people in charge of the project provided no technical information to start me off. They did dump piles of unrevealing marketing material on me. On projects like this, I routinely had to sift through the cruft to see whether there was anything worth saying—and rarely did I find anything. That was the case for this article as well.
Finally, after I pressed the marketing people one more time, they declared that all they could give me were two patent applications submitted by their technical staff. I don’t know why they made such an offer, the absurdity of which must have been known to them. Patent applications rank notoriously among the most obscure and unreadable documents in modern life. I think the managers must have gotten tired of the project and wanted to get rid of it. I no doubt astonished them by enthusiastically requesting the patent applications. And in fact, thanks to the skills I had gained working on Peer to Patent and other projects as a policy activist, I extracted the technical nuggets from these applications that enabled me to write a comprehensible technical article.
March 2015, according to my list of publications, saw the release of my first piece of sponsored content. I didn’t feel safe during the first few years of my transition, because nobody ever told me that I was making a transition. Managers must have clocked my output. They must have known that I was editing hardly any books and proposing even fewer. But no one said to me, “Andy, we know you can no longer perform the traditional roles of a senior editor, so we’re going to assign you new responsibilities as a producer of sponsored content.” For a year or more, I remained petrified that I’d be called on the carpet for failing at my responsibilities.
Occasionally I met my supervisor, Roumeliotis, and plied her for reassurance. I’d say openly, “I’m not recruiting many authors—is that OK?” And she always affirmed that what I was doing was what the company wanted. After a few such meetings, I finally relaxed. And the arrangement actually worked for a long time.
Backtraces: Three Decades of Computing, Communities, and Critiques by Andy Oram is licensed under CC BY-SA 4.0