A while back, Curt Monash (@curtmonash, blog) caught our attention by calling on tech vendors and solution providers to disclose which IT analysts’ white papers posted on their web pages, or otherwise used in marketing, are sponsored. That’s good practice, and one which most AR professionals have supported over the last decade or more.
But since AR is a two-way street, what about the reverse? Shouldn’t IT analysts (and actually, pretty much about everyone, including bloggers) disclose if a specific research area, project, note, blog post, white paper, speech, webcast, etc, is being paid for by a third party?
We popped the question on #archat and the answer was pretty much unequivocal, arguably in a self-selecting audience following @iiar (can’t blame them for this though 😉
Yes, analysts should disclose their work is sponsored, commissioned, paid for, etc.
This is the only honest way to behave, because there’s always going to be what Curt calls an attention bias.
Some, like James Governor (@monkchips) at RedMonk are going further are already disclosing all commercial relationships on all their posts (example). Of course there are objections (Gartner has relations with every firm, wouldn’t it be impractical for them to list every client) and methods of avoidance (for example, to be sponsored to investigate a topic, rather than specifically to write a particular research). Nevertheless, it’s one way forward.
What Curt does not mention is one more source of bias: the selective coverage bias, i.e. the fact that some analysts won’t cover non-client firms in the same depth as clients. And there I don’t see many showing the way apart from firms getting the bulk of their coverage from end-users: one very promising example there is the work that Carter Lusher (@carterlusher) is doing at Ovum to develop its 20th-century independence charter into something adapted to today’s market.
Of course there can be issues on the end-user analyst side of the market as well. Analyst reports are are often driven by inquiries from end-users rather than by systematic research work. Users might ask about something that is broken, or which they cannot get the system to do, or about a bad outcome, or about hype. And thus analysts who work only to write up client conversations sometimes don’t cover things that simply work, especially is the technologies have crossed the chasm.
There is no easy and complete solution to these ethical issues, although we think that when Carter’s work is completed it will produce an opportunity for a discussion across the whole industry. Below, we’ve selected come of the AR Chat’s comments on the topic. Wat’s your opinion?
@IIAR Also, analysts/vendors should not PR research without details of methodology & sample composition being accessible
@ggheorghiu@IIAR Because readers have the right to know either the report is commissioned or not (which can have an impact on the content)
Great post Duncan. The trouble with ethics is that there is an assumption we are all brave enough to adhere to a mutually agreed eithical code of conduct. Some will be prepared to and others will be too scared to.
Adding onto the pile here in full agreement that AR ethics re: white papers is just scratching the surface of AR ethics.
Don’t get me wrong; I agree that disclosures on white papers should be standard practice, and the vast majority of analyst firms in fact do disclose that they were paid a commission and by whom. But we should also remember that most of the readers of white papers were not born yesterday. IT buyers are not unsophisticated – they know immediately that “white paper” usually implies advocacy. Back in my IT days I’d read WPs because of educational values, but I understood the advocacy process of the WPs’ authorship. In terms of ethics, white papers are an AR 101 item, the ethical issues run far more deep and wide than white papers.
For example, what about pure “pay for play” analysts? There are some entire analyst sites where you only get covered if you pay (you know who you are). There are other analysts, with media outlets no less, who have “I will write [positive] pieces for a price” relationships with vendors. While they are only a few left, there are firms that run benchmarks that do not disclose funding relationships.
And since we are talking ethics let’s turn the tables onto the few “analysts” that do not accept paid relationships with vendors, who write about the vendors products or services, but never offer vendors the opportunity for fact check. Heck, I have dealt with analysts who never even asked for a briefing, who did not have access to beta product or any other products, who did not perform primary research, and had little to zero access to customers. So where is the “analysis” coming from, thin air? Ethical? Place those pieces in the Op Ed section of a newspaper, but don’t pass it off as “analysis” or “research.” Perhaps some “analyst” bloggers should have a disclaimer on their blogs and/or tweets saying something like “everything I post or tweet is based 100% on hearsay.”
And then there is that seldom-discussed dicey subject of the Gartners, IDCs and a few others having paid relationships with Wall Street. Are Gartner and IDC impacting share price therefore? Since their opinions are valued enough by Wall St. for Wall St. to pay for those opinions, and since Wall St. are the market makers, well… seems like a simple nested IF statement to me, yet no disclosure. Have I seen Wall St. reports that clearly reused IT analyst research, verbatim in fact? Absolutely, but without identifying source.
Ha, white papers are perhaps the most transparently handled ethical issue in the AR realm, ironically.
Hi Evan, I’m glad you accept the premise that ethics are wider than white papers. But I don’t agree with what you say about IT buyers: their degree of sophistication varies greatly. And because the problem is wider than just white papers, the reality is that many end users who read research might not realise the degree to which supposedly independent research is vendor-sponsored and not fundamentally independent.
While I agree with you that fact-checking should be encouraged, I don’t think advance copy is an ethical issue. I link it’s a legitimate choice for analysts to decide how to work. I also think that checking facts is separate from whether or not to give vendors advance copy. using journalistic standards, for example, you check facts by gathering multiple sources. Giving advance copy to the subject of a newspaper article is unusual. When I was an analyst I would very often give advance copy to vendors. But often that does not help: sometimes there were sharp disagreements and sometimes vendors dishonestly tried to mislead me or my colleagues. Therefore I think analysts have to fact-check without using the vendor as far as possible and focus the use of advance copy on factual clarifications.
One would also hope that research agendas and priorities at the macro level are being helped along and guided by research executives, Research Fellows and Chief Research Officer types. This could and should greatly reduce the extent of the attention bias that can creep in.
Of course, analysts often have great leeway in terms of what, who and how they cover vendors within that framework. A good executive analyst, reviewing underlings’ work, should always be questioning said analyst(s) on topics such as whether they are covering the whole market or just the key players. And if so, why or why not?
Bottom line, though, is that analysts are human. I’ve certainly seen a couple of specific cases, even at Gartner and Forrester in very recent months, of clear-cut bias and favoritism borne directly out of vendors’ dollars and time invested with analysts that had no bearing on the actual market realities.
In one situation, a blog post by a respected (senior) analyst was removed. In the other case, it is still online the last time I checked.
Cheers,
Gerry
Sorry Duncan, I think you are way, way off base on the advance copy item in certain contexts. While I am sure you are right that there are IT buyers who may get fooled now and then by a “white paper” the vast majority are that sophisticated. Still kind of moot since most analyst firms up in the disclaimer.
In terms of advance copy, however, if an analyst is delving into technology in their analysis, since the analyst has not developed the technology and is not directly implementing/using the technology in the vast majority of cases, the analysts owes the vendor ETHICALLY to at least provide an opportunity to fact check. In, fact, I would say in many cases where the technology analysis is not primary research based, the analysts ethically should try to obtain a briefing. If the vendor can’t comply, or will not respond to fact check, so be it, their funeral. But what separates, hopefully, industry analysts from two-bit opinion junkies is the analyst’s willingness to do homework.
If, however, an analyst is writing some soft opinion piece, bereft of technical facts, well, it is probably still best practice to share an advance copy with the vendor. If you don’t, well, not sure if you have crossed an ethical boundary, but you would be dancing on a fair and decent business behavior boundary.
Look, it is a free world, and anybody can say or write whatever the heck they want, that is proven every day. But since we are talking ethics in the industry analyst realm, there is hopefully a somewhat higher calling. Gartner, Forrester and IDC and any decent analyst/reserch house all do it for major reports and vendor-specific highlight pieces (e.g. “Vendor Ratings”). I find it curious that you wouldn’t come down on the side of advance copy, ethically speaking.
I side with Gerry here: getting draft review is a rule of engagement as far as I’m concerned.
Failure to do so usually results in the analyst being handed over to my PR colleagues, who have stricter access policies as a result.
Evan, I am on the side of advance copy on the basis of effectiveness. As I mention, that’s how we worked then I was an analyst. But as an *ethical* question, I don’t think it’s simple enough to say that it’s unethical to not give vendors advance copy. I could imagine, for example, a firm saying that it wanted to conduct its research only on the basis of publicly available information. I don’t think that would be the only effective way to work, but it’s a choice and perfectly moral one.
I had this comment by email from a principal analyst at a US firm:
“You are right about giving vendors advanced copy: we always do, but we seldom get back actual fact-based comments. Almost always we get attempts to change opinions and delay publication (assuming vendor doesn’t like it). To be fair this does not go for all vendors – but the majority”.
Actually, my “attention bias” comments absolutely pointed out how they could lead to a coverage bias. At least, they did somewhere in the series of posts I made on the matter. 🙂
As for writing about things one hasn’t been briefed on — well, if a company refuses a briefing, they then have no right to call you names for stating your considered opinion anyway. E.g. http://www.dbms2.com/2007/10/12/oracle-and-bea-sometimes-i-am-waaaay-early/ Beyond that, I don’t get this fact-check fetish anyway; too often, it’s just an excuse for the vendor to try to browbeat the analyst into changing or concealing an accurate opinion. This is especially true for open-source/blogging analysts; if you disagree with something, just leave a comment.