The Quintessence of Ham

Among the more eye-popping num­bers asso­ci­ated with LinkedIn’s recent ini­tial pub­lic offer­ing is the 100,000,000 mem­bers it claims. What do those hun­dred mil­lion peo­ple do with their LinkedIn accounts? If they’re like me, they qui­etly ignore the end­less spam but never quite moti­vate to unsub­scribe. Or maybe they occa­sion­ally click through a link returned by a Google search, only to dis­cover the limp résumé of some sad sack look­ing to escape the Enter­prise rent-a-car counter, not the super cool and attrac­tive “Sean Takats” that they went to high school with and are stalk­ing.

I’ve been think­ing a lot about these kinds of num­bers as the Zotero team pre­pares for a major sum­mit this sum­mer. In our first few years, we used to mea­sure Zotero’s growth in terms of down­loads, but we quit doing so well over a year ago, when that num­ber was north of four mil­lion, hav­ing dou­bled from two mil­lion just a few months ear­lier. We stopped because down­loads are never a very accu­rate mea­sure­ment of adop­tion, and they are espe­cially prob­lem­atic for Zotero, which is avail­able from a vari­ety of repos­i­to­ries. Most users get our soft­ware from either zotero.org or addons.mozilla.org, but Zotero has also popped up else­where, mainly because we don’t restrict its dis­tri­b­u­tion in any way. In the absence of any other met­ric, how­ever, down­loads are bet­ter than noth­ing, and Mende­ley for exam­ple still uses down­loads to arrive at its fig­ure of 900K+ “peo­ple,” accord­ing to Ian Mulvaney’s recent code4lib talk. And when it comes to com­mer­cial prod­ucts like End­Note, we of course have no idea at all.

A sec­ond way to mea­sure usage would be to tally user account reg­is­tra­tions. Cur­rently zotero.org hosts 620,000 accounts. Note that I say “accounts” and not “users.” Indeed there’s no rea­son to think that this fig­ure is any­thing more than very slightly more reli­able than down­loads. Zotero was around for years before we even had server accounts, and we have never aggres­sively pushed users of Zotero to reg­is­ter accounts by con­fronting them with a sign-up form before offer­ing the down­load. We think server accounts pro­vide incred­i­bly valu­able func­tion­al­ity, but we also feel that it’s a lit­tle sleezy to try to co-opt peo­ple into sign­ing up for some­thing they don’t want. So the “real” num­ber could be much higher! Among that mass of accounts, there are hun­dreds of thou­sands of real, active researchers but also, inevitably, count­less spam­mers wait­ing to be weeded and dor­mant accounts sit­ting idle. Or maybe it’s much lower! But even if we were to pre­tend that all 620,000 accounts were tended to by highly moti­vated schol­ars, we would still be faced with an order of mag­ni­tude drop when com­pared to down­loads. A quick look at Mendeley’s peo­ple direc­tory reveals a sim­i­lar dis­crep­ancy: it lists fewer than 70,000 user accounts, which is noth­ing to sneeze at but of course well south of the down­load fig­ure. How many accounts does Ref­Works have? Again, we can’t know.

A final way would be to count how many peo­ple are run­ning Zotero each day. Because Zotero auto­mat­i­cally checks for updated trans­la­tor code on a daily basis, we know that at least 275,000 instances of Zotero ran today. But wait a minute, what’s with this “instances” and “at least” busi­ness? Well, maybe some peo­ple are run­ning more than one copy of Zotero on a sin­gle machine. We could account for unique IP addresses, which moves the num­ber down slightly, but then we would ignore mul­ti­ple instances of Zotero shar­ing a sin­gle pub­lic IP address. And of course, this fig­ure only accounts for copies of Zotero that have auto­matic updates active, and that man­aged to con­nect to the inter­net. Other soft­ware ven­dors could pre­sum­ably track sync activ­ity or other met­rics to arrive at anal­o­gous figures.

The basic moral of the story, if you haven’t already guessed, is that these num­bers are all pure shit, though some are clearly worse than oth­ers. All we can do is pro­vide an hon­est expla­na­tion of how they’re derived.

Tagged soviet agricultural output, statistics, zotero

§ 10 Responses to On Usage Figures"

• adam.smith says:

  21 May 2011 at 7:43 am

  that’s quite inter­est­ing — I had no idea about these num­bers — 275k daily sync instances are quite impres­sive (and does seem like a good lower bound for # of users).
• Sean Takats says:

  23 May 2011 at 1:29 pm

  New blog post on Zotero user num­bers and what they really mean http://is.gd/x2Yjob
• zotero says:

  23 May 2011 at 3:06 pm

  New blog post on Zotero user num­bers and what they really mean http://is.gd/x2Yjob
• Karin Schöfegger says:

  23 May 2011 at 5:34 pm

  RT @stakats: On Usage Fig­ures http://t.co/QSlWK4l
• Dorothea Salo says:

  24 May 2011 at 1:47 am

  I have Zotero installed on two machines (per­sonal and work), and I’m sure I’m not the only one! This phe­nom­e­non is another that will dis­tort your count-the-translator-downloads number.
• Sean says:

  24 May 2011 at 8:27 am

  Dorothea: Of course. That’s pre­cisely why I wrote “instances.” Depend­ing on when an instance of Zotero is run­ning or not (e.g. if a com­puter is asleep), it’s also very pos­si­ble that it won’t get counted on any given day. The whole point of this post is to pro­vide greater trans­parency on Zotero’s num­bers and by exten­sion to high­light the gen­eral phe­nom­e­non of report­ing bogus usage stats.

  adam.smith: Yes, I agree that even with all the caveats, the num­bers are still rather impres­sive. That said, I think some of the best evi­dence of Zotero’s suc­cess is in the qual­ity and fre­quency of com­mu­ni­ca­tion in its user and devel­oper chan­nels. Com­pared to what one sees hap­pen­ing around other tools, it’s clear that Zotero is draw­ing on a far larger (and per­haps more moti­vated) community.
• Blog Z Francophone says:

  24 May 2011 at 9:14 am

  New blog post on Zotero user num­bers and what they really mean http://is.gd/x2Yjob
• Blog Z Francophone says:

  24 May 2011 at 9:15 am

  @stakats explique les sta­tis­tiques con­cer­nant Zotero http://is.gd/x2Yjob
• Frédéric Clavert says:

  24 May 2011 at 9:15 am

  @stakats explique les sta­tis­tiques con­cer­nant Zotero http://is.gd/x2Yjob
• greg says:

  5 June 2011 at 3:15 pm

  I know, it does say very much but still just curi­ous: what is the down­load num­bers right now if it was over 4 mil a year ago?