Pinboard (guardiantech)

Google Flu Trends Still Appears Sick: An Evaluation of the 2013-2014 Flu Season by David Lazer, Ryan Kennedy, Gary King, Alessandro Vespignani :: SSRN

guardiantech — 2014-03-31T21:19:29+00:00

David Lazer and colleagues wrote a Science paper on Google Flu Trends's inaccuracy. Google tweaked it. They looked again:

In response to its poor performance during the 2012-2013 flu season, Google Flu Trends (GFT) engineers announced a redesign of the GFT algorithm. Two changes were made: (1) dampening anomalous media spikes and (2) using ElasticNet, rather than regression, for estimation. This paper identifies several problems that persist in the new algorithm.

Is the problem with GFT the exception or the rule in this sort of big data processing? The impossibility of getting a response is very like other services.

Data Explorer >> International Telecomms Union

guardiantech — 2014-03-30T20:17:30+00:00

Explore and visualize key ICT indicators from 1960 to 2011 for about 200 economies worldwide by creating line graphs, bar graphs, maps and bubble charts.

One for the bookmarks. (When will it get mobile internet connections?)

Headlines from a mathematically literate world >> Math with Bad Drawings

guardiantech — 2013-12-05T14:58:39+00:00

Our World: Hollywood Breaks Box Office Records with Explosions, Rising Stars
Mathematically Literate World: Hollywood Breaks Box Office Records with Inflation, Rising Population

And many more, all of similar truth. (Via @RagingTBolt)

Trends in Pricing and Duration >> The Kickstarter Blog

guardiantech — 2013-08-02T19:16:32+00:00

September 2010:

A couple of weeks ago we linked to Craig Mod’s gorgeous and thorough breakdown of his Kickstarter project Art Space Tokyo. In the post Craig sampled other popular projects on Kickstarter, and explained how their choices and outcomes affected the way he constructed his project. Craig also offers some advice on optimal pricing and project length to future project creators. We were intrigued and impressed by Craig’s research and conclusions, and we’ve now compiled data from our entire database to see how it holds up. Let’s dig in.

Fascinating analysis of how to structure a Kickstarter (or crowdfunding) project so that it hits its target. Would be great to see this updated.

Predicting Google closures >> gwern.net

guardiantech — 2013-05-06T20:34:30+00:00

Gwern Branwen:

Naturally, we are preached at by apologists that Google owes us nothing and if it’s a problem then it’s all our fault and we should’ve prophesied the future better (and too bad about all the ordinary people who may be screwed over or the unique history1 or data casually destroyed). But how can we have any sort of rational expectation if we lack any data or ideas about how long Google will run anything or why or how it chooses to do what it does? So in the following essay, I collect data on 350 Google products and look for predictive variables. I find some while modeling shutdown patterns, and make some predictions about future shutdowns.

This is a truly awesome analysis. Seasonality, the "just survive this long" period.

On Teens and iPhones >> The Axx

guardiantech — 2013-04-14T21:35:25+00:00

Alex Guyot, 17-year-old attendee at CDO High School in Tuscon, Arizonia, following on from the in-some-quarters doubted survey by Piper Jaffray about US teens' eagerness to get iPhones (saying 48% had one, 62% planned to get one as their next mobile, 23% expect to buy an Android) :

Overall, Apple has simply done an amazing job at entrenching themselves in the minds and hearts of teenagers. Despite the results of any studies or articles saying otherwise, I can easily assert that the demand for iPhones from the teenage audience is in no way waning. In fact, more teens seem to want iPhones than ever before. Certainly there is still a presence of Android owners, which while clearly not as large as the number of students with iPhones, is still a competitive force, but the loyalty of this presence to it's chosen operating system seems to be shrinking much faster than it's growing. Android isn't in any danger of extinction in this high school's population, but the dominance of the iPhone is blatantly obvious.

Email client popularity >> Campaign Monitor

guardiantech — 2013-04-09T20:41:27+00:00

Each time a subscriber opens an email sent with Campaign Monitor, we keep track of which email client they're using. Since 2009, we've measured email client popularity across many billions of emails and occasionally collate that data to show interesting trends in email client usage.Mobile email now in the lead Earlier this year, our friends at Return Path predicted that mobile was to surpass web and desktop client usage by July, 2012. We found that this event happened as early as February, when mobile overtook webmail client usage.

The mail clients being used are surprising too. (Campaign Monitor says of its clients that "Today more than 700,000 designers, agencies and amazing companies of all shapes and sizes rely on Campaign Monitor to manage their email marketing. On any given week, we're lucky enough to add more than 5,000 new customers to this list.")

Court of Appeal bans Bayesian probability (and Sherlock Holmes) >> Understanding Uncertainty

guardiantech — 2013-02-27T14:16:04+00:00

In a recent judgement the English Court of Appeal has not only rejected the Sherlock Holmes doctrine shown above, but also denied that probability can be used as an expression of uncertainty for events that have either happened or not.
The case was a civil dispute about the cause of a fire, and concerned an appeal against a decision in the High Court by Judge Edwards-Stuart. Edwards-Stuart had essentially concluded that the fire had been started by a discarded cigarette, even though this seemed an unlikely event in itself, because the other two explanations were even more implausible.

The Reverend Bayes would be spinning in his grave if he were to hear this sort of talk.

The statisticians at Fox News use classic and novel graphical techniques to lead with data >> Simply Statistics

guardiantech — 2012-12-02T21:35:04+00:00

Jeff Leek:

when flipping through the graphs I thought it was interesting to highlight some of the techniques they use to persuade. Some are clearly classics from the literature, but some are (as far as I can tell) newly developed graphical “persuasion” techniques.

Where by "persuasion" he actually means "lying" (and the headline's word "lead" should actually be "lie".)

Nate Silver and data win the election >> Lane Filler

guardiantech — 2012-11-07T13:08:25+00:00

Silver has been saying that President Barack Obama was a favorite to win this race for a quite a while, and he’s been arguing of late that Mitt Romney’s chances had evaporated to practically nothing. In return, he’s been lambasted by a cohort of conservative columnists and “statisticians” who claim the polls are all wrong (yes, all of them), and Silver is a tool of a vast left-wing media conspiracy.

Magnum of champagne for Mr Silver. The real lesson being that

hard as it might be for charlatans, fortune tellers and political hacks of either party to believe, data always wins.

Digital Landscape Research >> Cabinet Office

guardiantech — 2012-11-06T15:31:29+00:00

This independent research was conducted by 2CV to provide a comprehensive view of UK adults’ use of the internet and of government information and transactional services online.

Contains some fascinating nuggets.

Nate Silver: One-term celebrity? >> POLITICO.com

guardiantech — 2012-11-04T21:53:49+00:00

The New York Times's resident political predictor says President Barack Obama currently has a 74.6 percent chance of winning reelection. It's a prediction that liberals, whose heart rates continue to fluctuate with the release of every new poll, want to take solace in but somehow can't. Sure, this is the guy who correctly predicted the outcome of the 2008 election in 49 of 50 states, but this year's polls suggest a nailbiter.

Pick your side: Silver's analytics or poll punditry. Tuesday's the day, don't forget.

iPod history >> Benedict Evans

guardiantech — 2012-07-24T13:15:37+00:00

Shows iPods by year of release, capacity and model. A very useful way of visualising how to segment a market.

Wikipedia article traffic statistics for the top 1,000 pages

guardiantech — 2012-07-05T21:19:15+00:00

Really quite scary.

October 2009: Android to grab No. 2 spot by 2012, says Gartner >> Computerworld

guardiantech — 2012-06-20T14:46:00+00:00

Note the date, of 2009:

While the Google-backed Android mobile operating system currently runs on less than 2% of all smartphones, Gartner Inc. predicts it will surge to 14% of the global smartphone market in 2012 - ahead of the iPhone, as well as Windows Mobile and BlackBerry smartphones.

Well, sorta correct. (The forecast was made by Ken Dulaney, who's still at Gartner, though not forecasting smartphone share any more.)

Android Expected to Reach Its Peak This Year as Mobile Phone Shipments Slow >> IDC

guardiantech — 2012-06-06T16:48:16+00:00

IDC predicts Windows Phone will be the No 2 mobile operating system by 2016.

One Billion Internet Users >>Jakob Nielsen's Alertbox

guardiantech — 2012-05-23T20:49:17+00:00

Written in December 2005:

It took 36 years for the Internet to get its first billion users. The second billion will probably be added by 2015; most of these new users will be in Asia. The third billion will be harder, and might not be reached until 2040.

Nielsen returned to the post in 2012 to note that the 2bn mark was passed in early 2011, just six years after the first, and four years earlier than he had expected. Most of the growth was in Asia, which should hit a billion in 2012. But he thinks that we won't see the second doubling to 4bn before 2017. He reckons it will be 3bn by then. Make a diary date...

Platform Versions >> Android Developers

guardiantech — 2012-05-02T06:20:55+00:00

Updated for the 14 days to May 1 showing the version running on devices accessing Google Play (the Android Market as was). Points of interest: the proportion of devices running 1.5 and 1.6 has remained static at 1% cumulatively for the past two months; Gingerbread (released near the end of 2010) is the majority, at 64.6%; Ice Cream Sandwich has passed Honeycomb (4.9% v 3.3%); devices running 2.1 ("Eclair") outnumber ICS ones (5.5% v 4.9%). ICS was released over six months ago.

Google CEO Larry Page is "excited" – but how much, exactly? >> AllThingsD

guardiantech — 2012-04-16T06:05:27+00:00

Having noticed the Google CEO’s fondness for using variations of “exciting” and “excited” when discussing his company’s products and businesses, we at AllThingsD had a bit of fun with the transcripts from his recent earnings calls.
Since Page reassumed the CEO role last year, he hit an all-time high of 16 mentions of “excited” on last year’s third-quarter call. His company’s performance made him both “incredibly excited” and “amazingly excited.”

Awesome.

iOS Ebb and Flow >> pxldot

guardiantech — 2012-03-07T06:55:16+00:00

I'd like to fill in the picture I began in my recent post on Android fragmentation by examining the changes in version distribution of the other major mobile operating system: iOS. Unfortunately, this was no quite as easy as it was for Android. Unlike Google, Apple does not publish the version distribution of its user base. What we do have, however, is a number of developers who have published the version distribution within their own apps, and if we can collect a large enough sample it may be feasible to use these in lieu of direct vendor-supplied data. Using 50 data points from different developers, we can indeed build an image of version distribution over time for iOS just as we did for Android. Note that these are bundled into major releases. Grouping these into the major releases reduced noise and also matched the groupings more closely to those I used for the Android post — while it may not be a direct Apples-to-Apples comparison, it is likely the best we can do.

Beautiful, beautiful graphics (which suggest that iOS 5 reached 75% of users in less than a month - huh?) but it overlooks the fact that Google really doesn't care about fragmentation. All it wants is to have people connecting to the internet and using its search engine.

This post on Google+ statistics is a billion* times better than any other post >> VentureBeat

guardiantech — 2012-01-21T22:35:36+00:00

Remember those stats about Google+ users?

This isn’t the first time Google has tried to mislead with statistics about Google+. In July, Page claimed that the service had 10 million users who shared 1 billion items a day.

That sounds incredibly impressive. But let’s do the math. That would mean that the average user was sharing 100 items a day. Robert Scoble was flooding my feed before I blocked him, but I don’t think even he was sharing 100 items a day. (I have since unblocked him.)

So how did we get to that number? Well, it turns out Google was counting every potential recipient of that message. A single message from Scoble today would count 240,000 times toward that number. That’s preposterous.

Google is by no means alone in how it plays with numbers. This deception happens nearly every day and is especially rampant in Silicon Valley where new business models are created and standard metrics aren’t always available. It also reflects the optimistic nature of the Valley. We want to see exponential growth. We see hockey sticks everywhere. Even worse, these statistics get thrown around in the echo chamber and presented as fact. And as they get reblogged and retweeted, they lose the disclaimers that made them technically true in the first place.

How widely installed are various plugins? >> RIAStats

guardiantech — 2011-12-01T22:00:22+00:00

The contrast between the installation penetration of Flash and Silverlight is interesting.

Indices of Deprivation: Linked Data Prototype >> Open Data Blog

guardiantech — 2011-09-13T05:25:12+00:00

From the OpenViz blog: "Those of you with longer memories will recall my post back in May, promoting the then new triple store for DCLG’s Indices of Deprivation.
"I’d love to report that this has generated a flurry of new apps. I think I’m not alone when I say: we need some more real-world examples which show us the true power and potential of blending together different data sources….over the web…..using open standards."

Very impressive demonstration. The thought that occurs on seeing it is that indices don't tell you much about behaviour or other things; they don't provide any predictive power, and that's what's lacking.

How to intepret analyst reports >> Mark Suster

guardiantech — 2011-09-06T20:05:07+00:00

A useful guide, though roundabout. Anecdotal - but good.

Steam Hardware & Software Survey

guardiantech — 2011-07-29T05:54:42+00:00

"Steam conducts a monthly survey to collect data about what kinds of computer hardware and software our customers are using. Participation in the survey is optional, and anonymous. The information gathered is incredibly helpful to us as we make decisions about what kinds of technology investments to make and products to offer."

It's also really interesting to see the cross-section of Steam users.

Do Bad Guys Ever Sleep? » Trulia Insights

guardiantech — 2011-07-05T21:25:04+00:00

"After we brought crime maps into the world, we decided take a deeper look at when crime typically happens throughout the day in 25 big cities across the country. What did we find? Well, as the old adage goes, crime doesn’t pay…but it sure does look like it’s working 9 to 5 (plus a nightshift) in some cities."

Fascinating graphs.

July 2008: Global PC Installed base passes 1 Billion >> Gizmag

guardiantech — 2011-06-08T20:05:18+00:00

An interesting data point from July 2008: "Gartner’s latest figures put the number of installed PCs worldwide as just having surpassed 1 billion units and at a growth rate of just under 12 percent annually, will surpass 2 billion units by early 2014.
"So currently, it would appear, there are at least three times as many active mobile phones in the world as there are PCs, and internet access via the mobile phone is outpacing wireless access from a PC in many of areas of the world."

Mature markets (North America, western Europe, Japan) counted for 58% of installed PCs, but just 15% of the population. Gartner expected uptake to increase rapidly in emerging markets. We'll have to ask them...

Of Data Scientists, Big Data, the City and Dancers >> Rev Dan Catt's Blog

guardiantech — 2011-06-05T17:31:36+00:00

"You can’t just turn your Data Scientist eye onto something and say 'Oh we’ll throw this into MapReduce, it’ll be awesome', you need to have been part of that data, to have lived it. We don’t have Big Data where I work at the Guardian, we have lots-of-data, we look at Big Data out there and attempt to consume the signals. I came from Flickr which had fairly big fast data, the Guardian is positively quaint in comparison (in terms of what it generates). I set myself the task of getting immersed in the flow of news, trying to understand how the organization worked, the signals, the input, the output. The difference between news on a Monday to news on a Friday, the waves that Google and other sites can throw at you and so on. Living in the data, watching its rhythms, the pulse, the flow. I’m getting there, it takes a while, maybe I’m just old :)
"To deal with big data you have to have been in it, not a Scientist but as a Dancer."

Fun With Charts: Making the Rich Look Poor >> Mother Jones

guardiantech — 2011-05-13T10:23:38+00:00

A really subtle example of lying with graphs - by changing the amounts that each subdivision includes. The graph used by the Wall Street Journal uses different sizes of division for the lower-paid (sub $100,000, it's $5-10K; above, you get a $500K-$1M division).

Because it's tilted, you don't read it, except to note that the biggest bar is at the $100K-200K division. So the middle classes must have all the cash, right? Wrong.

Mac vs. PC: A Hunch Rematch >> Hunch blog

guardiantech — 2011-04-26T22:18:18+00:00

Qualities observed in people from the Hunch network. This is one that definitely wouldn't attract any comments, right?