On my PR blog, I’ve just been ranting about poorly executed market research surveys, perhaps ill-targeted, resulting in unrepresentative samples, and maybe asking leading or ambiguous questions. Unfortunately, the outcome of some research can be news releases, reports and white papers that are tailored to support a company’s public relations rather than to genuinely inform a market. So, each time I get sent a survey, I tend to take a deep breath and regard its findings with a healthy degree of scepticism….
Take a survey undertaken last year by US-based BIM consultancy CASE with HP, for example, which generated a white paper (news release; report downloadable here – registration required). This was recently forwarded to me and, at first glance, it looks perfect for me and my interests:
Learn how 429 leading architecture, engineering, and construction firms are using this technology in a 10 page whitepaper by CASE and HP.
Topics covered include
– the rise of cloud based project management
– whether knowledge management systems are actually used
– the portion of firms using enterprise resource planning
However, upon reading the document, it became clear that some findings’ value are diminished due to methodological issues. To their credit, CASE/HP are honest about their survey approach and its limitations:
Our methodology is similar to the NBS National BIM Report and the Design Intelligence 2012 Technology & Innovation Survey …
This report is based on a survey of 429 AEC industry professionals undertaken between June 3 and June 13, 2013. The survey was conducted online and respondents were primarily recruited through invitations in the CASE email newsletter, a post on the Archdaily website, and links in social media.
Considering that respondents were recruited through digital media, it is unsurprising that they tend to self-identify as technological market leaders. This report should therefore be read with the understanding that it is not a representative survey of the entire AEC industry, but rather a glimpse at how leading firms are using technology. While the survey was answered by respondents in thirty-seven countries, the majority (69%) were from the United States of America. Despite increasing globalization in the AEC industry, there are still regional preferences towards particular software. … in general this report offers American perspective of the AEC industry.
So, in short, it was a self-selected sample, mainly US-based (hardly surprising for a New York-based consultancy), and heavily focused on CASE’s target market – architects and other designers (69% of respondents said their firms offered architecture services, 46% BIM services and 41% interior design). It could not be described as a cross-section of the AEC sector (only 20% offering construction services; no mention of specialist subcontractors or suppliers or manufacturers). If I interpret the numbers correctly, US respondents made up 296 of the sample and non-US respondents 133, from 36 other countries.
Looking at the project management findings, the survey found larger firms were more likely to use project management platforms: 54% of respondents said they did, while some smaller firms were starting to use cloud-based storage tools like Dropbox. Of the project management tools used, Newforma was the most popular – deployed by 56%, or 240 respondents, of the sample, followed by Autodesk 360 (27%), ProjectWise (18%) and Aconex (9%). Prolog (5%), GTeam (4%), and 4Projects (4%) ranked just ahead of a group of also-rans (Asite, CMiC, Conject, Enovia “below margin of error” – really, the narrow distinctions after the top three are almost meaningless: the margin of error here is +/- 5%).
The report, unsurprisingly, noted:
“There are regional biases towards particular project management software. The survey’s Australian respondents were more likely to use Aconex, while those in the United Kingdom were more likely to use Asite, Conject, and 4Projects (the sample size was not large enough in other countries to identify differences with any statistical significance).”
However, these so-called project management solutions vary widely in their capabilities. Newforma is a very different type of product to Aconex or 4Projects – it was largely an on-premise, intranet-type solution (though it is becoming more ‘hybrid’ – post) and is popular with large AEC design firms, while the latter are pure Software-as-a-Service tools aimed at supporting geographically dispersed, multi-company project-focused teams, and are more commonly deployed by clients and contractors than design consultants. Some respondents apparently also noted that “document management is not the same as project management”.
The survey also looked at future software intentions and identified growing interest in GTeam – 19% mentioned it, “which makes it the third most considered [behind Newforma and Autodesk360]”. Meanwhile, it also found:
Cloud based storage has seen new arrivals take significant marketshare from incumbents. Firms should look closely at the new range of provider options and be aware that the offerings are likely to change significantly in the coming years.
KM and ERP
If you’re interested, the knowledge management functionality was dominated by Microsoft’s Sharepoint, according to the survey’s respondents: 59% were using some form of KM solution, with Sharepoint used by 65% – even though “SharePoint isn’t targeted at the AEC industry and requires significant customization”. On ERP, used by 67% of the sample, respondents mentioned over 20 different ERP solutions, with Deltek commanding 49% share, ahead of Quickbooks on 17%.
Take care out there
Well done to CASE/HP for stressing the methodological constraints affecting their survey, and for sharing some insights into the current state of IT within a selection of largely US-based and largely design-focused firms. Maybe it gave the NYC-based firm CASE some US-oriented information it needed, but its value to some of the vendor firms discussed is limited. Newforma might be heading its project management listings but there are clearly sampling reasons why they feature prominently and other vendors do not. Take such results with a huge pinch of salt.
I’d say there are two issues at play here. One is the design of the survey and the other is the presentation of the results.
In terms of survey design, we have to accept that no survey is going to be fully representative. Even if we could survey every AEC firm in the entire world, there’d still be debates about whether that data is representative and generalisable to specific contexts. This is not to say that surveys are hopeless, and neither is it to say that we shouldn’t try to make more accurate surveys. But we have to accept that even the best designed survey will have some degree of inaccuracy.
The issue is then partly in the presentation of the results. I think our industry has a real problem with surveys being presented as completely infallible sources of truth. The reports often contain no discussion of limitations, they overstate the findings, and they tuck vague methodologies into appendices — or leave them out all together. I don’t want to name names, but many of the major industry surveys could be characterised in this way.
In this sense I think our survey, the CASE/HP survey, represents a number of best practices. It is definitely flawed for the reasons you point out, but I’m proud that you could make your case basically by quoting points already made in the whitepaper. Hopefully anyone else reading the whitepaper will read the sections you have quoted and come to realise, as you have, the limitations of what we have done.
The whitepaper might not be useful to you, but for many of our clients there was a real lack of knowledge about which of this technology was being used by leading firms. It is certainly not the final word, though it is a small step toward a better understanding.
(the author of the survey and white paper)
I applauded you/CASE for being open about the limitations of the research – as you say, there are many surveys that are presented as though their results are representative, when, at best, all they can be is anecdotal.
I clearly think the white paper was/is useful for illustration purposes (I wouldn’t have discussed its findings otherwise), and it helped me provide an indirect rebuke to other surveys that are less forthcoming about potential biases in their methodologies and presentation of results.
Keep fighting the good fight!