On my PR blog, I’ve just been ranting about poorly executed market research surveys, perhaps ill-targeted, resulting in unrepresentative samples, and maybe asking leading or ambiguous questions. Unfortunately, the outcome of some research can be news releases, reports and white papers that are tailored to support a company’s public relations rather than to genuinely inform a market. So, each time I get sent a survey, I tend to take a deep breath and regard its findings with a healthy degree of scepticism….
Take a survey undertaken last year by US-based BIM consultancy CASE with HP, for example, which generated a white paper (news release; report downloadable here – registration required). This was recently forwarded to me and, at first glance, it looks perfect for me and my interests:
Learn how 429 leading architecture, engineering, and construction firms are using this technology in a 10 page whitepaper by CASE and HP.
Topics covered include
– the rise of cloud based project management
– whether knowledge management systems are actually used
– the portion of firms using enterprise resource planning
However, upon reading the document, it became clear that some findings’ value are diminished due to methodological issues. To their credit, CASE/HP are honest about their survey approach and its limitations:
Our methodology is similar to the NBS National BIM Report and the Design Intelligence 2012 Technology & Innovation Survey …
This report is based on a survey of 429 AEC industry professionals undertaken between June 3 and June 13, 2013. The survey was conducted online and respondents were primarily recruited through invitations in the CASE email newsletter, a post on the Archdaily website, and links in social media.
Considering that respondents were recruited through digital media, it is unsurprising that they tend to self-identify as technological market leaders. This report should therefore be read with the understanding that it is not a representative survey of the entire AEC industry, but rather a glimpse at how leading firms are using technology. While the survey was answered by respondents in thirty-seven countries, the majority (69%) were from the United States of America. Despite increasing globalization in the AEC industry, there are still regional preferences towards particular software. … in general this report offers American perspective of the AEC industry.
So, in short, it was a self-selected sample, mainly US-based (hardly surprising for a New York-based consultancy), and heavily focused on CASE’s target market – architects and other designers (69% of respondents said their firms offered architecture services, 46% BIM services and 41% interior design). It could not be described as a cross-section of the AEC sector (only 20% offering construction services; no mention of specialist subcontractors or suppliers or manufacturers). If I interpret the numbers correctly, US respondents made up 296 of the sample and non-US respondents 133, from 36 other countries.
Looking at the project management findings, the survey found larger firms were more likely to use project management platforms: 54% of respondents said they did, while some smaller firms were starting to use cloud-based storage tools like Dropbox. Of the project management tools used, Newforma was the most popular – deployed by 56%, or 240 respondents, of the sample, followed by Autodesk 360 (27%), ProjectWise (18%) and Aconex (9%). Prolog (5%), GTeam (4%), and 4Projects (4%) ranked just ahead of a group of also-rans (Asite, CMiC, Conject, Enovia “below margin of error” – really, the narrow distinctions after the top three are almost meaningless: the margin of error here is +/- 5%).
The report, unsurprisingly, noted:
“There are regional biases towards particular project management software. The survey’s Australian respondents were more likely to use Aconex, while those in the United Kingdom were more likely to use Asite, Conject, and 4Projects (the sample size was not large enough in other countries to identify differences with any statistical significance).”
However, these so-called project management solutions vary widely in their capabilities. Newforma is a very different type of product to Aconex or 4Projects – it was largely an on-premise, intranet-type solution (though it is becoming more ‘hybrid’ – post) and is popular with large AEC design firms, while the latter are pure Software-as-a-Service tools aimed at supporting geographically dispersed, multi-company project-focused teams, and are more commonly deployed by clients and contractors than design consultants. Some respondents apparently also noted that “document management is not the same as project management”.
The survey also looked at future software intentions and identified growing interest in GTeam – 19% mentioned it, “which makes it the third most considered [behind Newforma and Autodesk360]”. Meanwhile, it also found:
Cloud based storage has seen new arrivals take significant marketshare from incumbents. Firms should look closely at the new range of provider options and be aware that the offerings are likely to change significantly in the coming years.
KM and ERP
If you’re interested, the knowledge management functionality was dominated by Microsoft’s Sharepoint, according to the survey’s respondents: 59% were using some form of KM solution, with Sharepoint used by 65% – even though “SharePoint isn’t targeted at the AEC industry and requires significant customization”. On ERP, used by 67% of the sample, respondents mentioned over 20 different ERP solutions, with Deltek commanding 49% share, ahead of Quickbooks on 17%.
Take care out there
Well done to CASE/HP for stressing the methodological constraints affecting their survey, and for sharing some insights into the current state of IT within a selection of largely US-based and largely design-focused firms. Maybe it gave the NYC-based firm CASE some US-oriented information it needed, but its value to some of the vendor firms discussed is limited. Newforma might be heading its project management listings but there are clearly sampling reasons why they feature prominently and other vendors do not. Take such results with a huge pinch of salt.