[Apologies: this is a long-than-usual post] A week after the Construction Computing Show in London, I finally got round to flicking through a copy of Construction Computing magazine that I picked up there. It incorporates an issue of CAD User magazine wherein I saw a now familiar face: Mark Ellis, business manager of Autodesk Collaboration Services – who attended this week’s NCCTP conference (read my post), where we had a brief chat.
Mark’s contribution to the magazine is an article “A new model for managing projects online”. His starting point is the growing prevalence of design-build projects in commercial work in Europe, which places new communication demands upon team members – demands that may be satisfied by online collaboration services such as (surprise, surprise) Autodesk Buzzsaw.
Mark talks about the practical benefits of using a collaboration system (or extranet) before turning to the “critical measure of success”: its ability to generate return on investment. He says:
“A study by market research firm the Hurwitz Group revealed the average annual ROI for a sample group of Buzzsaw customers of 370 per cent, and a breakeven time line on total cost of ownership of two to three months from implementation of the service. These savings come from the elimination of in-house printing and postage or CD costs and also time wasted waiting for documents to arrive. Ultimately, they also come from avoiding potential construction errors caused by miscommunication between project partners.”
Autodesk has used the Hurwitz research, conducted in the US not Europe, in several news releases (eg here and here) since the results were first announced in 2002 (unhelpfully, the Autodesk Buzzsaw press release archive doesn’t extend back before 2003, but you can find the text of the original release here).
"The Hurwitz Group screened and interviewed 50 Autodesk Buzzsaw customers by telephone and/or via Web-based survey regarding the costs associated with Buzzsaw as well as the benefits achieved. Responding firms represented all the major segments or disciplines within the building design and construction industry, including architectural and engineering firms, contractors and owners/developers. Individual respondents included Buzzsaw end users such as project managers and CAD managers, as well as senior IT personnel such as CIOs, CTOs and IT managers."
- Average annual ROI including material and labor savings are approximately $95,000
- Savings of more than $30,000 on paper, printing, delivery and travel
- Savings of more than $65,000 on the cost of managing project collaboration
Return on Opportunity (ROO) (ROO is a proprietary methodology created by Hurwitz Group for categorizing and quantifying difficult-to-measure end-user benefits):
- 97 percent of survey respondents reported Buzzsaw enabled faster and clearer communication with partners and suppliers
- 93 percent indicated Buzzsaw improved quality of work
- 77 percent reported an increase in team productivity
Such findings are invaluable in convincing sceptics about the value of online collaboration services. However, I have a few queries about this research (maybe Mark or someone at Autodesk or Hurwitz will read this and provide some details?).
- How many different projects, and what type or scale of projects, were covered by the research? It is conceivable that some of the 50-strong sample could have been working on the same schemes, and without knowing the scale of the projects it is difficult to put that $95,000 in context; it would be a substantial saving on a $1m project, but not so significant on a $10m scheme.
- How were the tangible costs calculated? In my experience, often survey respondents have little idea how much it costs to produce, copy, package and distribute a hard-copy drawing, let alone take full account of the employee costs involved? Was this estimated on the basis of the questionnaire returns or in some other way?
- I am not clear what is covered by "the cost of managing project collaboration".
- Mark claims some of the savings came from "avoiding potential construction errors caused by miscommunication between project partners". How was this calculated? Did Autodesk and/or Hurwitz have comparable projects which used conventional information exchange methods? Or did they have evidence of the cost of errors caused by miscommunication on previous, similar projects?
I ask these questions not to be destructive (indeed, around the time the Hurwitz research was published, US writer Jerry Laiserin had already called it "limited"), but to try and move the debate forward – particularly as the NCCTP is planning some research of its own. In my book, I quote Joel Orr who three years ago pointed out several reasons why there was little research evidence of productivity improvements:
- To speak of ‘productivity increases’ requires careful measurement of productivity prior to implementation of the new technology. This is difficult to do in the project-oriented world of construction, and companies do not seem motivated to do it.
- Some issues are self-evident, and construction professionals do not want to invest time in proving them – for example, the fact that electronic transmission of documents… is much faster and cheaper, and more auditable, than using courier services….
- For many of the parties to a construction project, productivity is not a clearly defined concept. To put it bluntly, if one is being paid by the hour, reducing the number of hours required to get a job done is not an attractive proposition…. Only the owner is clearly motivated to do more with less….
- Construction projects are not highly disciplined affairs. Unless the use of a new tool can be tied to payment, subcontractors will tend to do things ‘the old familiar way,’ despite any benefits they might gain from the new tool.
In Chapter 9 of my book, I expand on this theme. Some industry professionals often insist that every construction project is unique, and that the benefits gained from one project will not necessarily be experienced on the next. In the same vein, different benefits, or different levels of benefits, might be achieved through use of different collaboration systems. Some benefits might only be achieved because the project team also had an impact. It can be difficult to determine whether benefits have been achieved either solely or partly due to the technology or whether they were due to other unrelated factor(s). Perhaps more importantly, industry businesses will usually be reluctant to admit that they can sometimes be inefficient or make mistakes, that activities were duplicated, that tasks had to be redone, etc.
Seeking to build on the somewhat anecdotal evidence advanced by most of its members (usually based on single projects), the NCCTP is currently planning its own market research project to measure the costs and benefits, advantages and disadvantages of using online collaboration services. It will not be an easy project, but if the vendors are to expand adoption of the technologies, they do need to start measuring its effects, as Laiserin summed up:
"… AEC design and delivery processes are unlikely to improve until AEC users and their technology providers insist on and actually perform rigorous metrics on statistically valid samples of project experience. To quote the classic process/change management cliche: what isn’t measured can’t be changed."
