In the last decade, new Augmented Reality tech-niques emerged to visualize and interact with physi-cal shapes and related artifact knowledge. This en-ables so-called Augmented Prototyping approaches that combine physical models with principles of projection or video mixing to establish a dynamic prototype at a relatively low cost. For example, nam and Lee documented the prototyping setup for an interactive tour guide: a physical mockup was equipped with microswitches to control a computer simulation while the output is merged either by a see-through Head Mounted Display (see Figure 1, left) or by projector (Figure 1, right). In both cases, physical cues such as proportions, key layout and grasping control can be combined with alternate screen graphics and workfl ow.
Interactive Augmented Prototyping (IAP) requires imaging technologies (output) and sensor tech-nologies (input) to establish an interactive spa-tial experience. Furthermore, AP technology em-braces existing physical prototyping methods and can include virtual prototyping/ simulation tools. In comparison to traditional physical prototyping,
this combination of the physical and digital realms offers both a highly dynamic model that can cover engineering and aesthetic aspects simultaneously. The physical interaction allows a tactile and haptic dialogue between participants and artefact model. This article draws upon research presented in my doctoral research, where I apply AR techniques to the fi eld of industrial design. It illustrates the benefi ts and systems by summarizing a number of installations found in academia. It will then elabo-rate on the investigation of what the true impact of such techniques could be in practice.
IAP as design support in
literature
Since the inception of AR, speculative design sup-port scenarios for IAP have been devised. For ex-ample, Bimber et al. (2001) predicted fi ve applica-tion scenarios without much detail: augmented de-sign review, hybrid assembling, hybrid modelling/ sketching, visual inspection of moulded parts and hybrid ergonomic design. we surveyed the existing IAP applications that cover a mixture of domains and employ various display types (head-mounted displays, video mixing or see-through and so forth). A detailed characterisation of these IAP systems can be found in Table 3, including domain, objective, and interactivity. In terms of design domains, the applications cover information appliances, automo-tive, architecture and factory planning, while some systems propose a general-purpose IAP system.
auGmeNted PrototyPING:
AUGMEnTED REALITY TO SUPPORT
THE DESIGn PROCESS
JOUKE vERLInDEn
Figure 1. Two Augmented Prototyping scenarios (nam and Lee, 2003).
Publication System function objective Interactivity ar display type Klinker et al.
(2002) ”Fata Morgana”
Presentation Presentation of concept cars
virtual object on turntable, user moves around HMD video mixing Cheok et al., (2002) Geometric modeling
Generating curves and surfaces
Index fi nger is tracked, creation of control points in air HMD video mixing Fiorentino et al.(2002) ”Spacedesign” Geometric
modeling Surface modeling
Free-form surface modeling, inclusion of physical models. HMD video mixing Bimber et al., (2001) “Augmented Engineering” Interactive
Painting Several scenarios
Supporting sketching on mirror, grasp physical objects Semi-transparent screen Bandyopadhyay et al. (2002) “Dynamic shader lamps” Interactive Painting Painting on physical objects
Moving object and paintbrush, selecting color from virtual palette
Fixed multiple projectors (projector-based AR) verlinden (2003a) Interactive Painting
Exploring component features on CnC or clay models Moving object on turntable, Change texture/paint by menu. Fixed projector
verlinden (2003b) Interactive Painting 3D sketching and RP Change texture/paint Fixed projector Rauterberg et al.
(1998) “Built-it”
Layout design Factory planning: layout check, collaborative reviews
Moving objects in 2D plane Tabletop projection Underkoffer and
Ishii, (1999) “URP”
Layout design Interactive simulation for Urban Architecture refl ections/ shadows/ wind
Moving objects in 2D plane Tabletop projection
Fründ et al. (2003) Layout design Support automotive modeling Moving components in 3D space (rotation, translation, scaling) video mixing HMD
verlinden (2004a) Layout design nightclub layout with pedestrian fl ow simulation
Small-scale physical
objects Fixed projector
verlinden (2004a) Layout design Kitchen layout, full scale Full scale cardboard cabinets and voice control
Multiple fi xed projectors nam and Lee
(2003)
Simulate Information Appliances
Usability assessment of a digital tour guide
Operating the switches while grasping the object.
video mixing HMD and fi xed projector nam (2005) Simulate Information Appliances
Dialogue defi nition and evaluation
Sketching screens and screen transitions, operating the switches while grasping the object
Embedded screen and Fixed projector Kanai (2005) Simulate Information Appliances Usability assessment of a
remote control Operating switches Fixed projector
verlinden (2004b)
Simulate Information appliances
on the components themselves, cf . Figure 6. The Built-it system supports the layout of assembly lines in a similar way; simple data are projected on top of the blocks while a large view on the wall shows the resulting manufacturing plant in a 3D perspective.
Both systems exhibit the potential of using physi-cal design components as user interfaces – the parts are managed by direct manipulation and a design can be reconfi gured by multiple hands/ users simultaneously. Furthermore, the light re-fl ection and other simulation modules’ support in combination with this tangible interface show the combination of physical spatial reasoning and computational simulation. In contrast, an aug-mented modelling system developed at a German car manufacturer deployed virtual components on a physical global shape (Fründ et al., 2003). Modelling operations were limited to component placement (translation, orientation, scaling), while a Pinch Glove supported the dialogue.
Simulating information
appliances
Examples of this type of design support are pri-marily proposed for the design of information appliances, referring to consumer products that include electronics, e.g., mobile phones, MP3 players, etc. In this case, augmentation is used to overlay graphics or other types of visual feedback to a physical mode and simulate navigational behaviour. Much emphasis is put on measuring button interaction to assess the usability of a
de-Interactive painting
Interactive painting systems such as dynamic shader lamps indicate the advantages of digital drawing on physical objects (Bandyopadhyay et al., 2001). Based on Raskar’s shader lamps tech-nique, a white object is illuminated by a collec-tion of video projectors from different angles. A tracked wand acts as a drawing tool. when in contact with the object’s surface, strokes are captured and rendered in an airbrush effect. As it copies natural drawing on objects, this es-tablishes an easy-to-use interface that has been positively evaluated by kids and graphic artists. A restriction of such interactive painting systems is that the shape of the physical object cannot dif-fer from that of the virtual object — the haptic and visual display of the virtual object will then be misaligned with the physical object.
More intricate progress in software techniques al-lows sketching on arbitrary surfaces from various distances (Cao et al., 2006). Specifi c applications have been developed for customizing ceramic plates, cf Figure 5.
Five different scenarios were identifi ed by in-specting the design activities: presentation, geometric modeling, interactive painting, layout design and simulating information appliances. These are discussed in the following subsections.
Presentation
The main takeaway of most AR systems is the fact that product visualizations can be shared with other stakeholders in the design process (e.g. higher management or potential users). Such pre-sentation systems have been specifi cally devised in automotive design. For example, Klinker et al. (2002) investigated the presentation of (virtual) concept cars in a typical showroom by observing the behavior of designers and by presenting some proof-of-concept examples. The resulting system is shown in Figure 2.
Instead of using head-mounted displays, projec-tor-based AR have also been documented that employ mockups with and projectors, for exam-ple the system shown in Figure 3. This setup is on demonstration at the High-Tech Automotive Cam-pus in Helmond, the netherlands and comprises of a full-scale model of a racing car and 3 video projections that are carefully aligned. In both cases described above, the As such, the interac-tion is passive, merely to inspect designs that are modeled in separate applications.
Geometric modelling
Some geometric modelling tools that originated from vR were adapted to AR. For example, Cheok et al. (2002) presented a see-through AR system that tracks the index fi nger by ARToolkit (Kato and Billinghurst, 1999). The user can generate curves and surfaces that fl oat in the air. As opposed to regular vR, this enables awareness of phenome-nological space, which is relevant for most prod-uct design activities. However, the system pro-vides no tactile feedback as no physical objects are included. Furthermore, interaction is diffi cult to scale up to multiple users at a single location, as the movement envelopes of such tracking sys-tems are small. A similar system was presented by Fiorentino et al. (2002), who adapted their free-form vR modelling application to work with see-through display technologies and infrared 3D tracking. Again, interaction takes place in mid-air and although physical objects can be included, these are only used to project texture maps.Layout design
Layout design systems like URP (Underkoffl er and Ishii, 1999) and Built-it (Rauterberg et al., 1998) offer a number of fi xed physical components that can be reconfi gured on a planar surface. In URP, the components represent buildings, the augmen-tation focuses on the simulation of light refl ec-tion, shadows and wind and simulation results such as fl ow fi elds are directly projected in 2D
Figure 3. Full-scale car mockup with projection (htas, 2009).
Figure 5. blueBrush (van den Berg, 2006).
Figure 6. URP system displaying an interactive wind simulation (Underkoffer & Isshi, 1999).
Figure 4. Dynamic shader lamps system in use (Bandyopadhyay et al., 2001).
Figure 2. Example of a presentation system
sign by capturing and time-stamping click events. As presented in the introduction of this article, nam and Lee documented the design evaluation of a hand-held information appliance. They used ARToolkit to track the position and orientation of the object; the optical marker is attached to the back of the object. In this case, a video-based AR HMD and a projector-based AR are compared and evaluated; the projector-based AR display report-edly performs more accurately in the interaction tests. nam (2005) expanded this simulation to the interaction modelling of both screen and buttons for a hand-held tour guide by the use of state tran-sition graphs. Although the aspect of button inter-action is well elaborated, the modelling of shape and its features is limited (e.g., the location of the buttons). Support to track layout modifi cations is lacking and this has to be performed manually.
Kanai et al. (2007) developed a usability assess-ment tool to rapidly test interaction with mock-ups by RFID technology and video projection. The level of interactivity with the prototype is even higher when the user can model and interact with a product simulation: simple tags can be glued on foam mockups and the user dons a glove with a RF antenna.
Assessing the impact of IAP in industry
To assess the possible impact of such systems, weinterviewed 13 top design and engineering agen-cies in the netherlands (verlinden et al. 2010). we targeted senior project managers in various domains, ranging from well-known interior and furniture designers to studios in automotive and product design. with each participant, we showed a short demonstration of IAP and a 90-minute semi-structured interview. we asked their opin-ion regarding the strengths and weaknesses of
Figure 7. thermostat mockup with RFID based interaction (Kanai et al., 2007).
Characterizing the IAP support scenarios
The aforementioned design support scenarios can be characterized by three activities: browsing, inter-action with the artefact’s behaviour and alteration of the model. In the table below, a brief summary is shown.
Table 2. Characterization of IAP design support scenarios from literature.
IAP. The responses are summarized in Table 4, ordered by the number of participants that ex-pressed these. The most prominent strengths are in line with the envisioned benefi ts: allowing to explore multiple design variants and to commu-nicate with the client and other stakeholders. On the other hand, primary weaknesses signify the effort needed to realize such prototypes and the resulting quality.
Then we asked about the envisioned benefi ts of the IAP to their own design process. The results are depicted in Figure 4. The participants men-tion as most important benefi ts: Communicamen-tion
with external parties, Facilitate user testing proc-ess, facilitate concept development, improve in-sights, and reduce errors/mistakes. Concerning
the method, three main issues need addressing:
Table 3: Expressed IAP strengths and weaknesses and their occurrences.
i) does variability in a design matches the physi-cal – virtual division, ii) what is the trade-off be-tween speed versus quality, iii) how does this af-fect the emancipation of other stakeholders. The fi rst is related to product domain and activity – the fi t with some (like website or kitchen design) is less obvious. The second concern touches the need for speed versus the need for credible prototypes,
type browsing Interaction with
behavior alteration of model
Presentation
view model from all sides – either handheld or fi xed to the environment
-
-Interactive painting
view model from all sides – either handheld or fi xed to the environment -Adaptation of textures/ materials/ annotations Geometric modeling
view model from all sides – either handheld or fi xed to the environment
- Creation of new geometry (curves/surfaces)
Layout design view layout of multiple components Real-time rendering of environment (simulations) Change assembly (2D or 3D) Simulating information appliances
Manual handling (physical mockups)
Digital product
interaction Button/screen layout
Strength # weakness #
Strong idea, Fills a gap in current design process, different than present methods. 5
Might take too much effort to realize (labor
intensive, complex) . 4
Exploring/Presenting many variations rapidly
without making it costly. 4
Quality of the projection vs regular fi nished
models (e.g. wood fi nish). 3 Flexible with fabrics and lay-out, styling-tool. 3 Physical model is required, also needs updated when applied in a new situation. 3 Bring ideas rapidly to the customer . 3 Only 2,5 D (just a “skin”). 2 Might be spectacular — esp. with a handheld
version. 1 Requires a lot of knowledge/might be diffi cult. 2 Good for acquisition/marketing. 1 Tactility not the same as end version (interior). 1
Easy to use. 1 User can occlude the projection. 1
Cannot support 1:1 scale for large products/
setups. 1
Possibly wrong perception of the product by
During the presentation, audio and video feeds of all IAP systems are captured, while interaction with the pens and navigation through the pres-entation are captured as notes; input events are stored as well in the segment index, similar to the where were we System (Minneman and Har-rison,1993).
The augmented prototyping technologies allow a fusion of combine audio, video, model changes, and user annotations. This results in a large, data warehouse of multimedia indexed by semantic tags (decisions, tool usage, camera switching and the like). These sessions can be inspected later to refi ne and refl ect on the decisions and planned design activities. To our knowledge, this endeav-our is the fi rst to capture experiences by record-ing all channels of augmented reality sessions. In our current implementation, we have made a handheld system, based on a picoprojector, a we-bcam and a small UMPC tablet (Figure 10), run-ning a customized version of ARToolkit. The AR cube – the desktop version- is packed in a fl ight-case equipped with a ultra-short throw projec-tor and a specialized IR tracking system (Personal Space Technologies) and a large TabletPC running vRmeer – based on OpenSceneGraph. It can be placed on a table and be up and running in less than 10 minutes (Figure 11).
Conclusions
Although some commercial AR solutions have emerged, most inspiration can be drawn from the IAP installations created in academia. As design support tools, these systems showcase the power of tangible computing as natural and embodied interaction. By inspecting the collection three different interaction characteristics to support design emerged: browsing, interaction with the artefact’s behaviour, and alteration of the model. In assessing the impact of IAP, Dutch design stu-dios were surveyed. Their reactions are positive; Most of the proposed hardware solutions are al-ready available. However, software is regarded which differ based on customer and studio
tradi-tion. The last demarcates the role of the designer in the design process and the power that clients and other stakeholders have in the process to ex-plore alternatives that might be undesirable.
The results show that almost all design and en-gineering fi rms extensively use models during design reviews, and they consider IAP to have a potential to avoid miscommunication. However, they perceive the inclusion of such new technol-ogy as time consuming which might not be worth the risk of adopting such systems.
Augmenting the design
discourse
Product design is never a solitary process – often fellow designers are involved in projects, while the act of design requires collaboration with client, prospective users, marketers,
manufac-turers, engineers and other experts. It remains diffi cult to bridge the differences in knowledge, skills and attitudes among the stakeholders. when considering decision making during design process, physical models can also be regarded as “boundary objects” - as interfaces between the stakeholders in transition between design phases (Smulders’ et al., 2008). These boundary objects encompass both product specifi cations and argu-mentation, providing a platform to create shared insight and to freeze the status of a product de-sign for later use. The authors argue that mis-communication and misinterpretation are often related to the characteristics of the used bound-ary objects, which should bridge the interfaces between design phases and the related discourse domains. There is some evidence that the inclu-sion of prototypes enhances the performance of collaborative engineering teams (Yang, 2004)(Mc-Garry, 2005). virtual prototypes such as 3D ren-derings and virtual reality models do provide a good insight; yet have challenges in providing a proper perception of context, scale and propor-tions (Kuutti et al., 2001). Because the end result is typically a physical object, a materialization of the idea is better approachable than technical drawings or specifi cations – and often more eco-nomical to fabricate.
Based on the functions that other IAP systems portray, we hypothesized that a key element of augmentation is not just to enrich a physical mockup with additional product information but to add support for design reviews. Design reviews represent formal discussions between the stake-holders of a design process, and are key in deci-sion making during the design process (Huet et al., 2007). Our initial concept of the IAP Design Review system was been devised to support syn-chronous, co-located meetings that typically do not employ advanced recording techniques. The interaction concept is shown in Figure 9: allow-ing to host presentations and discussions on de-sign alternatives while using handheld and large projector-based AR systems to add information to the models as described in the previous sections.
as the missing link, while the proper confi gura-tion of the overall solugura-tion concept is diffi cult to grasp at this moment. Based on this feedback and other empirical studies, a design review concept was developed, incorporating two different pro-jector-based AR systems: handheld and a larger desktop model. Pilot studies and additional dem-onstrations show promising results.
Figure 8: Envisioned benefi ts of the I/O Pad (n=13).
Figure 9. Interaction concept of IAP as a design review system.
Figure 10. working prototype of our handheld IAP system
Figure 11. AR Cube in use with several 3D printed objects.
REFEREnCES
■ Bandyopadhyay, D., Raskar, R. and Fuchs, H. (2001) “Dynamic shader lamps: painting on movable objects “, proceedings International Symposium on Augmented Reality (ISMAR), pp.207–216.
■ Bimber, O., Stork, A. and Branco, P. (2001) “Projection-based augmented engineering “, Proceedings of International Conference on Human-Computer Interaction (HCI’2001), vol. 1, pp.787–791.
■ Bochenek, G.M., Ragusa, J.M. and Malone, L.C. (2001) “Integrating virtual 3-D display systems into product design reviews: some insights from empirical testing “, Int. J. Technology Management, vol. 21, nos. 3–4, pp.340–352.
■ Cao, x., Balakrishnan, R. (2006) “Interacting with dynamically defi ned information spaces using a handheld projector and a pen”, ACM UIST Symposium on User Interface Software and Technology, p.225-234.
■ Cheok, A.D., Edmund, n.w.C. and Eng, A.w. (2002) “Inexpensive non-sensor based augmented reality modeling of curves and surfaces in physical space “, proceedings ISMAR’02, pp.273–274.
■ Fiorentino, M., de Amicis, R., Monno, G. and Stork, A. (2002) “Spacedesign: a mixed reality workplace for aesthetic industrial de-sign“, Proceedings ISMAR’02, pp.86–96. ■ Fründ, J., Gausemeier, J., Matysczok, C.
and Radovski, R. (2003) “Cooperative design support within automobile advance develop-ment using augdevelop-mented reality technology “, Proceedings of CSCw in Design, pp.492–497. ■ Kanai, S., Horiuchi, S., Shiroma, Y.,
Yokoy-ama, A. and Kikuta, Y. (2007) “An integrated
environment for testing and assessing the usability of information appliances using di-gital and physical mock-ups “, Lecture notes in Computer Science, vol. 4563, pp.478–487. ■ Kato, H. and Billinghurst, M. (1999) “Marker
tracking and HMD calibration for a video-based augmented reality conferencing system “, Proceedings of International workshop on Augmented Reality (IwAR 99), pp.85–94.
■ Klinker, G., Dutoit, A.H., Bauer, M., Bayer, J., novak, v. and Matzke, D. (2002) “Fata Morgana – a presentation system for product design “, Proceedings of ISMAR ‘02, pp.76– 85.
■ Kuutti, K., Battarbee, K., Säde, S., Mat-telmäki, T., Keinonen, T., Teirikko, T. and Tornberg, A. (2001) “virtual prototypes in usability testing “, Proceedings of the 34th Annual Hawaii International Conference on System Sciences (Hicss-34), 3–6 January, vol. 5.
■ McGarry, B. (2005). Things to think with: understanding interactions with artefacts in engineering design. PhD Thesis. University of Queensland, School of Information Techno-logy and Electrical Engineering.
■ Minneman, S.L., Harrison, S.R. : where were we:Making and Using near-Synchronous, Pre-narrative video, proc. ACM Multimedia ‘93, pp. 207-214 (1993)
■ nam, T-J. (2005) “Sketch-based rapid proto-typing platform for hardware-software inte-grated interactive products “, Proceedings of CHI’05, pp.1689–1692.
■ nam, T-J. and Lee, w. (2003) “Integrating hardware and software: augmented reality
based prototyping method for digital pro-ducts “, Proceedings of CHI’03, pp.956–957. ■ Rauterberg, M., Fjeld, M., Krueger, H.,
Bich-sel, M., Leonhardt, U. and Meier, M. (1998) “BUILD-IT: a planning tool for construction and design “, video Program of CHI’98, pp.177–178.
■ Smulders, F.E., Lousberg, L., Dorst, K. (2008), “Towards different communication in collaborative design“International Journal of Managing Projects in Business, vol. 1(3), pp. 352-367.
■ Underkoffl er, J. and Ishii, H. (1999) “Urp: a luminous-tangible workbench for urban plan-ning and design “, Proceedings of CHI’99, pp.386–393.
■ van den Berg (2006) “Project Light Blue”, website http://studiolab.ide.tudelft.nl/van-denberg/lightblue.html
■ verlinden, J.C., de Smit, A., Horváth, I., Epema, E. and de Jong, M. (2003a) “Time compression characteristics of the augmen-ted prototyping pipeline “, Proceedings of Euro-uRapid’03, p.A/1.
■ verlinden, J.C., de Smit, A., Peeters, A.w.J. and van Gelderen, M.H. (2003b) “Develop-ment of a fl exible aug“Develop-mented prototyping system “, Journal of wSCG, vol. 11, no. 3, pp.496–503.
■ verlinden, J., de Smit, A. and Horváth, I. (2004a) “Case-based exploration of the augmented prototyping dialogue to support design “, Proceedings of TMCE 2004, pp.245– 254.
■ verlinden, J., van den Esker, w., wind, L. and Horváth, I. (2004b) “Qualitative
compa-rison of virtual and augmented prototyping of handheld products “, Proceedings of De-sign 2004, pp.533–538.
■ verlinden, J.C., Horvath, I., nam, T-J. (2009) “Recording augmented reality experiences to capture design reviews”, International Jour-nal on Interactive Design and Manufacturing, volume 3, number 3 / August, 2009, pp. 189-200.
■ Yang, M.Y. (2004) “An examination of pro-totyping and design outcome” proceedings DETC ’04, Paper no. DETC2004-57552.
links to youtube videos of these systems
■ Underkoffer’s URP and other Luminous Room Demos http://vimeo.com/2235474
■ Fleld et al. Built-it video vor CHI’98 at http://www.ibiblio.org/openvideo/video/ chi/chi98_03_m1.mpg
■ Bluebrush prototype van den Berg http:// www.youtube.com/watch?v=Dvvl9-C2RU0 ■ Tek-Jin nam “Sketch-based rapid
pro-totyping” http://www.youtube.com/ watch?v=wK4h9Goa7qc
■ verlinden AP videos http://youtu.be/F-3BrB-fi gEw
■ Fiorentino’s Spacedesign http://www.you-tube.com/watch?v=GOMx_sytCmU ■ Bandyopadhyay et al. Dynamic Shader
lights http://www.youtube.com/ watch?v=qfwdMZIo4Cg
