“How to Certify the Quality of your Software Product”
Q- Labs Company Profile
♦E stab lish ed 1989
• Conlr»uoin growth . i[» « m i6 o ria lii a ban and local pra un c«
♦ S tra te g ic Alliances & Network
• Co U a b o a te n v ,ifi waHd l«ad>ng I'vtuiina l orgarni J to ra and m rttu le *
♦ A d v is o r y b o a r d
• Pio f Rornfaoch, prel Basili Prol fa n lo n
♦W o rd in g wilh the leading organisations
• ABB, A Jcalłl, Enc*son, Bosch Smmtris.
Volvo, IBM T ilia . Toioryjr.
Main Service Areas
♦ S o ftw a re M a n a g e m e n t
♦ S o ftw a re P ro cess Im p ro ve m e n t
♦ S o ftw a re B est P ra ctices
♦ S o ftw a re A c q u is itio n Im p ro v e m e n t
♦ H u m a n Fa clo rs
♦ C e rtifica tio n o f S o ftw a re P ro d u cts
Actual Test Challenges
♦ Not te s lin g the rig h t1 thing
♦ Seem to te s t th e s a m e thing s
♦ Q ua lity g oe s d ow n in e ach p h a se
♦ C a n n o t m e a su re p ro d u ct ’re lia b ility'
♦ T o o m a n y field re p orts
♦ L ow F ield reliability
The Reality of the Sample
♦ Exhaustive testing is not practical due lo variable inputs and infinite languages
♦ A LL SOFTWARE TESTING IS SAMPLING
♦ How to choose the best sample?
The Implications of Sampling
♦ D o n 't k n o w if the s o ftw a re c o n ta in s n o fa u lts
♦ D o n 't kn o w if so ftw a re w ill n e v e r fail
♦ T w o kin d s of uses
• T ho se e ie c u ie d d o lin g Ie s1
• T h o s e le ft to? th e u s e r lo e x e c u te
♦ How to p re d ic t w h a t th e u ser sh o u ld e x p e c t to se e '’
POLSKIE T O W A R Z Y S T W O IN F O R M A T Y C Z N E — O D D Z IA Ł G Ó R N O Ś L Ą S K I
Jedenasta G órska Szkoła P T I S zc zy rk ’99 — 21-25 czerwca 1999 41
How We Normally Test
♦ VJe believe that the m e n ia l p ro cess fo r te stin g is la rg ely the sa m e for a ll p e o p le /o rg a n iza tio n s
♦ U n de rsta nd in g the Problem
♦ P lanning the Test
♦ B uilding Ihe T e st C a se s
♦ R u nn in g Tests
♦ Looking at T e st R e sults
E
Understanding the Problem
♦ R ead through e xislin g d o c u m e n la tio n
♦ \Atork throu g h e xislin g p ro to type s
♦ Ta!k w ith d om ain expe rts
♦ Build a m e n ta l m o d el o f the system
• \A h a l i t do e *
• H o w i ! m a y d o ¡1
■ H ow on e s h o u ld th in k ab o u l te s tin g il
♦ D o c u m e n l p ortion s o f the m e n ta l m o d el a s n e e d e d
-Laos
Planning the Test
♦ Define b ou nd a rie s fo r test
♦ G ather up in pu ts, etc.
♦ C re a te m e n ta l m o d e l o f system u sag e
♦ D efine w h a t w ill be tested
• N orm al e a s e s (m a g e )
• Requirements
• C ritica l s itu a tio n s
■ B o u n d a ry eases
• ‘L ik e ly to fa il' ea ses
♦ D e te rm in e h o w m u ch testin g in e a c h a re a
♦ A void looking at code
[•>grr;~3
Building the Test Cases
♦ C re a te a test case o n e at a tim e, co n s is te n t w ith m e n ta l m o d e l/d o cu m e n ta tio n
♦ R euse, as useful, from o th e r te sis
• T e st ( Iru c lu re
• Te*1 steps
• R ig h t c o m b in a tio n o f re u s e a n d n e w t e i t i
♦ C o m p u te e x p e cle d results
• P a in fu l a c tiv ity
■ L o o k i o i fa c ilita tin g th e w o ik (e a s y d a ta , etc )
Running the Test
♦ E xe cu te te sts a s a llow ed by e n v iro n m e n t
♦ D e te rm in e i f test passe d/fa ile d
♦ R e cord le s t re su lts
Looking at Test Results
♦ O rg a n ize re su lts b ase d on m e a s u re s m o rg an iza tion
• F a ilu re s p e r KL O C
• O p e n T R s v s . TolaJ TRs
♦ M a ke release d ecision based upon sch ed ule and te stin g in form atio n
P O LS KIE T O W A R Z Y S T W O IN F O R M A T Y C Z N E — O D D Z I A Ł G Ó R N O Ś L Ą S K I
4 2 C h ristoph e D ebou — ; Q -Labs — Statistical Usage Testing „How to Certify the Qualify o f you r Software Product”
Jedenasta Górska Szkoła P T I Szczyrk '99 — 21-25 czerwca 1999 4 3
[Q-Lobsl
Example Usage Model
Q-Labs
Evaluation Measures
♦ Quality
• R e lia b ility a n d C o n fid e n ce
■ M e a n T im e lo F a ilu r e
♦ S to p p a g e C riteria
■ A c lu a l S tale and A rc C ove ra ge
■ A c lu a l S u ie s an d A ic s n o t C o v e te d
• D is c rim in a n t
• D ista n ce
E E IS
Tool Support
♦ too lS E T _C erlH y - a C A S E tool to su pp ort S U T
♦ A u to m a te s a ll co m p u ta tio n a l step s - test p la n n in g /m o d e l a na lysis, te s t g en eration, p ro d u ct ce rtifica tion
♦ A u to m a tic g e n e ra tio n o f te sts - random sa m p les and c o v e ra g e test
♦ Full C A P I - A llo w s in teg ra tion o f S U T in to an a u to m a te d testin g e n v iro n m e n t
OErnrct
Project Experiences
♦ P rim arily E m b e d d e d S yste m s
♦ D o m a in s in clude T e le co m m u n ica tio n s, D e fen se.
A e ro s p s c e , M e d ical D e vices, C A S E Tools
♦ Th re e E x a m p le P ro jects
• F A A
• IB M -5 S D
Alcatel
♦ E v a lu a te fe a s lb ilily of U sage S pe cifica iio n and F u n ctio n a l Te st
♦ M o d e le d th re e IS D N sw itch fea tures
♦ R e s u lt s
• 30% m ore e ffe ctive (defects per te s t case)
• L a rg e n u m b e r o f re q u ire m e n ts is s u e s n o te d a n d c ia r ite d e a tly
• U sag « m o d e lin g moce in te re s tin g th a n c re a tin g te s t s p e c ific a tio n *
• S h ift in th o u g h t p ro ce ss for te stin g
• R e d u c e d le a d tim e due to e a rlie r d e le c t de te ction
♦ H T T P //w w w .a lc a te l d e /p ro jekte /usst
h u r r a
FAA
♦ T r a ffc S itua tio na l D isp la y
♦ N u m e ro u s and m a jo r re q u ire m e n ts is su es o b s e rve d a nd noted
♦ A t le ast 7 m a jo r m o d e ls (> 35 0 s ta te s e ach ), co m p o s e d o f at least 50 m in o r m o d e ls - u sed M o d e l C o m p osition p ro to type
♦ S tim u lu s level m odels
♦ M o d e le d full system , o rg an ize d b y fu n c tio n a l gro u p s and in cre m e n ta l d eliveries
♦ R e su lts - U nbiase d test T e st Jo u sag e. Large n u m b e rs o t failures found
P O LS KIE T O W A R Z Y S T W O IN F O R M A T Y C Z N E — O D D Z I A Ł G Ó R N O Ś L Ą S K I
4 4 C h ristoph e D ebou — Q -Labs — Statistical Usage Testing „Hm» to Certify the Quality o f you r Software Product”
Q-Labs
IBM-SSD Tucson
♦ M a ss S to ra g e S olutio ns - D evices, co n tro lle rs
♦ U sin g a pp ro ach sin ce 1990, o nly w a y th e y test
♦ M a n y p ro du cts re le ase d , u sag e and p ro to c o l m o d e ls
♦ C o rre c tn e s s a nd P erfo rm an ce
♦ High d e g re e of a u to m a tio n - full p ro c e s s + » 9 0 % o f te s ts e valu a te d a uto m a tically
♦ R e le ase c rite ria 1000 co n s e c u tiv e p a sse s in fin al ve rsio n w ith re p re se n ta tive sa m p le
♦ Ideas b e in g m o v e d a c ro ss d ivision
Applying the Techniques
♦ C o n c e p ts re a dily a p p lic a b le w ith 3 *3.5 d a y s o f tra in ing
♦ T o o l h a n d le s m a th e m a tic s
♦ O ne a p p ro a ch - m o d e l Fe atu re s
♦ C an m o d e l c o m p o n e n ts /o b je c ts
♦ C an m o d e l fro m a u s e r p erspe ctive
♦ A n y th in g th a t c a n b e b o u n d e d a n d v ie w e d a s a b la ck b o x ca n be te ste d u sin g S UT
Conclusion
♦ A ll le strn g is s a m p lin g - h o w d o w e fin d th e b est sa m p le
♦ U sa ge m o d e ls a llo w a s y s te m 'u s to fcw c h aracte rize d
♦ F o cu s is on u s a g e m o d e l d e v e lo p m e n t - Tests co m e fo r fre e
♦ S tro n g e r q u a n tita tive b a s is fo r re le a se d e c is io n
• .« i.— .
References
♦ H D Mills, M. D ye r a nd R C. L ing e r, C le a n ro o m so ftw a re e ng in ee rin g , IE E E S o ftw a re , S e p t 87
♦ J.A VWiiMaker, M . G. T h o m a s o n A M a rko v C h ain M o d e l fo r S ta tistica l U sa g e Te sting , IE E E T ra n s S oftw . E ng 2 0, 10 (O ct 1994). pp, 8 1 2 -8 2 4
♦ S W S he re r, A K ou ch akd ja n, P. G. A rn o ld , E xp e rie n ce u sing C le a nroo m S o ftw a re E ng in ee ring . IEE E S oftw a re , m a y 1996
♦ A .B e ck. H B eer, E S S I P ro cess Im p ro v e m e n t E x p e rim e n t 2 3 8 4 3 - U S S T - U s a g e S pe cifica tio n a n d S ta tistica l T esting, P ro ce e d in g s o f the E urosta r9 8, M u n ich
P O LS KIE T O W A R Z Y S T W O I N F O R M A T Y C Z N E — O D D Z I A Ł G Ó R N O Ś L Ą S K I
Jedenasta Górska Szkoła P T I Szczyrk '99 — 21-25 czerwca 1999 4 5