Wrocław, January 22, 2019 Lech Madeyski
Associate Professor
Software Engineering Department
Faculty of Computer Science and Management Wrocław University of Science and Technology Poland
B : lech.madeyski@pwr.edu.pl
↸ : http://madeyski.e-informatyka.pl
Reviewer’s opinion
on Ph.D. dissertation authored by Sylwia Kopczy´nska
entitled:
Supporting Non-functional Requirements Elicitation with Templates
1 Problem and its impact
Few books in software engineering have been as influential and timeless as The Mythical Man- month (Anniversary Ed.) by Frederick Brooks, Jr. (Brooks 1995). Hence, let me start with a quote from this famous book: “The hardest single part of building a software system is deciding precisely what to build. No other part of the conceptual work is so difficult as establishing the detailed technical requirements [...] No other part of the work so cripples the resulting system if done wrong. No other part is more difficult to rectify later.” Hence, it is no wonder that one of the major causes why do software projects fail so often is attributed to the poor understanding and definition of the software product requirements (Charette 2005; Chemuturi 2013; Chaos Report 1995).
Requirements are sometimes classified as Functional Requirements (FRs) and Non-Functional Re-
quirements (NFRs). The author of the PhD dissertation (the candidate) focuses on the latter. When
we say “non-functional” it may have a bit strange connotation that the requirements do not function
or do not serve any function. In fact such requirements may not serve business process functions
directly but they are serving a very useful purpose — they assist in better functioning of business
processes as they state conditions under which functionality is useful. For example, usability (on which the candidate focuses among others), performance or availability may not serve any business process, but if we do not manage to embed them in our software product, using the software (i.e., performing the business processes using the software) becomes tiresome at best (Chemuturi 2013).
That is why the problem how to support NFRs elicitation is of utmost importance and impact in software engineering.
The candidate in her dissertation focuses on supporting NFRs elicitation using catalog of templates which is one of the approaches to support elicitation of NFRs. Templates are natural language statements with some parameters to fill in and optional parts to select during elicitation. For vari- ous reasons, including lack of solid empirical evidence on benefits and costs of building, using and maintaining of a catalog of NFR templates, their use in software projects is not common. The candi- date decided to fill this gap and provide an empirical evidence, using case studies, experiments, as well as simulations, that would allow managers of software projects to make an informed decision whether to use or not to use a catalog of NFR templates. In my opinion, the problem is well chosen and of potentially high practical impact on how real software development is to be performed.
2 Contribution
In empirical sciences, theories are verified through observations. Until adequately confirmed by subsequent observations obtained through empirical studies, a theory (or potential law) is called a hypothesis or conjecture. Hence, I will discuss the original contributions of the candidate in terms of hypotheses and conjectures, which was popularized in software engineering by Endres and Rombach (2003) and (what I truly appreciate) reused by the candidate.
C1) Deeper understanding of (insight on) how important it is for agile practitioners to have NFRs specified in their software projects.
Using survey as a research method (in Chapter 3 of the dissertation), the candidate extended the existing research in the field, e.g.,paper by Cao and Ramesh (2008), investigating a larger set of 31 agile Requirements Engineering (RE) practices, evaluating the relative importance of the practices, as well as the relationship between their perceived importance and the experi- ence of the respondents.
Results: 47% of 118 agile practitioners considered having NFRs specified as an important aspect for software projects, while for the next 30% this was even critical.
Comment on the value of the contribution: In traditional (non-agile) approaches to software development, the importance of NFRs was unquestionable, as mentioned by the candidate on page 118 of the dissertation. Hence, the contribution in the form of an empirical evidence for the hypothesis (labeled Hypothesis 1 in the dissertation) that NFRs are important in agile projects as well, may not sound exciting (after all it is probably what most of us would ex- pect). Nevertheless, there is still value in the empirical evidence provided by the candidate.
It is worth mentioning that part of the contribution has been successfully published as a re-
search paper (Ochodek and Kopczy´nska 2018) in high impact factor international scientific
journal (Journal of Systems and Software) classified in quartile Q1 in COMPUTER SCIENCE,
SOFTWARE ENGINEERING JCR Category. Furthermore, the contribution of the candidate, as
a co-author of the aforementioned paper, is clearly described in the dissertation and in my
opinion evident. This suggests possessing skills necessary to publish research findings in a top software engineering (SE) journal.
C2) Empirical evidence of the usefulness of catalog of NFR templates.
The research method was case study with 7 software projects, and a controlled experiment with 107 subjects eliciting NFRs for a fictitious e-commerce application. Both studies are presented in Chapter 4 of the dissertation.
Results of the case study:
• The percentage of initially elicited NFRs that remained valid till the end of each project was above 80% for all but one project (labeled Hypothesis 2a in the dissertation).
• The percentage of final NFRs that were elicited at the beginning of each project was for almost all the projects above 80%,
• The percentage of initially elicited NFRs that have been derived from the templates was for almost all the projects above 60%.
Results of the experiment: Using template-supported elicitation inexperienced elicitors pro- vided NFRs that were less ambiguous, more detailed, more testable and more complete than those elicited with the ad hoc approach for both individuals and teams (Hypothesis 2b). How- ever, “using templates to support elicitation of NFRs does not speed up the elicitation process”
(Hypothesis 2d).
In both studies over 80% of subjects who used template-supported elicitation regarded NFR templates as useful. Hence, “Elicitors regard NFR templates as useful for elicitation of NFRs”
(Hypothesis 2c).
Comment on the value of the contribution: Threats to validity of the findings are discussed in detail (following the guidelines by Wohlin et al. (2012)) and addressed to enough extent.
Even taking into account possible minor issues raised in Section 3 of my review, the presented empirical evidence is sound (using both experiment and case study) and of high practical im- portance. It is worth mentioning that the contribution has been successfully published in two research papers, one in the proceedings of the International Workshop on Requirements Pat- terns (Kopczy´nska and Nawrocki 2014) and one (Kopczy´nska, Nawrocki, and Ochodek 2018) in high impact factor international scientific journal (Information and Software Technology) classified in quartile Q1 in COMPUTER SCIENCE, SOFTWARE ENGINEERING JCR Category.
Furthermore, the contribution of the candidate, as the co-author of the aforementioned pa- pers, is clearly described in the dissertation and in my opinion evident. Again, this suggests possessing skills necessary to conduct valuable research and to publish research findings in top software engineering (SE) venue.
C3) Preliminary empirical evidence of how the benefits and costs of using catalog change during catalog evolution.
Using 41 industrial projects with 2231 NFRs, the evolution of a catalog of NFR templates was
simulated in two ways: 1) the subject of the first study was a single catalog evolution driven
by the order in which the projects (i.e., requirements specifications) had been acquired. 2) in
the second study, 10000 different random permutations of the original sequence of projects
was considered to study the influence of the order of projects on the distribution of catalog
value and maintenance cost over time.
Results of the simulation studies: After “learning” templates from about 40 projects one can expect:
• at least 75% of NFRs of the next project to be derived from the templates of such catalog, and
• up to 10% of the templates may need to be modified or added, which constitutes a maintenance effort.
Such a catalog (on a basis of about 40 projects) was called mature. On a basis of the performed analyses the following hypotheses were formulated:
• Hypothesis 3a. “The greater the number of projects used to develop a catalog of NFR tem- plates, the greater the value the catalog provides [measured by the percentage of NFRs of the next project that can be derived from the templates of a catalog] 1 and the smaller the number of maintenance activities the catalog requires.”
• Hypothesis 3b. “The greater the number of projects used to develop a catalog of NFR tem- plates, the greater its size and the smaller its usage.”
Comment on the value of the contribution: The presented contribution is original and unique (none of the found studies investigated the problems concerning the maintenance process of a catalog of templates (Kopczy´nska, Nawrocki, and Ochodek 2018, s.88)), and of great practical importance to companies willing to improve the elicitation of NFRs with templates. It should be emphasized that the described contribution is based on the previously mentioned research paper (Kopczy´nska, Nawrocki, and Ochodek 2018) published in Information and Software Technology. The contribution of the candidate is clearly described and without any doubt.
C4) Deeper understanding of/insight on to what extent user feedback in app stores is a good source of NFRs and their templates.
User reviews of software applications stored in three app stores (Apple App Store, Google Play, and Amazon Appstore) were analyzed.
Results: It appeared that over 45% of the user reviews left in the app stores (and 20% of state- ments) concerned non-functional aspects and they could be used as a basis for formulating NFRs and their templates. Hence, the candidate formulated Hypothesis 4 that “App stores are a good source of NFRs and their templates”.
Comment on the value of the contribution: The presented contribution is original and one of the first, along with paper by Lu and Liang (2017), focused on on analyzing the specificities of non-functional requirements on a a basis of existing app store user reviews. Moreover, the contribution is of practical importance to companies working on subsequent releases of ap- plications available through app stores. The paper by Lu and Liang (2017) is mentioned in the paper co-authored by the candidate (Groen et al. 2017) on which the contribution in the dissertation is based upon (“we are aware of only one study that has classified statements accord- ing to quality aspects, and this research by Lu and Liang was published nearly simultaneously to this paper” (Groen et al. 2017, p.80)), but (probably by mistake) seems to be not mentioned in the dissertation. It is important that the paper on which the dissertation builds upon has been published in the proceedings of the International Requirements Engineering Conference (RE) (Groen et al. 2017, s.9) — the top RE conference ranked as “A” in the CORE Rankings of
1
"[...]" will be used to mark a note added by the author of this review.
Worldwide Journals and Conferences. The contribution of the candidate, as the co-author of the aforementioned paper, is clearly described in the dissertation and evident. What I would like to emphasize is that in this case the candidate collaborated with four researchers with German affiliation and it was not the only paper co-authored by the candidate together with foreign researchers. That bodes very well for the future.
Based on the performed studies the candidate formulated conjectures that can be treated as useful rules-of-thumb by practitioners, e.g., “Conjecture 3d. After “learning” from about 40 projects one can expect that less than 10% of the NFRs templates from their catalog would be used in the next project.”
From the presented results it follows that NFRs are important also in agile projects and a catalog of NFR templates can be useful to perform NFRs elicitation. Creating such a catalog according to the evolutionary paradigm seems to be beneficial. However, only mature catalog (of about 40 projects) can bring benefits of high coverage of requirements by the templates, and of little maintenance effort. Unfortunately, in such situation we face the problem of the low effectiveness of browsing through a mature (large) catalog which is an interesting path for future research.
3 Correctness
The scientific research steps have been performed according to the generally accepted standards in the field of empirical software engineering. In the dissertation, the candidate clearly presents related work, research methods and results then discuss results, as well as threats to validity. Dis- cussion of threats to validity in the dissertation is exemplary and increases the confidence wrt.
validity of the findings. The research methods employed are accurate to the outlined problems and the presented results were feasible to obtain. I had an opportunity to review or to be a member of grading committee of a number of habilitation and PhD dissertations in Poland and Sweden, and this one is absolutely of high quality.
That said, in empirical research there is always something to improve or just to do in an alternative way. Therefore, the following list of possible (mostly minor) issues should be treated as something we could discuss about, but not something that could undermine my very positive opinion about the presented research.
In Chapter 4 of the dissertation, both compared approaches, but especially the ad hoc approach, being the control treatment, could be better defined as otherwise it is difficult to be sure with which technique the intervention treatment method is compared and whether all of the subjects followed the same, well defined control treatment.
On page 27, the candidate has written: “rather than the physical location of the customer, it seems more important to consider the effectiveness of the communication. And thus, we defined the practice of “available/on-site customer””. I am not convinced that defining a new (somehow artificial) prac- tice was necessary as in the second edition of the famous book “Extreme Programming Explained:
Embrace Change” (not mentioned in the dissertation; it seems — on a basis of the publication year
— that in the dissertation only the first edition of this book is referenced) the “On-site customer”
practice (which the candidate decided to rename) evolved into the “Real Customer Involvement”
practice. Perhaps, it would make sense to reuse this name proposed by Beck instead of coining a new name for this agile practice (labeled P01 Available / On-site customer in Table 3.1).
On page 29, the candidate has introduced practice P31: “P31 Define a fixed iteration length. The
length of each iteration is defined, justified, and does not change (excluding external causes like na- tional holidays, etc.) [48, 112, 208].” Generally speaking, not all agile methods are necessarily time-boxed, for example Kanban.
On page 40, the agile practice labeled P15 is entitled “test-driven development (P15)”, while on other pages of the dissertation (e.g., on page 39) the practice is entitled “Prepare acceptance tests before coding”. Furthermore, strictly speaking, Behavior-Driven Development (BDD) and Acceptance Test- Driven Development (ATDD) that could be hidden behind the practice entitled “Prepare acceptance tests before coding” are not the same as Test-Driven Development (TDD). The later is focused on unit tests that drive development and do not embrace acceptance tests.
On page 41, the candidate mention using Bonferroni correction as a threat to validity of the ob- tained results (“The second threat regards using Bonferroni correction, which is known to be very conservative.”) and simply accepts the threat. However, this threat is easy to mitigate as there are more powerful alternatives to the Bonferroni correction. Why not use them? More powerful alternatives to the classic Bonferroni correction may help to reduce the risk of missing a significant result. Such alternatives are, for example, Holm’s sequential Bonferroni correction (Holm 1979), the Simes–Hochberg correction (Hochberg 1988; Simes 1986) and Hommel’s correction (Hommel 1988). All these are uniformly more powerful than the Bonferroni correction. Holm’s correction is not only uniformly more powerful than the Bonferroni correction, but also always controls the family wise error (Dmitrienko et al. 2005; Madeyski 2010).
It might be that there is something wrong with the reported number of responses to the Agile RE survey. In Section 3.1.5 it is reported that the authors “received 138 responses (95 in the first run, 43 in the second run). Two of them did not meet the validation criteria (see Section 3.1.4).” Hence, the final number of responses in the Agile RE survey was 136. This number is also reported in Section 3.1.7. However, in Section 3.2.3, it is mentioned that “There were 147 responses to the Agile RE survey (2 of them did not meet the validation criteria)”. The difference in the number of responses is nine.
On page 49, there is the following passage “although there are some practices perceived as more important, specifying NFRs is still quite high – the fourth tier.”. Are you sure that being in the fourth tier (out of six analyzed tiers), i.e., within quartile 3 (Q3), is “quite high”? The more so that at least 18 (more important practices) is in the higher tiers, while only 8 (less important practices) is in the lower tiers.
On page 64, in subsection entitled “Tool support”, there is a passage that “A demo version [of a MeetingAssistant tool] can be downloaded from the website of the project [140].” The reference [140]
is a web site http://norts.cs.put.poznan.pl/experiment where I did not find the tool, but I found a link to GitHub https://github.com/NFRsAtPUT where, in turn, I found three repositories of the candidate called “Data”, “Tools”, and “DataAnalysis”. Sharing raw data, analysis procedures and tools would be beneficial indeed for empirical software engineering research (Madeyski and Kitchenham 2017). It would also allow me to double-check the aforementioned artifacts, including also statistical analyses. However, all of the aforementioned GitHub repositories were empty (apart from the last one including README.md file containing only the title of the repository). It is worth mentioning that there is evidence that sharing data, analysis procedures (but I think tools as well) increase citations. Furthermore, some CORE A conferences, also in software engineering, accept data papers. This is something to consider in future.
In the experiment described in Section 4, it looks like there was a pseudo random assignment of
subjects to modes of elicitation (team or individual) and treatment method, but I did not find a
related discussion in Section 4.4.5 (Threats to validity).
Statistical tests performed by the candidate were followed by calculating effect size using Cliff’
and interpreting the values of this effect size measure using related thresholds, which is in line with recent recommendations related to using robust statistical methods for empirical software engineering, e.g., (Kitchenham et al. 2017).
On the other hand the post-hoc power analysis, used in the dissertation, might be questionable even though I know that it may be found in many papers. This is the reason why I absolutely do not blame the candidate. I just want to explain why (together with Prof. Barbara Kitchenham) we think it is questionable. When planning an experiment, researchers need to determine the sample size needed to ensure an adequate power level, where 0.8 is often considered to be an appropriate level.
However, during planning, the effect size is unknown, which has led to researchers mistakenly reporting the power of their experiment from the observed effect size. This is an error because:
1. Once the experiment has been performed, a post-hoc analysis of power only confirms that if we find a statistically significant effect size, the calculated power is large and if we find a non-significant effect the calculated power is small. Thus, the power analysis has told us nothing of interest about our experiment.
2. In the event that there is no significant effect, we might wrongly assume, even if the sample size is large, that this is due to the low power of the experiment. It might be the case described by the candidate on page 75 ( “The calculated post-hoc power for these tests was equal to 0.327 and 0.128. Therefore, the chance of detecting the difference (if it exists in population) was limited.”) and page 77-78 (“the post-hoc power of the tests for which the null hypotheses could not be rejected was low in most cases and ranged between 0.176 and 0.502. Therefore, the main factor influencing our chances of rejecting the null hypotheses was the limited number of teams in the experiment.”). In fact, a non-significant effect size with a large power is strong evidence in favour of the null hypothesis.
3. In the event that there is a statistically significant effect and a small sample size, post-hoc analysis will indicate a high power value, but it is far more likely that the experiment has overestimated the true effect size. Furthermore, subsequent replications using the same sam- ple size on the assumption that it is sufficiently powerful, will then be under-powered leading to further misleading results
On page 80, the candidate discusses the threat of multiple testing: “In the experiment, we run a
separate statistical inference test for each dependent variable and ISO 25010 subcharacteristic. There-
fore, a problem of multiple testing could materialize. However, even if some of the null hypotheses
were falsely rejected, we did not build our conclusions on the results of particular tests but on a general
view of how a specific approach performed for multiple different categories. Therefore, the problem of
multiple testing should not threaten the overall study outcomes.” Does it mean that the correction
for multiple testing, even the simplest Bonferroni correction, was not performed in this case? If so,
it is not clear to me why in some statistical analyses correction is performed (e.g., the Bonferroni
correction mentioned in chapter 3.1.6) but in other not?
4 Knowledge of the candidate
In my view the candidate has a very good general knowledge and understanding of the discipline with a focus on software engineering in general and requirements engineering in particular. It is reflected in a thorough literature review in the dissertation (and closely related papers), as well as the number of references in the dissertation (246). It suggests that the candidate has read a lot of papers related to the topic of research.
Design, execution and analysis of (not easy to conduct) empirical studies (including case studies and experiments) shows the spectrum of knowledge and skills of the candidate. And once again, discussion of threats to validity and using the latest statistical methods is exemplary.
5 Conclusion
Taking into account what I have presented above and the requirements imposed by Article 13 of the Act of 14 March 2003 of the Polish Parliament on the Academic Degrees and the Academic Title (with amendments) 2 , my evaluation of the dissertation according to the three basic criteria is the following:
A. Does the dissertation present an original solution to a scientific problem? (the selected option is marked with X)
X O O O O
Definitely YES Rather yes Hard to say Rather no Definitely NO
B. After reading the dissertation, would you agree that the candidate has general theoretical knowl- edge and understanding of the discipline of Computing, and particularly the area of Software Engineering?
X O O O O
Definitely YES Rather yes Hard to say Rather no Definitely NO
C. Does the dissertation support the claim that the candidate is able to conduct scientific work?
X O O O O
Definitely YES Rather yes Hard to say Rather no Definitely NO
The dissertation clearly shows that the candidate is able to conduct scientific research in software engineering. In my view the work is comparable with dissertations presented in BTH in Sweden, where I had a pleasure to be a PhD Grading Committee Member. That said it is worth mentioning that BTH is among the world’s most outstanding higher education institutions within software engineering. In systems and software engineering, BTH was ranked sixth in the world and first within the EU, according to the Journal of Systems and Software. I therefore strongly support
2
