Soutenance mémoire
Measuring software functional size
PHASE 1: REFINE RESEARCH METHODOLOGY
Pilot project experiment
Purpose and objective of the experiment
Phase 1 consist of a pilot project with software engineering undergraduate students in order to refine the research methodology. The objective of the pilot project was to analyze any flaws and risks from the applied pilot project methodology to refine this research methodology for the subsequent research phases.
The requirements documents
The requirements document used for this research project came from the following source: in an undergraduate software requirements course, the professor had selected a ConOps document (referred to as the ‘Vision’ document in that course) describing portions of an autonomous submarine, known as S.O.N.I.A. (Système d’opération nautique intelligent et autonome) that performs a series of underwater tasks to accomplish missions without human intervention. This S.O.N.I.A. submarine had digital cameras, sound probes, hydrophones, and several motors allowing autonomous navigation. From that chosen ConOps document, each team of students had to write the SRS for the submarine vision editor software. Ten teams had written ten different SRS documents describing functional and non functional requirements that would be compliant to elicited needs in the ConOps. Eight teams had written use cases to describe functional behaviour of the vision editor. For the pilot project SRS documents from two teams were discarded because they contained scarce or no functional requirements.
The participants
The inspectors
The inspector-participants for the pilot project experiment were twenty undergraduate students who were attending the software requirements course. They were grouped into eight teams of two, one team of one, and one team of three students. They all had limited experienced in applying inspections or peer reviews.
The measurer
The researcher, skilled and experienced with the COSMIC method, was the sole measurer for the pilot project.
The experiment steps
Training session and training material on the CRIM inspection approach were given by the researcher to all inspector-participants. Then each team provided the SRS document they had written to another team and an identical SRS copy to the researcher. Each team applied the inspection approach on the SRS document they received from another student team, collecting defects, issues, and effort data on the inspection form. Meanwhile, the researcher applied the COSMIC method to each SRS document, identifying defects and issues, and measuring effort spent performing their measurement and defect identification tasks. Once done, a quick logging meeting (between 15 and 20 minutes) was held with every team to explain critical and minor defects, ensuring with team members that these were real defects and of the adequate type. Finally, the researcher collected and analyzed defect, size, and effort data from the participants (see Appendix IX) and presented to them a result summary.
Lessons learned from the pilot project and methodology requirements
At the end of this pilot project, lessons learned were derived from several identified weaknesses, allowing the researcher to define a research methodology that would provide greater objectivity and quality of the results: 1) The pilot project methodology had a risk of biased data due to the direct involvement of the researcher in providing the measurement data. Ideally, measurement and inspection results must be independent from the researcher to avoid any biased data to be analyzed. Furthermore, to ensure greater quality of measurement data, experts in the COSMIC method must be involved. 2) Experiment data was handwritten on the inspection form and on the SRS document. This data required to be collected in a spreadsheet for analysis. Mistakes can be made during data entry, which would corrupt data and related analysis. Therefore, any collected data that is entered in a database or spreadsheet should be verified by a different person than the researcher to ensure data integrity with data sources. 3) Eight different SRS documents were inspected and measured, all describing a software solution to the same problem domain and needs. There were only one to three inspectors per SRS document (average: two inspectors per SRS), and only one measurer per SRS document. To obtain a substantial quantity of data to analyze, a significantly large sample of SRS documents would be required, and on which the same methodology must be applied. Obtaining a large sample of SRS documents was unrealistic. Attempts to obtain SRS documents from the industry failed, mostly because the industry managers contacted were not willing to share their requirements, even if confidentiality agreements were proposed. Other attempts in finding SRS documents from academics posed similar problems: lack of SRS documents or unwillingness and shyness of their authors to make it available in a research context, once they understood that more than ten people would inspect these documents identifying a potentially large number of defects. For practical considerations, the number of SRS documents must be kept as small as possible, ideally only one, to allow gathering significant data for analysis purposes. Selecting only one SRS may affect the applicability of the proposed approach to different applications types or different means of requirement documentation. 4) With an average of two inspectors per SRS document and only one measurer, there was a risk of too many variables and a lack of generalization power. The research strategy adopted was then to limit the experiment scope to a single SRS document, but to have a number of experiment participants as large as possible to provide meaningful data analysis. 5) The Gilb & Graham inspection approach with CRIM’s adaptations was consistently applied by all participants after they had participated in a relatively short training workshop (between one and two hours). Therefore, a training workshop along with inspection training material should be given to all inspection participants to ensure repeatability of the inspection method. 6) The inspection method was applied in a ‘serial’ mode (see Section 3.5.1 for more details) for the pilot project so that participants, who all had limited experience in performing peer reviews, would learn from their peers about identifying defects. But they had a week to perform their individual inspection, which was not likely to happen in controlled environment experiments. Therefore, due to time constraints for single experiments, the ‘parallel’ inspection mode was selected for the subsequent research phases.
Overview of the experimental research framework
From the lessons learned from the pilot project, and to help structure the experimentation process, the Basili (Basili, Selby and Hutchens, 1986) research framework has been used. This research framework includes four phases:
Experiment definition, where the problem to solve and the research scope are laid out.
Experiment planning, where experimental design, criteria, and measurement are selected.
Experiment operation, where preparation, execution, and data analysis are applied.
Experiment interpretation, where interpretation is explored along with potential extrapolation of results and industry impacts.
|
Table des matières
INTRODUCTION
CHAPTER 1 FUNCTIONAL REQUIREMENTS QUALITY AND SIZING IN THE LITERATURE
1.1 Functional requirements
1.1.1 Requirements usage in the software development life-cycle
1.1.2 Elicitation mechanisms
1.1.3 Documentation standards
1.1.3.1 IEEE Std 1362-1998
1.1.3.2 IEEE Std 830-1998
1.1.4 Quality attributes
1.1.5 Software engineering best practices models
1.1.6 Practices leading to requirements quality
1.2 Ensuring quality of requirements
1.2.1 Reviews applied to requirements
1.2.2 Inspections and peer reviews approaches
1.2.2.1 The Fagan approach
1.2.2.2 The Gilb and Graham approach
1.2.2.3 The CRIM adaptations to the Gilb & Graham approach
1.2.2.4 Other peer review approaches
1.2.3 Benefits of applying peer reviews to requirements
1.2.4 Peer review measures
1.2.4.1 Base measures
1.2.4.2 Derived measures
1.3 Measuring software functional size
1.3.1 Physical software size with lines of code – SLOC
1.3.2 Functional size with function points
1.3.2.1 Albrecht FPA
1.3.2.2 IFPUG
1.3.2.3 Mark II
1.3.2.4 NESMA
1.3.2.5 FiSMA
1.3.2.6 COSMIC
1.3.3 Overview of software size history
1.3.4 Functional size measurement: ISO standards .
1.4 The COSMIC method: ISO 19761
1.4.1 Overview of the COSMIC method
1.4.2 Data movements
1.4.3 Functional processes and their triggering events
1.4.4 The measurement unit of the COSMIC method
1.4.5 Applicability and non-applicability of the COSMIC method
1.4.6 Measurers
1.5 Use of functional size measurement to improve requirements
1.5.1 Experience-based measurement results
1.5.2 Experience with use cases in addition to peer reviews
1.5.3 Relationships between quality of requirements and quality of FSM results
1.5.4 A cognitive approach for applying the COSMIC method
CHAPTER 2 RESEARCH ISSUES AND RESEARCH OBJECTIVE
2.1 List of research issues
2.2 Research motivation
2.3 Research goal and chosen approach
2.4 Research objectives
2.5 Originality of the proposed research
2.6 Overview of research methodology
CHAPTER 3 PHASE 1: REFINE RESEARCH METHODOLOGY
3.1 Pilot project experiment
3.1.1 Purpose and objective of the experiment
3.1.2 The requirements documents
3.1.3 The participants
3.1.3.1 The inspectors
3.1.3.2 The measurer
3.1.4 The experiment steps
3.1.5 Lessons learned from the pilot project and methodology requirements
3.2 Overview of the experimental research framework
3.3 Selection and preparation of experiment material
3.4 The uObserve SRS
3.4.1 Preparing the uObserve SRS for experimental usage
3.4.2 The uObserve system overview
3.4.3 The uObserve SRS structure
3.5 The inspection approach
3.5.1 Inspection modes
3.5.2 Inspectors
3.5.3 Inspection material
CHAPTER 4 PHASE 2: COMPARE COSMIC AND INSPECTIONS
4.1 Overview of Phase 2 experiments
4.2 Description of the experimental protocols
4.2.1 Objectives of the experiments
4.2.2 Purpose of experiments
4.2.3 The requirements document
4.2.4 The experiment step
4.3 Experiment participants
4.3.1 The inspectors .
4.3.2 The measurers
4.4 Data from experiments
4.4.1 Defects and issues for the whole SRS document
4.4.1.1 Inspectors defects and issues for the whole SRS document
4.4.1.2 Measurers defects and issues for the whole SRS document
4.4.1.3 Uniquely identified defects and issues
4.4.1.4 Rejected defects during logging meeting or during defect verification
4.4.2 Defects and issues related to functional requirements
4.4.2.1 Inspectors defects and issues related to FR
4.4.2.2 Measurers defects and issues related to FR
4.4.2.3 Uniquely identified defects and issues related to Functional Requirements
4.4.3 Effort data
4.4.3.1 Inspection effort
4.4.3.2 Measurement effort
4.5 Analysis of efficiency and unit cost
4.5.1 Assumptions for analysis
4.5.1.1 Assumptions related to derived measures
4.5.1.2 Assumptions related to the number of participants in an inspection team
4.5.1.3 Assumption of FSM without inspection
4.5.2 Regrouping inspectors into inspection teams for analysis
4.5.2.1 The best teams of inspectors
4.5.2.2 The median teams of inspectors
4.5.2.3 The worst teams of inspectors
4.5.2.4 Impact on efficiency of adding one inspector
4.5.2.5 Impact on unit cost of adding one inspector
4.5.3 Efficiency analysis of measurers in finding defects
4.5.3.1 Measurers as a fifth member of an inspection team
4.5.3.2 Measurers as a fourth member of an inspection team
4.5.3.3 Measurers as a third member of an inspection team
4.5.3.4 Efficiency of measurers without inspection
4.5.4 Unit cost analysis of measurers in finding defects
4.5.4.1 Measurers as a fifth member of an inspection team
4.5.4.2 Measurers as a fourth member of an inspection team
4.5.4.3 Measurers as a third member of an inspection team
4.5.4.4 Unit cost of measurers without inspection
4.5.5 Summary of efficiency and unit cost analyses
4.5.5.1 Summary of efficiency analyses
4.5.5.2 Summary of unit cost analyses
4.6 Defect analysis
4.6.1 Nature of defects found by inspectors
4.6.2 Nature of defects found by measurers
4.6.3 Level of details of identified defects
4.6.4 Identification of new defects through experimental sessions
4.6.5 Analysis of defects potentially affecting FSM results
CHAPTER 5 PHASES 3 & 4: THE INFLUENCE OF DEFECTS ON FSM
RESULTS
5.1 Functional size data from Phase 2 experiments
5.1.1 FSM data from experts
5.1.2 FSM data from measurers with limited experience
5.1.3 FSM analyses
5.1.3.1 Analysis of FSM result differences among expert measurers
5.1.3.2 FSM quality challenges for measurers with limited experience
5.2 Overview of Phase 3 and Phase 4 experiments
5.3 Phase 3: Updating the uObserve SRS using identified defects
5.4 Phase 4: Expert measurers experimental session
5.4.1 Purpose of the expert measurers experiment
5.4.2 The requirements document
5.4.3 The participants (measurers)
5.4.4 The experiment steps
5.4.5 Expert measurers experiment data
5.4.5.1 Defects, issues, and assumptions data
5.4.5.2 Functional size data
5.4.5.3 Effort data
5.4.6 Expert measurer experiment results analysis
5.5 Limited experience measurers experiment
5.5.1 Purpose and objective of the limited experience measurers experiment
5.5.2 The requirements document
5.5.3 The participants (measurers)
5.5.4 The experiment steps
5.5.5 Limited experience measurers experiment data
5.5.5.1 Defects, issues, and assumption data
5.5.5.2 Functional size data
5.5.5.3 Effort data
5.5.6 Limited experience measurers: experiment results analyses
5.5.6.1 Challenges applying the COSMIC method
5.6 The influence of defects on functional size
CHAPTER 6 DISCUSSION
6.1 Summary of data analysis
6.2 Exploring the measurer participation in an inspection team
6.2.1 The analogy of the chicken and the egg: involving measurers first or last?
6.2.2 Involving the measurer as an inspection team member
6.3 The relationship between defects and functional sizing
6.4 Practical training on the application of the COSMIC method
6.5 The cost of functional size measurement
6.6 Threats to validity
6.6.1 Evolutions of the COSMIC method and their impact on this research
6.6.2 Language barrier of the SRS
6.6.3 SRS document from only one system used in experiments
6.7 Extrapolation of research results
6.7.1 Training on inspection
6.7.2 Training of measurers
6.8 Future research avenues
CHAPTER 7 RECOMMENDATIONS AND INDUSTRY IMPACTS
7.1 Proposed improvements to the application of the COSMIC method
7.1.1 Adding a measurer’s role in peer reviews
7.2 Proposed improvement to the COSMIC measurement manual
7.2.1 Proposed improvement to the COSMIC measurement process
7.2.2 Proposed improvement to the measurement labelling rule
7.2.3 Proposed improvement to the measurement reporting rule
7.2.3.1 The list of identified defects
7.2.3.2 The list of measurement assumptions
7.2.4 Recording of defects and related measurement assumptions
7.3 Research outcomes: Publications of intermediate results and other artefacts
7.4 Industry impacts
7.4.1 Contribution to the COSMIC Guideline to ensure measurement verification
7.4.2 Measurement checklist to avoid common measurement errors
7.5 Proposed improvements related to practical training
7.5.1 Include practical exercises in the COSMIC practitioners’ training
7.5.2 Using the uObserve SRS v1.0, v2.0 or v3.0 to assess measurers’ skills
7.5.3 Using the uObserve SRS v1.0 to assess inspectors’ skills
ANNEX I LIST OF APPENDICES
LIST OF BIBLIOGRAPHICAL REFERENCES
Télécharger le rapport complet
