User Tools
Sidebar
project-wiki:design_reviews:subsystem_engineering_review
Table of Contents
Subsystem Engineering Review
Subsystem Engineering is completed when the design is approved, following the Subsystem Engineering design review. The procedures of this review are defined here.
NOTE: The product development process that is taught in Capstone applies to all
engineered products, from simple devices such as a CNC-machined top to complex
systems such as a commercial airliner. All engineered products go through an engineering
phase. For many products, the development process is facilitated by engineering individual
subsystems. For simpler products, the engineering is done on the system as a whole.
As a Capstone team responsible for customizing your product development process, you
should work with your coach to make the determination as to whether or not you should
engineer your product as subsystems. If you do not have subsystems, you should
either substitute the word "System" whenever you see the word "Subsystem" or
ignore the word "Subsystem" in the rest of this page.
No grade is associated with the Subsystem Engineering reviews, although an advisory grade will be given for the Subsystem Engineering design review. The advisory grade will help you get a sense of the quality of your artifacts and how well you are doing.
The Subsystem Engineering design review occurs in late February or early March at a date negotiated between the team and the pod instructor. The team will propose a date in the Project Milestones Table. The review will be scheduled to be consistent with the instructor and the external review coaches' schedules. The desirability and transferability of the design are evaluated by a careful review of the design artifacts submitted by the team, followed by the Subsystem Engineering design review.
Remember that if you chose not to engineer your product as a collection of subsystems, you should replace the word “Subsystem” with “System” in the discussion of this review.
At the review, the team will give a presentation no longer than five minutes that includes the Project Objective Statement, a summary of the system design, and the evaluation of the engineered subsystems. Following the presentation, the reviewers and the team will discuss the desirability and transferability of the design, with special emphasis on the measured performance demonstrated in the artifacts and the presentation. Suggestions for improvement will be considered. The objective of the review is to strengthen the design by finding and making plans to remove any weaknesses present.
Artifact Submission for Subsystem Engineering
In preparation for the Subsystem Engineering review, the team will prepare and submit a pdf file containing the artifacts listed below. These artifacts should answer the questions, “What is the complete system design, including the subsystem designs and their integration?” and, “How do you know that the subsystems independently meet the Project Success Agreement and other requirements and will be compatible with one another?”
The submitted artifacts and the review should demonstrate that the design has evolved enough to meet the requirements for the end of Subsystem Engineering. Satisfactorily addressing the questions above constitute the minimum requirements for completion of Subsystem Engineering.
While not required, it is helpful for both the team and the reviewers to include a measured performance memo at the front of the submitted artifacts. This memo should: (a) highlight measurements that show the subsystems work properly, and (b) reference artifacts that show more detail on these measurements. The presence or absence of the summary memo will not affect the review grade.
- Primary Artifacts:
- Measured performance summary for each subsystem
- Requirements matrix or Software Requirements Specification for each subsystem. Requirements matrices should include measured and/or predicted values for all performance measures
- Measured and predicted performance summary for the system
- System requirements matrix or Software Requirements Specification for the full system, with predicted values for performance measures
- Supporting Artifacts:
- System Design Package (including ,at a minimum, bill of materials or software bill of materials, system and subsystem assembly drawings, part drawings for all custom parts, and interface definitions for subsystem interfaces). Pay close attention to the details described on pages 98 -- 100 of Product Development.This package should be complete and detailed enough that an independent production system could produce your design.
- Up-to-date approved Project Success Agreement, revised as necessary
- Test results obtained using models and/or prototypes, for both the subsystems and the system. Pay special attention to Figure 1 (Figure 2.12 in the textbook) as you consider the test results needed.
- Definitions of models and/or prototypes used in testing
- Test procedures that were used to obtain the results listed above
- Computer files used in the testing work
- Other detailed artifacts that support your evaluation of the performance of the subsystems and the system
Figure 1: The stage approval process. Performance testing and artifact checking are always done as part of approval. Validation is always done at System Refinement and is often done at Concept Development (the Architecture Review). Validation is often not done at Subsystem Engineering, because the subsystem prototypes are not usually accessible for the market to evaluate.
Evaluation Rubrics
The submitted artifacts will be reviewed for how clearly and completely the answer the questions above. The system and subsystem designs and tests presented in the artifacts will also be reviewed for their quality. Although the emphasis in the review will be on Question 2, an advisory grade will be given on the system design package as well. The advisory grade will not count in your final grade.
The rubrics used in the evaluation is shown below. There are different rubrics for hardware or mixed projects and software projects. Discuss with your pod instructor which rubrics you will use.
The scoresheets used by the evaluators are available on Box for hardware or mixed projects and for software projects.
Evaluation rubric for Subsystem Engineering stage
| Element | Item(weight) | Excellent (9-10) | Good (7-8) | Poor (0-6) |
|---|---|---|---|---|
| Primary Artifacts – reviewers evaluate all | Measured performance summary for each subsystem (or for system if subsystems not used) (1.5) | Clearly shows the measured performance for the subsystems. All subsystems included. Refers to repeatable testing and test procedure artifacts to clearly show how testing was done. Honestly assesses subsystem performance. Under effective revision control. | Shows most of the measured performance for each subsystem. All important subsystems included. Refers to testing and test procedure artifacts for some testing. Assessment of subsystem performance is optimistic. Somewhat under revision control | Leaves significant portions of subsystem performance unmeasured. Some subsystems are not included. Provides little or no evidence supporting performance claims. Is not honest about real performnce of system. Is not under revision control. |
| Requirements matrix for each subsystem (or system, if subsystems not used), including measured and/or predicted values for all performance measures (0.5) | Requirements matrices for all subsystems. Subsystem requirements matrices have been extended beyond the Architecture review by the addition of predicted and measured values for performance measures. Under effective revision control. If an alternative requirement specification is used, it is updated to reflect current subsystem performance. | Requirements matrices for all subsystems. Some predicted and measured performance is reported. Somewhat under reivision control. If an alternative specification is used, some subsystem performance is shown. | Significant omissions from subsystems requirements matrices. Predicted and measured performance haphazard or missing. Not under revision control. If an alternative specification is used, subystems performance is not shown or makes little sense. | |
| Predicted ( and initial measured) performance summary for the system. Predicted performance is required; measured performance is desired but may not yet be available. (1) | System performance is indicated for all Key Success Measures and for most performance measures. Measurements and predictions have evidence in the form of artifacts. The stated performance is largely consistent with target values, but there may be some changes required during System Refinement. Effectively under revision control. | System performance is indicated for all Key Success Measures and a few other performance measures. Some evidence in the form of artifacts is provided for the stated performance. Performance is close to target values. Somewhat under revision control. | Little or no system performance is indicated. Performance is asserted; little or no evidence supports the assertions. Target values are unlikely to be met. Not under revision control. | |
| System Requirements Matrix, including predicted values for system performance measures (0.5) | Matrix adds Predicted Values and some Measured Values to the information from Concept Development. Meets all of the criteria for the requirements matrix information found in Table 5.1 or Table A.3 of the textbook. Matrix is under effective revision control. If an alternative requirement specification is used, it contains predicted system performance expectations. | Matrix is mostly well done, but falls short of the listed criteria in one or two areas. Measurements and Predictions appear superficial. Revision control is superficial. | Matrix has some serious weaknesses that raise questions about its utility. Measurements and predictions are unbelievable or missing. Not under revision control. | |
| Supporting Artifacts – reviewers spot check | System Design Package, including (at a minimum) a BOM, system and subsystem assembly drawings, part drawings, and interface definitions. (2.5) | Drawing package is complete. Bill of Materials is complete. All custom parts have drawings. All necessary logic, flow, block, wiring, and schematic diagrams are included. Drawings follow appropriate standards. Drawings have been checked by someone other than that creator. The entire system is specified. Software organization is clear and effective, and follows the guidelines in the Design Artifacts section of the Student Book. Drawing package is under effective revision control. | Drawing package is mostly complete. Bill of Materials is mostly complete. Most custom parts have drawings. Non-geometric drawings are mostly present. Appropriate standards are mostly followed. Drawing checking appears to have been somewhat superficial. Key subsystems are specified. Software organization is defined, and mostly follows the guidelines. Revision control is superficial. | Drawing package is substantially incomplete. Many necessary drawings are missing. Standards are followed poorly or ignored. Checking is minimal or nonexistent. Software organization is not defined, or does not follow the guidelines. Not under revision control. |
| Test procedures and results (0.75) | All necessary supporting artifacts on testing are included. There is nothing the reviewer felt should be included that is missing. For Subsystem Engineering, likely testing artifacts include * Test results obtained using models and/or prototypes * Test procedures used to obtain results in Primary Artifacts * Computer files used in testing work | Most necessary testing artifacts are included. One or two things the reviewer felt should be included are missing. | The testing supporting artifacts are incomplete. There are not enough supporting artifacts to support approval. | |
| Other supporting artifacts (0.25) | All other necessary supporting artifacts are included. There is nothing the reviewer felt should be included that is missing. For Subsystem Engineering, likely other artifacts include * Up-to-date signed Project Success Agreement * Definitions of models and/or prototypes * Other detailed artifacts that support your evaluation of the performance of the subsystems and system | Most necessary artifacts are included. One or two things the reviewer felt should be included are missing. | The other supporting artifacts are incomplete. There are not enough supporting artifacts to support approval. | |
| Transferability (1) | The supporting artifacts are clearly written, and allow the reader to create the subsystem, replicate a test and/or use a prototype to repeat the work. The supporting artifacts have appropriate mechanics and are properly under revision control. | The supporting artifacts are somewhat clear. They provide enough guidance to allow the reader to apply engineering judgment to create a subsystem, replicate a test and/or use a prototype to repeat the work. The supporting artifacts have acceptable mechanics; the revision control is superficial. | Supporting artifacts are unclear and/or otherwise difficult to understand. They provide insufficient information to allow the reader to create a subsystem, replicate a test, and/or use a prototype to repeat the work. The artifact mechanics are unacceptable. Revision control is missing. | |
| Quality (1) | The supporting artifacts reflect thoughtful, appropriate engineering practice. The correct activities were performed and properly reported in order to support approval. The reviewer is confident in the information provided to support approval. | The supporting artifacts reflect engineering practice. Some of the correct activities were performed and properly reported. There is some data supporting approval, but it is not thorough or complete. The reviewer may have some questions about the information provided to support approval. | The supporting artifacts reflect poor engineering practice. There is missing evidence for some expected activities. With the existing artifacts, the reviewer has insufficient information to make an informed judgment to approve stage completion. | |
| Formatting (1) | Complies with all formatting requirements in the Student Book and is carefully and thoughtfully formatted. Bookmarks and/or hyperlinks are provided to help the reader navigate the document. | Mostly complies with formatting requirements. One or two documents need to be fixed to comply with formatting requirements | Shows little or no evidence of following formatting requirements. Haphazard formatting, difficult to read | |
| Rubric revision 1.3 made from ScoreSheetMaker revision 1.5 | ||||
Evaluation rubric for Subsystem Engineering stage on software projects
| Element | Item(weight) | Excellent (9-10) | Good (7-8) | Poor (0-6) |
|---|---|---|---|---|
| Primary Artifacts – reviewers evaluate all | Performance Summary - Key Success Measures (1.5) | Clearly summarizes current performance against Key Success Measures. References test procedures and test results artifacts. Honestly assesses performance. Includes predicted results for future software revisions. Under effective revision control. | System performance is mostly addressed but leaves some questions. Refers to testing and test procedure artifacts for some testing. Lacking details to instill confidence in future performance. Somewhat under revision control. | Leaves significant portions of performance unaddressed. Some Key Success Measures are not addressed. Provides little or no evidence supporting performance claims. Is not honest about real performance of system. Is not under revision control. |
| Performance Summary - Software Requirements Specification (SRS) (1.5) | Clearly demonstrates current performance against requirements listed in SRS. References test procedures and test results artifacts. Honestly assesses performance. Under effective revision control. | Some predicted and measured performance is reported, but other requirements listed in the SRS are not addressed. Somewhat under revision control. | Several requirements in the SRS are not addressed. Predicted and measured performance is haphazard or missing. Not under revision control. What is included does not make sense. | |
| Test Procedures (1) | Test procedures are clearly defined. Test procedures are adequate for thorough testing of software modules. All modules have a test plan. Effectively under revision control. | Test procedures appear to be incomplete. One or two obvious tests that should be included are not. Most procedures are reproducible, but some are unclear. Somewhat under revision control. | Test procedures are haphazard and do not make sense. Test procedures are missing for several critical tests. Not under revision control. | |
| Test Results (0.5) | Test results are clear and accurately recorded. May include links to videos of software running, if applicable. | One or two test results are missing or unclear. Revision control is superficial. | Test results are missing with no explanation. Not under revision control. | |
| Supporting Artifacts – reviewers spot check | System Design Package, including System Design Package, including (at a minimum) a SBOM, diagrams, code documentation, and interface definitions. (2.5) | Software Bill of Materials is complete. All necessary logic, flow, block, wiring, and schematic diagrams are included. Interfaces are clearly defined. Software organization is clear. Code is commented well and transferable. Artifacts are under effective revision control. | Software Bill of Materials is mostly complete. Some diagrams do not clearly convey architecture. Software organization is defined, and mostly follows the guidelines. Code is a bit confusing, but someone could pick it up after brief review. Interface definitions are mostly clear and present. Revision control is superficial. | Software Bill of Materials is lacking key elements. Diagrams are confusing, if present. Software architecture does not make sense. Code is difficult to follow and lacks helpful comments. Interface definitions are missing or unclear. Not under revision control. |
| Completeness (1) | All necessary supporting artifacts are included. There is nothing the reviewer felt should be included that is missing. For Subsystem Engineering, likely supporting artifacts include: * Up-to-date Project Success Agreement * Computer files used in testing work * Other detailed artifacts that support your evaluation of the performance of the subsystems and system | Most necessary artifacts are included. One or two things the reviewer felt should be included are missing. | The secondary artifacts are incomplete. There are not enough supporting artifacts to support approval. | |
| Transferability (1) | The supporting artifacts are clearly written, and allow the reader to create the subsystem, replicate a test and/or use a prototype to repeat the work. The supporting artifacts have appropriate mechanics and are properly under revision control. | The supporting artifacts are somewhat clear. They provide enough guidance to allow the reader to apply engineering judgment to create a subsystem, replicate a test and/or use a prototype to repeat the work. The supporting artifacts have acceptable mechanics; the revision control is superficial. | Supporting artifacts are unclear and/or otherwise difficult to understand. They provide insufficient information to allow the reader to create a subsystem, replicate a test, and/or use a prototype to repeat the work. The artifact mechanics are unacceptable. Revision control is missing. | |
| Quality (1) | The supporting artifacts reflect thoughtful, appropriate engineering practice. The correct activities were performed and properly reported in order to support approval. The reviewer is confident in the information provided to support approval. | The supporting artifacts reflect engineering practice. Some of the correct activities were performed and properly reported. There is some data supporting approval, but it is not thorough or complete. The reviewer may have some questions about the information provided to support approval. | The supporting artifacts reflect poor engineering practice. There is missing evidence for some expected activities. With the existing artifacts, the reviewer has insufficient information to make an informed judgment to approve stage completion. | |
| Formatting (1) | Complies with all formatting requirements in the Student Book and is carefully and thoughtfully formatted. Bookmarks and/or hyperlinks are provided to help the reader navigate the document. | Mostly complies with formatting requirements. One or two documents need to be fixed to comply with formatting requirements | Shows little or no evidence of following formatting requirements. Haphazard formatting, difficult to read | |
| Rubric revision 0.1 made from ScoreSheetMaker revision 1.5 | ||||
Mechanics
The submitted artifacts will receive a grade based on the quality and transferability of the artifacts. However, the grade will be advisory only; it will not affect your final grade. The desire is to have you focusing on excellent artifacts that show the quality of your design, rather than having you write a report to be graded.
It is expected that most of the artifacts submitted for the Subsystem Engineering Review will be included in the Final Design Report for your Project.
The mechanics of Subsystem Engineering reviews are as follows:
- No later than 10:00 AM three class days (not counting weekends or holidays) ahead of the scheduled review, the team submits the artifact package, as described above. The submission is done from the Capstone dashboard. The review coaches and the pod instructor will receive an email when the submission is completed. If the submission is late, the evaluating coaches will not provide a grade for the initial submission.
- No later than 8:00 AM one class day (not counting weekends or holidays) ahead of the scheduled review, the two coaches who are assigned to review the team will complete their grading and written evaluation of the artifact package. These evaluations will be available for download from the Capstone dashboard. If the initial submission was late, grades will not be available, and the written evaluations may not be available before the review.
- The team prepares an oral presentation no more than five minutes in length that includes the project objective statement and discusses why the team believes they are ready for stage approval, including responses to the evaluator comments where appropriate.
- At the review, the team will make their brief presentation and demonstrate prototypes used to evaluate the subsystem designs.
- The reviewers, the team, and the coach will discuss the answers to the questions as contained in the artifacts and the presentation for 28 minutes. Weaknesses in the answers or the artifacts will be explored, and recommendations for improvement will be given.
- After consulting with the review coaches, the instructor will spend two minutes to communicate the results of the review to the team. Possible results are:
- Stage approval is granted unconditionally.
- Stage approval is conditionally granted, subject to the team making some required revisions. The team must clear the revisions with their pod instructor before approval is granted. The team presents the revised submission directly to the pod instructor for approval. After receiving approval, the team submits the revised submission on the Capstone dashboard.
- Stage approval is denied. The team will need to improve the design and/or the evaluation, schedule a new review, and return to step 1.
- When the team has received stage approval, they proceed to System Refinement.
Formatting
The artifacts in the submission should be formatted according to the instructions in Artifact Formatting.
If your sponsor requires a bound printed report, follow the additional instructions in Formatting Printed Reports.
project-wiki/design_reviews/subsystem_engineering_review.txt · Last modified: 2023/08/30 16:23 by bdj2
Except where otherwise noted, content on this wiki is licensed under the following license: Copyright Brigham Young University, all rights reserved
