Capstone Project Wiki

User Tools

Site Tools

Trace:
project-wiki:design_reviews:subsystem_engineering_review

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
project-wiki:design_reviews:subsystem_engineering_review [2022/04/04 08:15]
cds4byu [Evaluation Rubric] 		project-wiki:design_reviews:subsystem_engineering_review [2023/08/30 16:23] (current)
bdj2 [Evaluation Rubrics]
Line 27: Line 27:
  
  
-The Subsystem Engineering design review occurs in February at a date negotiated between the team and the +The Subsystem Engineering design review occurs in late February or early March at a date negotiated between the team and the 
 pod instructor. The team will propose a date in the Project Milestones Table.  The review will be scheduled to be pod instructor. The team will propose a date in the Project Milestones Table.  The review will be scheduled to be
 consistent with the instructor and the external review coaches' schedules. consistent with the instructor and the external review coaches' schedules.
Line 46: Line 46:
 finding and making plans to remove any weaknesses present. finding and making plans to remove any weaknesses present.
  
-==== Questions ====+==== Artifact Submission for Subsystem Engineering ==== 
 + 
 +In preparation for the Subsystem Engineering review, the team will prepare and submit a pdf file containing the artifacts listed below. These artifacts should answer the questions, "What is the complete system design, including the subsystem designs and their integration?" and, "How do you know that the subsystems independently meet the Project Success Agreement and other requirements and will be compatible with one another?" 
 + 
 +The submitted artifacts and the review should demonstrate 
 +that the design has evolved 
 +enough to meet the requirements for the end of Subsystem Engineering. Satisfactorily 
 +addressing the questions above constitute the minimum requirements for completion of 
 +Subsystem Engineering. 
 + 
 +While not required, it is helpful for both the team and the reviewers to include 
 +a measured performance memo at the front of the submitted artifacts. 
 +This memo 
 +should: (a) highlight measurements that show the subsystems work properly, and 
 +(b) reference artifacts that show more detail on these measurements. 
 +The presence or absence of the summary memo will not affect the review grade.
  
-  - What is the complete system design, including the subsystem designs and their integration? 
     - Primary Artifacts:     - Primary Artifacts:
-      * No primary artifacts exist for the design. 
-    - Supporting Artifacts: 
-      * System Design Package (including ,at a minimum, bill of materials, system and subsystem assembly drawings, part drawings for all custom parts, and interface definitions for subsystem interfaces). Pay close attention to the details described on [[https://byucapstone.byu.edu/mattson-sorensen.pdf#page=111|pages 98 -- 100]] of [[https://byucapstone.byu.edu/mattson-sorensen.pdf|Product Development]].This package should be complete and detailed enough that an independent production system could produce your design. 
-  - How do you know that the subsystems independently meet the Project Success Agreement and other requirements and will be compatible with one another? 
-    - Primary artifacts: 
       * Measured performance summary for each subsystem       * Measured performance summary for each subsystem
-      * Requirements matrix for each subsystem, including measured and/or predicted values for all performance measures+      * Requirements matrix or Software Requirements Specification for each subsystem. Requirements matrices should include measured and/or predicted values for all performance measures
       * Measured and predicted performance summary for the system       * Measured and predicted performance summary for the system
-      * System requirements matrix with predicted values for performance measures+      * System requirements matrix or Software Requirements Specification for the full system, with predicted values for performance measures
     - Supporting Artifacts:     - Supporting Artifacts:
 +      * System Design Package (including ,at a minimum, bill of materials or software bill of materials, system and subsystem assembly drawings, part drawings for all custom parts, and interface definitions for subsystem interfaces). Pay close attention to the details described on [[https://byucapstone.byu.edu/mattson-sorensen.pdf#page=111|pages 98 -- 100]] of [[https://byucapstone.byu.edu/mattson-sorensen.pdf|Product Development]].This package should be complete and detailed enough that an independent production system could produce your design.
       * Up-to-date approved Project Success Agreement, revised as necessary       * Up-to-date approved Project Success Agreement, revised as necessary
       * Test results obtained using models and/or prototypes, for both the subsystems and the system. Pay special attention to Figure 1 ([[https://byucapstone.byu.edu/mattson-sorensen.pdf#page=44|Figure 2.12]] in the textbook) as you consider the test results needed.       * Test results obtained using models and/or prototypes, for both the subsystems and the system. Pay special attention to Figure 1 ([[https://byucapstone.byu.edu/mattson-sorensen.pdf#page=44|Figure 2.12]] in the textbook) as you consider the test results needed.
Line 69: Line 79:
 [{{ project-wiki:design_reviews:approval.png | Figure 1: The stage approval process.  Performance testing and artifact checking are always done as part of approval.  Validation is always done at System Refinement and is often done at Concept Development (the Architecture Review). Validation is often not done at Subsystem Engineering, because the subsystem prototypes are not usually accessible for the market to evaluate.}}] [{{ project-wiki:design_reviews:approval.png | Figure 1: The stage approval process.  Performance testing and artifact checking are always done as part of approval.  Validation is always done at System Refinement and is often done at Concept Development (the Architecture Review). Validation is often not done at Subsystem Engineering, because the subsystem prototypes are not usually accessible for the market to evaluate.}}]
    
-==== Content ====+==== Evaluation Rubrics ====
  
-The submitted artifacts and the review should demonstrate 
-that the design has evolved 
-enough to meet the requirements for the end of Subsystem Engineering.  The 
-artifacts should address the questions listed above.  Satisfactorily 
-addressing these questions constitute the minimum requirements for completion of 
-Subsystem Engineering. 
  
-While not required, it is helpful for both the team and the reviewers to include +The submitted artifacts will be reviewed for how clearly and completely the answer the questions above.  The system and subsystem designs and tests presented in the artifacts will also be reviewed for their quality. 
-a measured performance memo at the front of the submitted artifacts. +Although the emphasis in the review will be on Question 2, an advisory grade will be given on the system design package as well.  The advisory grade will not count in your final grade.
-This memo +
-should: (a) highlight measurements that show the subsystems work properly, and +
-(b) reference artifacts that show more detail on these measurements. +
-The presence or absence of the summary memo will not affect the review grade.+
  
-==== Evaluation Rubric ====+The rubrics used in the evaluation is shown below.  There are different rubrics for hardware or mixed projects and software projects.  Discuss with your pod instructor which rubrics you will use. 
 + 
 +The scoresheets used by the evaluators are available on Box for [[https://byu.app.box.com/file/1271492850609|hardware or mixed projects]] and for [[https://byu.app.box.com/file/1271491926396|software projects]]. 
  
  
-The submitted artifacts will be reviewed for how clearly and completely the answer the questions above.  The system and subsystem designs and tests presented in the artifacts will also be reviewed for their quality. 
-Although the emphasis in the review will be on Question 2, an advisory grade will be given on the system design package as well.  The advisory grade will not count in your final grade. 
  
-The rubric shown below will be used by the evaluators.  The scoresheet used by the evaluators is available 
-[[https://byu.app.box.com/file/831598289002|on Box]]. 
  
 <nodisp 16> <nodisp 16>
-NOTE TO INSTRUCTORS: This table is automatically created by [[https://byu.app.box.com/file/830893783568|ScoreSheetMaker.xlsm]].  Make the changes in ScoresheetMaker.xlsm so they will be+NOTE TO INSTRUCTORS: This table is automatically created by [[https://byu.app.box.com/file/1281677015283|ScoreSheetMaker.xlsm]].  Make the changes in ScoresheetMaker.xlsm so they will be
 consistent with the scoresheets. consistent with the scoresheets.
 </nodisp> </nodisp>
Line 105: Line 103:
 |::: ^<html><small>System Requirements Matrix, including predicted values for system performance measures (0.5)</small></html> |<html><small> Matrix adds Predicted Values and some Measured Values to the information from Concept Development.  Meets all of the criteria for the requirements matrix information found in Table 5.1 or Table A.3 of the textbook. Matrix is under effective revision control.  If an alternative requirement specification is used, it contains predicted system performance expectations. </small></html> |<html><small> Matrix is mostly well done, but falls short of the listed criteria in one or two areas.  Measurements and Predictions appear superficial.  Revision control is superficial. </small></html> |<html><small> Matrix has some serious weaknesses that raise questions about its utility. Measurements and predictions are unbelievable or missing.  Not under revision control.   </small></html> | |::: ^<html><small>System Requirements Matrix, including predicted values for system performance measures (0.5)</small></html> |<html><small> Matrix adds Predicted Values and some Measured Values to the information from Concept Development.  Meets all of the criteria for the requirements matrix information found in Table 5.1 or Table A.3 of the textbook. Matrix is under effective revision control.  If an alternative requirement specification is used, it contains predicted system performance expectations. </small></html> |<html><small> Matrix is mostly well done, but falls short of the listed criteria in one or two areas.  Measurements and Predictions appear superficial.  Revision control is superficial. </small></html> |<html><small> Matrix has some serious weaknesses that raise questions about its utility. Measurements and predictions are unbelievable or missing.  Not under revision control.   </small></html> |
 |Supporting Artifacts -- reviewers spot check ^<html><small>System Design Package, including (at a minimum)  a BOM, system and subsystem assembly drawings, part drawings, and interface definitions. (2.5)</small></html> |<html><small> Drawing package is complete.  Bill of Materials is complete.  All custom parts have drawings.  All necessary logic, flow, block, wiring, and schematic diagrams are included.  Drawings follow appropriate standards.  Drawings have been checked by someone other than that creator.  The entire system is specified.  Software organization is clear and effective, and follows the guidelines in the Design Artifacts section of the Student Book.  Drawing package is under effective revision control. </small></html> |<html><small> Drawing package is mostly complete.  Bill of Materials is mostly complete.  Most custom parts have drawings.  Non-geometric drawings are mostly present.  Appropriate standards are mostly followed.  Drawing checking appears to have been somewhat superficial.  Key subsystems are specified.  Software organization is defined, and mostly follows the guidelines.  Revision control is superficial. </small></html> |<html><small>  Drawing package is substantially incomplete.  Many necessary drawings are missing.  Standards are followed poorly or ignored. Checking is minimal or nonexistent.   Software organization is not defined, or does not follow the guidelines. Not under revision control. </small></html> | |Supporting Artifacts -- reviewers spot check ^<html><small>System Design Package, including (at a minimum)  a BOM, system and subsystem assembly drawings, part drawings, and interface definitions. (2.5)</small></html> |<html><small> Drawing package is complete.  Bill of Materials is complete.  All custom parts have drawings.  All necessary logic, flow, block, wiring, and schematic diagrams are included.  Drawings follow appropriate standards.  Drawings have been checked by someone other than that creator.  The entire system is specified.  Software organization is clear and effective, and follows the guidelines in the Design Artifacts section of the Student Book.  Drawing package is under effective revision control. </small></html> |<html><small> Drawing package is mostly complete.  Bill of Materials is mostly complete.  Most custom parts have drawings.  Non-geometric drawings are mostly present.  Appropriate standards are mostly followed.  Drawing checking appears to have been somewhat superficial.  Key subsystems are specified.  Software organization is defined, and mostly follows the guidelines.  Revision control is superficial. </small></html> |<html><small>  Drawing package is substantially incomplete.  Many necessary drawings are missing.  Standards are followed poorly or ignored. Checking is minimal or nonexistent.   Software organization is not defined, or does not follow the guidelines. Not under revision control. </small></html> |
-|::: ^<html><small>Completeness (1)</small></html> |<html><small>All necessary supporting artifacts are included.  There is nothing the reviewer felt should be included that is missing.  For Subsystem Engineering, likely supporting  artifacts include<br>* System Design Package<br>* Up-to-date Project Success Agreement<br>* Test results obtained using models and/or prototypes<br>* Definitions of models and/or prototypes<br>* Test procedures used to obtain results in Primary Artifacts<br>* Computer files used in testing work<br>* Other detailed artifacts that support your evaluation of the performance of the subsystems and system </small></html> |<html><small> Most necessary artifacts are included.  One or two things the reviewer felt should be included are missing. </small></html> |<html><small>  The secondary artifacts are incomplete.  There are not enough supporting artifacts to support approval. </small></html> |+|::: ^<html><small>Test procedures and results (0.75)</small></html> |<html><small>All necessary supporting artifacts on testing are included.  There is nothing the reviewer felt should be included that is missing.  For Subsystem Engineering, likely testing  artifacts include<br>* Test results obtained using models and/or prototypes<br>* Test procedures used to obtain results in Primary Artifacts<br>* Computer files used in testing work<br> </small></html> |<html><small> Most necessary testing artifacts are included.  One or two things the reviewer felt should be included are missing. </small></html> |<html><small>  The testing supporting artifacts are incomplete.  There are not enough supporting artifacts to support approval. </small></html> | 
 +|::: ^<html><small>Other supporting artifacts (0.25)</small></html> |<html><small>All other necessary supporting artifacts are included.  There is nothing the reviewer felt should be included that is missing.  For Subsystem Engineering, likely other  artifacts include<br>* Up-to-date signed Project Success Agreement<br>* Definitions of models and/or prototypes<br>* Other detailed artifacts that support your evaluation of the performance of the subsystems and system </small></html> |<html><small> Most necessary artifacts are included.  One or two things the reviewer felt should be included are missing. </small></html> |<html><small>  The other supporting artifacts are incomplete.  There are not enough supporting artifacts to support approval. </small></html> |
 |::: ^<html><small>Transferability (1)</small></html> |<html><small> The supporting artifacts are clearly written, and allow the reader to create the subsystem, replicate a test and/or use a prototype to repeat the work.  The supporting artifacts have appropriate mechanics and are properly under revision control. </small></html> |<html><small> The supporting artifacts are somewhat clear.  They provide enough guidance to allow the reader to apply engineering judgment to create a subsystem, replicate a test and/or use a prototype to repeat the work.  The supporting artifacts have acceptable mechanics; the revision control is superficial. </small></html> |<html><small> Supporting artifacts are unclear and/or otherwise difficult to understand.  They provide insufficient information to allow the reader to create a subsystem, replicate a test, and/or use a prototype to repeat the work.  The artifact mechanics are unacceptable.  Revision control is missing. </small></html> | |::: ^<html><small>Transferability (1)</small></html> |<html><small> The supporting artifacts are clearly written, and allow the reader to create the subsystem, replicate a test and/or use a prototype to repeat the work.  The supporting artifacts have appropriate mechanics and are properly under revision control. </small></html> |<html><small> The supporting artifacts are somewhat clear.  They provide enough guidance to allow the reader to apply engineering judgment to create a subsystem, replicate a test and/or use a prototype to repeat the work.  The supporting artifacts have acceptable mechanics; the revision control is superficial. </small></html> |<html><small> Supporting artifacts are unclear and/or otherwise difficult to understand.  They provide insufficient information to allow the reader to create a subsystem, replicate a test, and/or use a prototype to repeat the work.  The artifact mechanics are unacceptable.  Revision control is missing. </small></html> |
 |::: ^<html><small>Quality (1)</small></html> |<html><small> The supporting artifacts reflect thoughtful, appropriate engineering practice.  The correct activities were performed and properly reported in order to support approval.  The reviewer is confident in the information provided to support approval. </small></html> |<html><small> The supporting artifacts reflect engineering practice.  Some of the correct activities were performed and properly reported.  There is some data supporting approval, but it is not thorough or complete.  The reviewer may have some questions about the information provided to support approval. </small></html> |<html><small> The supporting artifacts reflect poor engineering practice.  There is missing evidence for some expected activities.  With the existing artifacts, the reviewer has insufficient information to make an informed judgment to approve stage completion. </small></html> | |::: ^<html><small>Quality (1)</small></html> |<html><small> The supporting artifacts reflect thoughtful, appropriate engineering practice.  The correct activities were performed and properly reported in order to support approval.  The reviewer is confident in the information provided to support approval. </small></html> |<html><small> The supporting artifacts reflect engineering practice.  Some of the correct activities were performed and properly reported.  There is some data supporting approval, but it is not thorough or complete.  The reviewer may have some questions about the information provided to support approval. </small></html> |<html><small> The supporting artifacts reflect poor engineering practice.  There is missing evidence for some expected activities.  With the existing artifacts, the reviewer has insufficient information to make an informed judgment to approve stage completion. </small></html> |
Line 112: Line 111:
  
  
 +
 +<nodisp 16>
 +NOTE TO INSTRUCTORS: This table is automatically created by [[https://byu.app.box.com/file/1281677015283|ScoreSheetMaker.xlsm]].  Make the changes in ScoresheetMaker.xlsm so they will be
 +consistent with the scoresheets.
 +</nodisp>
 +**Evaluation rubric for Subsystem Engineering stage on software projects**
 +^Element ^ Item(weight) ^ Excellent (9-10) ^ Good (7-8) ^ Poor (0-6) ^
 +|Primary Artifacts -- reviewers evaluate all ^<html><small>Performance Summary - Key Success Measures  (1.5)</small></html> |<html><small>Clearly summarizes current performance against Key Success Measures. References test procedures and test results artifacts. Honestly assesses performance. Includes predicted results for future software revisions. Under effective revision control. </small></html> |<html><small>System performance is mostly addressed but leaves some questions. Refers to testing and test procedure artifacts for some testing. Lacking details to instill confidence in future performance. Somewhat under revision control. </small></html> |<html><small>Leaves significant portions of performance unaddressed. Some Key Success Measures are not addressed. Provides little or no evidence supporting performance claims. Is not honest about real performance of system. Is not under revision control. </small></html> |
 +|::: ^<html><small>Performance Summary - Software Requirements Specification (SRS) (1.5)</small></html> |<html><small>Clearly demonstrates current performance against requirements listed in SRS. References test procedures and test results artifacts. Honestly assesses performance. Under effective revision control. </small></html> |<html><small>Some predicted and measured performance is reported, but other requirements listed in the SRS are not addressed. Somewhat under revision control.  </small></html> |<html><small>Several requirements in the SRS are not addressed. Predicted and measured performance is haphazard or missing. Not under revision control. What is included does not make sense. </small></html> |
 +|::: ^<html><small>Test Procedures  (1)</small></html> |<html><small>Test procedures are clearly defined. Test procedures are adequate for thorough testing of software modules. All modules have a test plan. Effectively under revision control. </small></html> |<html><small>Test procedures appear to be incomplete. One or two obvious tests that should be included are not. Most procedures are reproducible, but some are unclear. Somewhat under revision control. </small></html> |<html><small>Test procedures are haphazard and do not make sense. Test procedures are missing for several critical tests. Not under revision control. </small></html> |
 +|::: ^<html><small>Test Results  (0.5)</small></html> |<html><small>Test results are clear and accurately recorded. May include links to videos of software running, if applicable.  </small></html> |<html><small>One or two test results are missing or unclear. Revision control is superficial. </small></html> |<html><small>Test results are missing with no explanation. Not under revision control. </small></html> |
 +|Supporting Artifacts -- reviewers spot check ^<html><small>System Design Package, including System Design Package, including (at a minimum) a SBOM, diagrams, code documentation, and interface definitions.  (2.5)</small></html> |<html><small>Software Bill of Materials is complete. All necessary logic, flow, block, wiring, and schematic diagrams are included. Interfaces are clearly defined. Software organization is clear. Code is commented well and transferable. Artifacts are under effective revision control. </small></html> |<html><small>Software Bill of Materials is mostly complete. Some diagrams do not clearly convey architecture. Software organization is defined, and mostly follows the guidelines. Code is a bit confusing, but someone could pick it up after brief review. Interface definitions are mostly clear and present. Revision control is superficial. </small></html> |<html><small>Software Bill of Materials is lacking key elements. Diagrams are confusing, if present. Software architecture does not make sense. Code is difficult to follow and lacks helpful comments. Interface definitions are missing or unclear. Not under revision control. </small></html> |
 +|::: ^<html><small>Completeness (1)</small></html> |<html><small>All necessary supporting artifacts are included.  There is nothing the reviewer felt should be included that is missing.  For Subsystem Engineering, likely supporting  artifacts include:<br>* Up-to-date Project Success Agreement<br>* Computer files used in testing work<br>* Other detailed artifacts that support your evaluation of the performance of the subsystems and system </small></html> |<html><small> Most necessary artifacts are included. One or two things the reviewer felt should be included are missing. </small></html> |<html><small> The secondary artifacts are incomplete. There are not enough supporting artifacts to support approval. </small></html> |
 +|::: ^<html><small>Transferability (1)</small></html> |<html><small>The supporting artifacts are clearly written, and allow the reader to create the subsystem, replicate a test and/or use a prototype to repeat the work. The supporting artifacts have appropriate mechanics and are properly under revision control. </small></html> |<html><small>The supporting artifacts are somewhat clear. They provide enough guidance to allow the reader to apply engineering judgment to create a subsystem, replicate a test and/or use a prototype to repeat the work. The supporting artifacts have acceptable mechanics; the revision control is superficial. </small></html> |<html><small>Supporting artifacts are unclear and/or otherwise difficult to understand. They provide insufficient information to allow the reader to create a subsystem, replicate a test, and/or use a prototype to repeat the work. The artifact mechanics are unacceptable.  Revision control is missing. </small></html> |
 +|::: ^<html><small>Quality (1)</small></html> |<html><small>The supporting artifacts reflect thoughtful, appropriate engineering practice. The correct activities were performed and properly reported in order to support approval. The reviewer is confident in the information provided to support approval. </small></html> |<html><small>The supporting artifacts reflect engineering practice.  Some of the correct activities were performed and properly reported. There is some data supporting approval, but it is not thorough or complete. The reviewer may have some questions about the information provided to support approval. </small></html> |<html><small>The supporting artifacts reflect poor engineering practice. There is missing evidence for some expected activities. With the existing artifacts, the reviewer has insufficient information to make an informed judgment to approve stage completion. </small></html> |
 +| ^<html><small>Formatting (1)</small></html> |<html><small>Complies with all formatting requirements in the Student Book and is carefully and thoughtfully formatted. Bookmarks and/or hyperlinks are provided to help the reader navigate the document. </small></html> |<html><small>Mostly complies with formatting requirements. One or two documents need to be fixed to comply with formatting requirements </small></html> |<html><small>Shows little or no evidence of following formatting requirements. Haphazard formatting, difficult to read </small></html> |
 +|<html><small>Rubric revision 0.1 made from ScoreSheetMaker revision 1.5</small></html>|||||
  
 ==== Mechanics ==== ==== Mechanics ====
project-wiki/design_reviews/subsystem_engineering_review.1649081728.txt.gz · Last modified: 2022/04/04 08:15 by cds4byu

Page Tools

Except where otherwise noted, content on this wiki is licensed under the following license: Copyright Brigham Young University, all rights reserved
Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki