User Tools

Site Tools


project-wiki:design_reviews:subsystem_engineering_review

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
project-wiki:design_reviews:subsystem_engineering_review [2022/09/09 11:27]
cds4byu [Evaluation Rubrics]
project-wiki:design_reviews:subsystem_engineering_review [2023/08/30 16:23] (current)
bdj2 [Evaluation Rubrics]
Line 27: Line 27:
  
  
-The Subsystem Engineering design review occurs in February at a date negotiated between the team and the +The Subsystem Engineering design review occurs in late February or early March at a date negotiated between the team and the 
 pod instructor. The team will propose a date in the Project Milestones Table.  The review will be scheduled to be pod instructor. The team will propose a date in the Project Milestones Table.  The review will be scheduled to be
 consistent with the instructor and the external review coaches' schedules. consistent with the instructor and the external review coaches' schedules.
Line 46: Line 46:
 finding and making plans to remove any weaknesses present. finding and making plans to remove any weaknesses present.
  
-==== Questions ====+==== Artifact Submission for Subsystem Engineering ==== 
 + 
 +In preparation for the Subsystem Engineering review, the team will prepare and submit a pdf file containing the artifacts listed below. These artifacts should answer the questions, "What is the complete system design, including the subsystem designs and their integration?" and, "How do you know that the subsystems independently meet the Project Success Agreement and other requirements and will be compatible with one another?" 
 + 
 +The submitted artifacts and the review should demonstrate 
 +that the design has evolved 
 +enough to meet the requirements for the end of Subsystem Engineering. Satisfactorily 
 +addressing the questions above constitute the minimum requirements for completion of 
 +Subsystem Engineering. 
 + 
 +While not required, it is helpful for both the team and the reviewers to include 
 +a measured performance memo at the front of the submitted artifacts. 
 +This memo 
 +should: (a) highlight measurements that show the subsystems work properly, and 
 +(b) reference artifacts that show more detail on these measurements. 
 +The presence or absence of the summary memo will not affect the review grade.
  
-  - What is the complete system design, including the subsystem designs and their integration? 
     - Primary Artifacts:     - Primary Artifacts:
-      * No primary artifacts exist for the design. 
-    - Supporting Artifacts: 
-      * System Design Package (including ,at a minimum, bill of materials, system and subsystem assembly drawings, part drawings for all custom parts, and interface definitions for subsystem interfaces). Pay close attention to the details described on [[https://byucapstone.byu.edu/mattson-sorensen.pdf#page=111|pages 98 -- 100]] of [[https://byucapstone.byu.edu/mattson-sorensen.pdf|Product Development]].This package should be complete and detailed enough that an independent production system could produce your design. 
-  - How do you know that the subsystems independently meet the Project Success Agreement and other requirements and will be compatible with one another? 
-    - Primary artifacts: 
       * Measured performance summary for each subsystem       * Measured performance summary for each subsystem
-      * Requirements matrix for each subsystem, including measured and/or predicted values for all performance measures+      * Requirements matrix or Software Requirements Specification for each subsystem. Requirements matrices should include measured and/or predicted values for all performance measures
       * Measured and predicted performance summary for the system       * Measured and predicted performance summary for the system
-      * System requirements matrix with predicted values for performance measures+      * System requirements matrix or Software Requirements Specification for the full system, with predicted values for performance measures
     - Supporting Artifacts:     - Supporting Artifacts:
 +      * System Design Package (including ,at a minimum, bill of materials or software bill of materials, system and subsystem assembly drawings, part drawings for all custom parts, and interface definitions for subsystem interfaces). Pay close attention to the details described on [[https://byucapstone.byu.edu/mattson-sorensen.pdf#page=111|pages 98 -- 100]] of [[https://byucapstone.byu.edu/mattson-sorensen.pdf|Product Development]].This package should be complete and detailed enough that an independent production system could produce your design.
       * Up-to-date approved Project Success Agreement, revised as necessary       * Up-to-date approved Project Success Agreement, revised as necessary
       * Test results obtained using models and/or prototypes, for both the subsystems and the system. Pay special attention to Figure 1 ([[https://byucapstone.byu.edu/mattson-sorensen.pdf#page=44|Figure 2.12]] in the textbook) as you consider the test results needed.       * Test results obtained using models and/or prototypes, for both the subsystems and the system. Pay special attention to Figure 1 ([[https://byucapstone.byu.edu/mattson-sorensen.pdf#page=44|Figure 2.12]] in the textbook) as you consider the test results needed.
Line 69: Line 79:
 [{{ project-wiki:design_reviews:approval.png | Figure 1: The stage approval process.  Performance testing and artifact checking are always done as part of approval.  Validation is always done at System Refinement and is often done at Concept Development (the Architecture Review). Validation is often not done at Subsystem Engineering, because the subsystem prototypes are not usually accessible for the market to evaluate.}}] [{{ project-wiki:design_reviews:approval.png | Figure 1: The stage approval process.  Performance testing and artifact checking are always done as part of approval.  Validation is always done at System Refinement and is often done at Concept Development (the Architecture Review). Validation is often not done at Subsystem Engineering, because the subsystem prototypes are not usually accessible for the market to evaluate.}}]
    
-==== Content ==== 
- 
-The submitted artifacts and the review should demonstrate 
-that the design has evolved 
-enough to meet the requirements for the end of Subsystem Engineering.  The 
-artifacts should address the questions listed above.  Satisfactorily 
-addressing these questions constitute the minimum requirements for completion of 
-Subsystem Engineering. 
- 
-While not required, it is helpful for both the team and the reviewers to include 
-a measured performance memo at the front of the submitted artifacts. 
-This memo 
-should: (a) highlight measurements that show the subsystems work properly, and 
-(b) reference artifacts that show more detail on these measurements. 
-The presence or absence of the summary memo will not affect the review grade. 
- 
 ==== Evaluation Rubrics ==== ==== Evaluation Rubrics ====
  
Line 93: Line 87:
 The rubrics used in the evaluation is shown below.  There are different rubrics for hardware or mixed projects and software projects.  Discuss with your pod instructor which rubrics you will use. The rubrics used in the evaluation is shown below.  There are different rubrics for hardware or mixed projects and software projects.  Discuss with your pod instructor which rubrics you will use.
  
-The scoresheets used by the evaluators are available on Box for [[https://byu.app.box.com/file/998803361883|hardware or mixed projects]] and for [[https://byu.app.box.com/file/1010818483530|software projects]]. +The scoresheets used by the evaluators are available on Box for [[https://byu.app.box.com/file/1271492850609|hardware or mixed projects]] and for [[https://byu.app.box.com/file/1271491926396|software projects]]. 
  
  
Line 99: Line 93:
  
 <nodisp 16> <nodisp 16>
-NOTE TO INSTRUCTORS: This table is automatically created by [[https://byu.app.box.com/file/830893783568|ScoreSheetMaker.xlsm]].  Make the changes in ScoresheetMaker.xlsm so they will be+NOTE TO INSTRUCTORS: This table is automatically created by [[https://byu.app.box.com/file/1281677015283|ScoreSheetMaker.xlsm]].  Make the changes in ScoresheetMaker.xlsm so they will be
 consistent with the scoresheets. consistent with the scoresheets.
 </nodisp> </nodisp>
Line 119: Line 113:
  
 <nodisp 16> <nodisp 16>
-NOTE TO INSTRUCTORS: This table is automatically created by [[https://byu.app.box.com/file/830893783568|ScoreSheetMaker.xlsm]].  Make the changes in ScoresheetMaker.xlsm so they will be+NOTE TO INSTRUCTORS: This table is automatically created by [[https://byu.app.box.com/file/1281677015283|ScoreSheetMaker.xlsm]].  Make the changes in ScoresheetMaker.xlsm so they will be
 consistent with the scoresheets. consistent with the scoresheets.
 </nodisp> </nodisp>
Line 128: Line 122:
 |::: ^<html><small>Test Procedures  (1)</small></html> |<html><small>Test procedures are clearly defined. Test procedures are adequate for thorough testing of software modules. All modules have a test plan. Effectively under revision control. </small></html> |<html><small>Test procedures appear to be incomplete. One or two obvious tests that should be included are not. Most procedures are reproducible, but some are unclear. Somewhat under revision control. </small></html> |<html><small>Test procedures are haphazard and do not make sense. Test procedures are missing for several critical tests. Not under revision control. </small></html> | |::: ^<html><small>Test Procedures  (1)</small></html> |<html><small>Test procedures are clearly defined. Test procedures are adequate for thorough testing of software modules. All modules have a test plan. Effectively under revision control. </small></html> |<html><small>Test procedures appear to be incomplete. One or two obvious tests that should be included are not. Most procedures are reproducible, but some are unclear. Somewhat under revision control. </small></html> |<html><small>Test procedures are haphazard and do not make sense. Test procedures are missing for several critical tests. Not under revision control. </small></html> |
 |::: ^<html><small>Test Results  (0.5)</small></html> |<html><small>Test results are clear and accurately recorded. May include links to videos of software running, if applicable.  </small></html> |<html><small>One or two test results are missing or unclear. Revision control is superficial. </small></html> |<html><small>Test results are missing with no explanation. Not under revision control. </small></html> | |::: ^<html><small>Test Results  (0.5)</small></html> |<html><small>Test results are clear and accurately recorded. May include links to videos of software running, if applicable.  </small></html> |<html><small>One or two test results are missing or unclear. Revision control is superficial. </small></html> |<html><small>Test results are missing with no explanation. Not under revision control. </small></html> |
-|::: ^<html><small>System Design Package, including System Design Package, including (at a minimum) a SBOM, diagrams, code documentation, and interface definitions.  (2.5)</small></html> |<html><small>Software Bill of Materials is complete. All necessary logic, flow, block, wiring, and schematic diagrams are included. Interfaces are clearly defined. Software organization is clear. Code is commented well and transferable. Artifacts are under effective revision control. </small></html> |<html><small>Software Bill of Materials is mostly complete. Some diagrams do not clearly convey architecture. Software organization is defined, and mostly follows the guidelines. Code is a bit confusing, but someone could pick it up after brief review. Interface definitions are mostly clear and present. Revision control is superficial. </small></html> |<html><small>Software Bill of Materials is lacking key elements. Diagrams are confusing, if present. Software architecture does not make sense. Code is difficult to follow and lacks helpful comments. Interface definitions are missing or unclear. Not under revision control. </small></html>+|Supporting Artifacts -- reviewers spot check ^<html><small>System Design Package, including System Design Package, including (at a minimum) a SBOM, diagrams, code documentation, and interface definitions.  (2.5)</small></html> |<html><small>Software Bill of Materials is complete. All necessary logic, flow, block, wiring, and schematic diagrams are included. Interfaces are clearly defined. Software organization is clear. Code is commented well and transferable. Artifacts are under effective revision control. </small></html> |<html><small>Software Bill of Materials is mostly complete. Some diagrams do not clearly convey architecture. Software organization is defined, and mostly follows the guidelines. Code is a bit confusing, but someone could pick it up after brief review. Interface definitions are mostly clear and present. Revision control is superficial. </small></html> |<html><small>Software Bill of Materials is lacking key elements. Diagrams are confusing, if present. Software architecture does not make sense. Code is difficult to follow and lacks helpful comments. Interface definitions are missing or unclear. Not under revision control. </small></html>
-|Supporting Artifacts -- reviewers spot check ^<html><small>Completeness (1)</small></html> |<html><small>All necessary supporting artifacts are included.  There is nothing the reviewer felt should be included that is missing.  For Subsystem Engineering, likely supporting  artifacts include:<br>* Up-to-date Project Success Agreement<br>* Computer files used in testing work<br>* Other detailed artifacts that support your evaluation of the performance of the subsystems and system </small></html> |<html><small> Most necessary artifacts are included. One or two things the reviewer felt should be included are missing. </small></html> |<html><small> The secondary artifacts are incomplete. There are not enough supporting artifacts to support approval. </small></html> |+|::: ^<html><small>Completeness (1)</small></html> |<html><small>All necessary supporting artifacts are included.  There is nothing the reviewer felt should be included that is missing.  For Subsystem Engineering, likely supporting  artifacts include:<br>* Up-to-date Project Success Agreement<br>* Computer files used in testing work<br>* Other detailed artifacts that support your evaluation of the performance of the subsystems and system </small></html> |<html><small> Most necessary artifacts are included. One or two things the reviewer felt should be included are missing. </small></html> |<html><small> The secondary artifacts are incomplete. There are not enough supporting artifacts to support approval. </small></html> |
 |::: ^<html><small>Transferability (1)</small></html> |<html><small>The supporting artifacts are clearly written, and allow the reader to create the subsystem, replicate a test and/or use a prototype to repeat the work. The supporting artifacts have appropriate mechanics and are properly under revision control. </small></html> |<html><small>The supporting artifacts are somewhat clear. They provide enough guidance to allow the reader to apply engineering judgment to create a subsystem, replicate a test and/or use a prototype to repeat the work. The supporting artifacts have acceptable mechanics; the revision control is superficial. </small></html> |<html><small>Supporting artifacts are unclear and/or otherwise difficult to understand. They provide insufficient information to allow the reader to create a subsystem, replicate a test, and/or use a prototype to repeat the work. The artifact mechanics are unacceptable.  Revision control is missing. </small></html> | |::: ^<html><small>Transferability (1)</small></html> |<html><small>The supporting artifacts are clearly written, and allow the reader to create the subsystem, replicate a test and/or use a prototype to repeat the work. The supporting artifacts have appropriate mechanics and are properly under revision control. </small></html> |<html><small>The supporting artifacts are somewhat clear. They provide enough guidance to allow the reader to apply engineering judgment to create a subsystem, replicate a test and/or use a prototype to repeat the work. The supporting artifacts have acceptable mechanics; the revision control is superficial. </small></html> |<html><small>Supporting artifacts are unclear and/or otherwise difficult to understand. They provide insufficient information to allow the reader to create a subsystem, replicate a test, and/or use a prototype to repeat the work. The artifact mechanics are unacceptable.  Revision control is missing. </small></html> |
 |::: ^<html><small>Quality (1)</small></html> |<html><small>The supporting artifacts reflect thoughtful, appropriate engineering practice. The correct activities were performed and properly reported in order to support approval. The reviewer is confident in the information provided to support approval. </small></html> |<html><small>The supporting artifacts reflect engineering practice.  Some of the correct activities were performed and properly reported. There is some data supporting approval, but it is not thorough or complete. The reviewer may have some questions about the information provided to support approval. </small></html> |<html><small>The supporting artifacts reflect poor engineering practice. There is missing evidence for some expected activities. With the existing artifacts, the reviewer has insufficient information to make an informed judgment to approve stage completion. </small></html> | |::: ^<html><small>Quality (1)</small></html> |<html><small>The supporting artifacts reflect thoughtful, appropriate engineering practice. The correct activities were performed and properly reported in order to support approval. The reviewer is confident in the information provided to support approval. </small></html> |<html><small>The supporting artifacts reflect engineering practice.  Some of the correct activities were performed and properly reported. There is some data supporting approval, but it is not thorough or complete. The reviewer may have some questions about the information provided to support approval. </small></html> |<html><small>The supporting artifacts reflect poor engineering practice. There is missing evidence for some expected activities. With the existing artifacts, the reviewer has insufficient information to make an informed judgment to approve stage completion. </small></html> |
 | ^<html><small>Formatting (1)</small></html> |<html><small>Complies with all formatting requirements in the Student Book and is carefully and thoughtfully formatted. Bookmarks and/or hyperlinks are provided to help the reader navigate the document. </small></html> |<html><small>Mostly complies with formatting requirements. One or two documents need to be fixed to comply with formatting requirements </small></html> |<html><small>Shows little or no evidence of following formatting requirements. Haphazard formatting, difficult to read </small></html> | | ^<html><small>Formatting (1)</small></html> |<html><small>Complies with all formatting requirements in the Student Book and is carefully and thoughtfully formatted. Bookmarks and/or hyperlinks are provided to help the reader navigate the document. </small></html> |<html><small>Mostly complies with formatting requirements. One or two documents need to be fixed to comply with formatting requirements </small></html> |<html><small>Shows little or no evidence of following formatting requirements. Haphazard formatting, difficult to read </small></html> |
 |<html><small>Rubric revision 0.1 made from ScoreSheetMaker revision 1.5</small></html>||||| |<html><small>Rubric revision 0.1 made from ScoreSheetMaker revision 1.5</small></html>|||||
- 
- 
  
 ==== Mechanics ==== ==== Mechanics ====
project-wiki/design_reviews/subsystem_engineering_review.1662744439.txt.gz · Last modified: 2022/09/09 11:27 by cds4byu