class: center, middle, inverse, title-slide # What the <span class="citation">@*</span>%# is Program Evaluation? ### Dr. Abhik Roy --- <style> section { display: flex; display: -webkit-flex; } section p { margin: auto; } section { height: 600px; width: 60%; margin: auto; border-radius: 20px; background-color: #212121; } section p { text-align: center; font-size: 30px; background-color: #212121; border-radius: 20px; font-family: Roboto Condensed; font-style: bold; padding: 15px; color: #bff4ee; } div.a { text-align: center; } .center2 { margin: 0; position: absolute; top: 50%; left: 50%; -ms-transform: translate(-50%, -50%); transform: translate(-50%, -50%); } </style> # Before Starting There are many types of evaluations (e.g. personnel, product, etc.) each with their own distinct purposes, approaches, and general outcomes. For the sake of this slideshow, we'll use the term **evaluation** to refer to **program evaluations**. --- # First things First: Evaluation is not - research -- or -- - assessment -- .center2[  ] --- # Assessment - <span style='color:#c8acf4'>Asks</span> what is going on at this very moment? -- - <span style='color:#18fff9'>Provides</span> a snapshot that may be in conjunction with other snapshots to gain a basic picture. -- - Can be answered with a yes or no but only for "right now" -- - Is a part of the evaluative process --- # Research - <span style='color:#c8acf4'>Asks</span> *how so?* and *what are the general implications based on the sample?* -- - <span style='color:#18fff9'>Provides</span> information to propagate further research -- - Cannot be answered with a yes or no and is intended to propagate knowledge -- - Is generally disjoint from evaluation except in research methodology and design --- # Evaluation is Based on <table> <thead> <tr> <th style="text-align:left;"> standards associated with </th> <th style="text-align:left;"> <i>with the intent to uncover</i> </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> <span style='color:#81b29a'>Validity</span> </td> <td style="text-align:left;"> <span style='color:#81b29a'>the degree to which the variable measures what it is intended to measure</span> </td> </tr> <tr> <td style="text-align:left;"> <span style='color:#e07a5f'>Reliability</span> </td> <td style="text-align:left;"> <span style='color:#e07a5f'>the extent to which the measurements remain consistent over repeated tests of the same subject under identical conditions</span> </td> </tr> </tbody> </table> --- # Evaluation is Conducted to <table> <thead> <tr> <th style="text-align:left;"> </th> <th style="text-align:left;"> <i>with the caveat that</i> </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> <b>collect evidence</b> </td> <td style="text-align:left;"> access to resources and data will be limited </td> </tr> <tr> <td style="text-align:left;"> <b>get to the truth</b> </td> <td style="text-align:left;"> an evaluation may never see the light of day </td> </tr> </tbody> </table> --- # So what is evaluation then? There are two overarching types -- - Informal -- - Formal --- # Informal evaluation You do this everyday! .center2[  ] --- # Formal evaluation .center2[  ] .footnote[(Davidson, 2005; Scriven, 1991)] --- # The In a Nutshell Evaluation Description -- <br><br> <div class="a">An evaluation tries to answer</a> <br><br> <div class="a" style="color:#2bd1fc;font-size: 1.6em;"> <b><i> does it work and why does it work?</i></b> </div> --- # What do you have to do? - Think for yourself - Be or become independent - Have or learn the ability to map out a study from start to finish with the knowledge that it will likely not got to plan - Adjust to stakeholder and environmental needs - Understand and be comfortable with that fact that you will not have any content knowledge of a majority of programs you will evaluate --- .center2[  ] --- # Evaluation Components --- # The Three Aspects of Evaluation -- - Criteria -- - Standards -- - Indicators --- # Criteria - <p style="color:#5ee1ba;">Definition</p> - An assessment a program's ability to achieve its intended outcomes (i.e. does what its supposed to do) AND make a meaningful differences as a consequence to its operation. -- <br> - <p style="color:#e87965;">Example</p> - A car may be worth buying if it meets the following criteria: it is reliable, has a five star safety rating, has good fuel consumption, and has the ability to self-park. --- # Standards - <p style="color:#5ee1ba;">Definition</p> - The levels of performance expressed as a rating or grade -- <br> - <p style="color:#e87965;">Examples</p> - *Quantitative*: Minimum of a 2.8 GPA to gain entrance into a graduate program - *Qualitative*: Performance in a thesis defense --- # Indicators - <p style="color:#5ee1ba;">Definition</p> - Aspects that can be measured within an evaluation that may tell us what is actually going on. -- <br> - <p style="color:#e87965;">Example</p> - The score on the Stanford-Binet test is an indicator of the IQ as a variable. -- <br> - <p style="color:#5ec7e1;">Note</p> - Validity and reliability are not addressed here - Indicators by themselves do not have to have these aspects. --- # The Three Purposes of Evaluation To determine -- - Merit -- - Worth -- - Significance --- # Merit - <span style='color:#dfbbbb'>Synonymous</span> with **Quality** -- - <span style='color:#c8acf4'>Asks</span> *does something do well in what it is supposed to do?* -- - <p style="color:#e87965;">Example</p> - *Question*: Does the SNA course succeed in building an understanding of the content and practice of social networks? - *Outcome*: If so, it probably has merit. Otherwise likely not --- # Worth - <span style='color:#dfbbbb'>Synonymous</span> with **Value** -- - <span style='color:#c8acf4'>Asks</span> *how valuable is this and to whom?* -- - <p style="color:#e87965;">Example</p> - *Question*: Do students who pass the SNA course tend to use social network analysis in their future work? - *Outcome*: If so, it probably has worth. Otherwise likely not --- # Significance - <span style='color:#dfbbbb'>Synonymous</span> with **Importance** -- - <span style='color:#c8acf4'>Asks</span> *how important is this and to whom?* -- - <p style="color:#e87965;">Example</p> - *Question*: Is the SNA course necessary for the sustainability of qualified methodologists in the United States? - *Outcome*: If so, it probably has significance. Otherwise likely not --- # One Caveat! - Most evaluation questions are not so black and white -- - They include some parts of merit, worth, and significance which are often used as clues --- # An evaluation is generally… - based on values (often of the person or people paying you) -- - erratic (both in practice and work life) -- - not well defined (and cannot be since you're dealing with people!) -- - political --- # Private evaluation work is… <table> <thead> <tr> <th style="text-align:left;"> positive in that </th> <th style="text-align:left;"> good for people who </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> it can be extremely lucrative </td> <td style="text-align:left;"> are able to work with a diverse set of people </td> </tr> <tr> <td style="text-align:left;"> you can pick and choose what you wish to study and like working solo </td> <td style="text-align:left;"> are ethical </td> </tr> <tr> <td style="text-align:left;"> you may get to travel </td> <td style="text-align:left;"> can adjust their writing and comminication based on type of audience </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:left;"> have a strong methodological background </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:left;"> enjoy diversity in content </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:left;"> leave biases at the door </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:left;"> love and thrive in uncertainty </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:left;"> work well in a specified timeframe </td> </tr> </tbody> </table> --- # Public evaluation work is… <table> <thead> <tr> <th style="text-align:left;"> positive in that </th> <th style="text-align:left;"> good for people who </th> </tr> </thead> <tbody> <tr> <td style="text-align:left;"> you get to have a standard 40-hour workweek </td> <td style="text-align:left;"> are able to work with the same people everyday </td> </tr> <tr> <td style="text-align:left;"> you do not have to compete for studies and like working solo </td> <td style="text-align:left;"> are ethical </td> </tr> <tr> <td style="text-align:left;"> you often get to stay within your home region </td> <td style="text-align:left;"> can write and communicate well to limited audiences </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:left;"> have strengths in certain methodological areas </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:left;"> enjoy homogeneity in content </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:left;"> understand that personal biases are part of the job </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:left;"> love and thrive in certainty </td> </tr> <tr> <td style="text-align:left;"> </td> <td style="text-align:left;"> work well in a longer timeframe </td> </tr> </tbody> </table> --- # Primary Purposes -- - **Formative**: Conducted with the intent to improve -- - **Summative**: Conducted with the intent to inform decision making and/or determine judgement --- # Common Types - **Needs Assessment**: <span style='color:#c8acf4'>Asks</span> *what are the needs of stakeholders and/or sponsors?* -- - **Process**: <span style='color:#c8acf4'>Asks</span> *is a program doing what it says its doing?* -- - **Outcomes**: <span style='color:#c8acf4'>Asks</span> *what is degree to which the program is having an effect on the target population’s behaviors?* -- - **Impact**: <span style='color:#c8acf4'>Asks</span> *what is the degree the degree to which the program meets its goal(s)?* --- # The Evaluator's Roaddmap: A Logic Model - A logic model is a graphic depiction akin to a road map that presents the shared relationships among the resources, activities, outputs, outcomes, and impact for your program. It is intended to depict the relationship between your program’s activities and its intended effects (CDC, 2018) -- - You can display change within a program by filling one out before an evaluation commences and after its conclusion, respectively --- # Linear example  .footnote[Many other variants can be found at repositories like those at the [University of Wisconsin-Madison Division of Extension](https://fyi.extension.wisc.edu/programdevelopment/logic-models/)] --- # Logic Model Terms  --- # Example: Logic Model  .footnote[<span style="color:#5ec7e1;">Note:</span> This is a subset of a much larger model] --- # What Evaluation is Like .center2[  ] --- # References + Centers for Disease Control and Prevention, Program Performance and Evaluation Office. (2019, May 7). Logic Models. [https://www.cdc.gov/eval/logicmodels/index.htm](https://www.cdc.gov/eval/logicmodels/index.htm) + Davidson, E. J. (2005). *Evaluation methodology basics: The nuts and bolts of sound evaluation*. Sage. + Scriven, M. (1991). *Evaluation thesaurus* (4th ed.). Sage. --- # Acknowledgement Cartoons were created and are owned by Chris Lysy. To see more of his work including those included in this slideshow, please head over to [So what is evaluation anyway?](https://freshspectrum.com/what-is-evaluation-anyway/) --- # That's Just the Tip! If you have an interest in evaluation, questions and/or comments, please send them along to [Abhik.Roy@mail.wvu.edu](mailto:Abhik.Roy@mail.wvu.edu) You may also stop by Allen Hall 504O if you prefer a face-to-face chat.