Center for Culturally Responsive Evaluation and Assessment

Foundations of Culturally Responsive Evaluation
Tuesday Full Day
     

Presenters:

• Rodney Hopson, PhD., Professor University of Illinois Urbana-Champaign
• Karen Kirkhart, PhD., Professor, Syracuse University

Workshop Description: This workshop addresses theoretical foundations of Culturally Responsive Evaluation (CRE) and the strategies that operationalize it in evaluation practice. It is delivered as a conversation between presenters and among participants. Following opening introductions, we will set the context with a reflection on the relevance of diversity, equity and intersectionality in the current moment of our country and where the evaluation profession sits within that. Against this backdrop, we highlight the history of CRE’s development and identify key theoretical elements.

The workshop then transitions from theory to practice in three segments. The first segment pairs analysis of evaluation contexts with reflections on one’s own cultural location as an evaluator. This prepares us for the second segment, which considers methods that are culturally congruent with their contexts of practice, noting potential strengths and limitations of each. CRE values the return of benefit to the community, and the third segment examines both methods and issues in communicating findings. We pair examples from the literature with your own examples as participants to connect workshop content with your contexts, interests, and concerns. In closing the workshop, we will return to fundamental issues such as grounding CRE in social justice and how this location poses important metaevaluation questions that connect to both ethics and validity.

Presenters Biographies:

Rodney K. Hopson is Professor, Department of Educational Psychology, College of Education, University of Illinois – Urbana Champaign. 

He received his  Ph.D. from the Curry School of Education, University of Virginia and has done post-doctoral/sabbatical studies in the Faculty of Education, University of Namibia, the Johns Hopkins Bloomberg School of Public Health and Centre of African Studies, Cambridge University. He served as President of the American Evaluation   Association in 2012.
Dr. Hopson’s research interests lie in social politics and policies, foundations of education, sociolinguistics, ethnography, and evaluation. His work raises questions that 1) analyze and address the differential impact of education and schooling on marginalized and underrepresented groups in diverse global nation states and 2) seek solutions to social and educational conditions in the form of alternative paradigms, epistemologies, and methods for the way the oppressed and marginalized succeed and thrive despite circumstances and opportunities that suggest otherwise.

Karen E. Kirkhart is Professor Emerita and Director of the Baccalaureate Program, School of Social Work, David B. Falk College of Sport and Human Dynamics, Syracuse University. Dr. Kirkhart is also an affiliated faculty member of the Center for Culturally Responsive Evaluation and Assessment (CREA) at the University of Illinois, Urbana-Champaign

She has been active in the evaluation profession for over thirty-five years. She holds a Ph.D. in Social Work and Psychology from The University of Michigan, Dr. Kirkhart served as President of the American Evaluation Association in 1994. Her leadership has been recognized with the Robert Ingle Award for Outstanding Service to the AEA. Dr. Kirkhart’s scholarly contributions to multicultural validity, cultural competence in evaluation, and evaluation influence have been recognized by the AEA with the Paul F. Lazarsfeld Award for Outstanding Contribution to Evaluation Theory.

Utilization of a Racial Equity Lens to help Guide Strategic Engagement and Evaluation
Tuesday Full Day           

Presenters: Paul Elam, Ph.D., Chief Strategy Officer, MPHI
                   LaShaune Johnson, Ph.D., Assistant Professor, Creighton University
                   Mindelyn Anderson, Ph.D., Program Director, American University
                   Willard Walker, Public Policy Associates, Inc.

Workshop Description: The field of evaluation is being challenged to move from the traditional role of evaluation, and its perceived role of objectivity, to a process that considers who is being evaluated and who is conducting the evaluation.  Over the past three years, Public Policy Associates, Inc. (PPA) has worked to develop useful frameworks, tools, and approaches that evaluators could consider to focus on the ways that race and culture might influence an evaluation process; this has resulted in the development of a framework for conducting evaluation using a racial equity lens.

This workshop focuses on the practical use of a racial equity lens when conducting evaluation.  The framework argues that culture and race are important considerations when conducting an evaluation because we believe that there are both critical and substantive nuances that are often missed, ignored, and/or misinterpreted when an evaluator is not aware of the culture of those being evaluated.

Participants will be provided with a Template for Analyzing Programs through a Culturally Responsive and Racial Equity Lens, designed to focus deliberately on an evaluation process that takes race, culture, equity, and community context into consideration.  

Presenters will also share a “How-to Process” focused on the cultural competencies of individuals conducting evaluations, how such competencies might be improved, and strategies for doing so.  This “How-to Process” is the result of thinking around developing a self-assessment instrument for evaluators, is based primarily on the cultural-proficiencies literature, and relates specifically to components of the template.

Participants will have the opportunity to engage in small-group exercises to apply the concepts contained in the template to real world evaluation processes.  Based on these experiences, participants will gain practical knowledge on the use of the lens.

• Share strategies and tools to intentionally include a culturally responsive and racial equity lens in strategic program evaluation.

Presenters' Biographies:

Willard Walker, Affiliated Consultant 

He has extensive experience working with workforce development, state policy, school-to-work, and race and diversity issues.  Mr. Walker has provided training on issues of inclusion, equity, and diversity for the Lansing Board of Water & Light; coordinated an Employer Conference on Workplace Diversity in the Lansing area; and assisted in a crime analysis evaluation for the Michigan Department of Human Services, Bureau of Juvenile Justice.  Mr. Walker led the work of developing the Template for Analyzing Programs through a Culturally Responsive and Racial Equity Lens in his work for the W.K. Kellogg Foundation Michigan Team Evaluation.
He has worked with clients to improve recruitment and retention of African-American teachers, including development of a coordinated recruitment planning process for urban districts.  He conducted research to address the plight of young males of color in a project initiated through the Joint Center for Political and Economic Studies.

LaShaune Johnson, Ph.D., Assistant Professor, Creighton University

She is an experienced researcher of online learning among pre-health professional students, breast cancer disparities, adult and adolescent obesity, and pediatric health literacy among immigrant/refugee populations.  She is faculty in the Master of Public Health program, and in the Master of Medical Anthropology program.  She is currently the co-chair of the Metro African American Breast Cancer Task Force in Omaha, Nebraska, and is the co-director of the “In Search of a Medical Home”, a culturally sensitive, Muslim community-based educator project in Central Missouri, funded by the American Academy of Pediatrics and the Society for Community Research and Action.  She is one of the contributors of a recently published textbook, Social Capital and Community Well-Being, which is being used by the Serve Here Connecticut academic debt reduction program.  She is also one of the contributors to an upcoming edited volume on Black LGBT health issues.         
Dr. Johnson has employed novel community-based methods to support Omaha’s Adolescent Health Project developmental evaluation (participatory video) and another method (Photovoice) to investigate health services for obese patients in Connecticut and Nebraska.  For the Breast Cancer Task Force, she co-designed a peer educator/advocate program; this program is in its second year and is expanding to add training for patient navigators.  She is currently a member of the Building Healthy Futures Evaluation Advisory Board in Omaha.  She was a member of the inaugural Annie E. Casey Foundation LEEAD (Leaders in Equitable Evaluation and Diversity) program.   

Dr. Mindelyn Anderson currently serves as the Program Director of the Masters of Science in Measurement and Evaluation at American University and is the founder of Mirror Group LLC.

Most recently, she completed a Leaders in Equitable Evaluation and Diversity (LEEAD) Fellowship at the Annie E. Casey Foundation. Her previous higher education appointments include Honors Faculty in Residence and Assistant Professor of Sociology and African American Studies at Northeastern University and Marilyn Yarbrough Fellow at Kenyon College. Dr. Anderson has also held fellowships at American Institutes for Research and Mathematica Policy Research.
For 15 years, Dr. Anderson has researched social inequality and stratification, race and migration, education and social mobility. She has conducted evaluations with community-based, regional, national and international non-profit, for-profit, and educational organizations. Her utilization-focused, participatory evaluation practice is informed by Culturally-Responsive Evaluation and values diversity, equity, and inclusion as integral components of high quality evaluation. Her scholarship has been supported by sponsors including the National Science Foundation and National Endowment for the Humanities.

Narrative and Collaborative Inquiry in Culturally Responsive Evaluation: Exploring Critical Methods in Practice
Tuesday Half Day
8:00 AM-Noon

Presenters: Leah C. Neubauer, EdD, MA, Assistant Professor, Northwestern University
                   Andrea Guajardo, , MPH, Director of Community Engagement and Social Accountability, University of the Incarnate Word

Workshop Description: 

This workshop will focus on critical methodologies and methods in culturally responsive evaluation.  Participants will be introduced to the theoretical and methodological foundations of critical, narrative and collaborative inquiry and their applicability in responsive evaluation. The co-presenters will facilitate a whole-person learning environment by engaging participants in a variety of activities designed to enhance their observational skills. Following the foundational primer and interactive activities, the session will focus on applying four critical methodologies and corresponding methods to culturally responsive evaluation case studies in community health and education: narrative inquiry, , cooperative inquiry, and collaborative inquiry.  These case studies are situated in communities of color – representing racial and ethnic minority groups in Chicago and San Antonio (USA). The applied, analytical activities illustrate how knowledge and lived experiences of the oppressed or marginalized are validated when they become part of the evaluation process while explicitly recognizing multiple forms of knowledge that exists in communities of color. Aligned with the conference theme of intersectionality, the activities call attention socially-stratified intersectional understanding, action-oriented praxis, and justice-focused-policy. Individual experiences shared during this process create an collective consciousness that represents larger communities which occupy space at the margins of society and challenge dominant methodologies that rationalize inequity, dominance superiority.    Strategies for creating space for critical methodologies as a means of pursuing social and racial justice will be addressed.  Participants should come prepared to share their experiences, engage in dialogue, and contribute to individual and group learning, un-learning and skill-building.

Workshop Outline

8:00am-8:20am: Introductions and Overview

8:20am-8: Rationale: What’s Evaluation Theory and How Does It Relate Social Justice

8:45am-9:: Critical Inquiry and Methodologies Overview Critical Got To Do With It?

9:15am-9:30am: Break

9:30am-10:: Case Study and Application:  How Does This Look In Practice?

10:30am-11:: Peer Review and Critique:  Towards Unlearning, Relearning and New Learning

11:00am-11:15am: Break

11:15am-11:: Critical Inquiry Strategy Development:  What’s Next In Your Practice?

11:45am-12:00pm: Critical Self and Group Reflection

Presenters Biographies:

Andrea Guerrero-Guajardo (Cherokee/Tejano), Ph.D., MPH, is currently Director Community Engagement and Social Accountability at the University of the Incarnate Word School of Osteopathic Medicine.

Her approach to community-based medical education is firmed entrenched in the principles of community organizing and participatory decision making with the goal of developing local capacity and improved wellbeing. Her community-based work is focused on determinants of health and community responsive evaluation that addresses disparities caused by social, political, and economic inequities within historically oppressed and marginalized populations. Dr. Guerrero-Guajardo works collaboratively with stakeholders at the local, state, and national levels to develop prudent public health policy and promote population health. She is an experienced trainer in these topics and was recognized by the Obama White House for her expertise and contributions to best practices for coalition building and community collaboration. Her leadership roles within the American Evaluation Association (AEA) supported Multi-Ethnic Issues in Evaluation and Latino Responsive Evaluation Discourse, and she was selected to be a 2013 Minority Serving Institutions (MSI) fellow at AEA. Dr. Guerrero-Guajardo lives in San Antonio with her husband, Ben, and their four children. 

Culturally Relevant Evaluation and Research from a Quantitative Perspective
Tuesday Half Day
8:00 AM-Noon           

Presenter: Toks Fashola, PhD., Professor, American University

Workshop Description: This workshop addresses the importance of culturally relevant evaluation and assessment tools specifically related to STEM.  This workshop addresses the use of these tools in nurturing and expanding the pipeline of underrepresented minorities.  It does this by exploring constructs, theories, and methods for evaluating STEM knowledge acquisition both inside and outside the classroom in K-16 settings.

Presenter's Biography:

Indigenous Evaluation Caucus
Tuesday Half Day
1:00 PM - 5:00 PM           

Presenters: Katherine Tibbetts, PhD., Liliʻuokalani Trust
                   Fiona Cram, PhD., Katoa Ltd.
                   Joan LaFrance, EdD., Mekinak Consulting

Workshop Description: 

Evaluation in Indigenous contexts is gaining more visibility and broader recognition among Indigenous evaluators, other evaluators practicing in Indigenous communities, and--arguably most importantly--among key stakeholders in those communities. A key milestone for evaluation practitioners was the publication of the first issue of New Directions in Evaluation focused exclusively on Indigenous Evaluation. 

The co-editors of the Indigenous Evaluation volume will facilitate this session intended to strengthen and broaden our community of practice. Participants will engage in a modified Appreciative Inquiry process to describe the progress we've made in defining and advancing the practice and acceptance of indigenous evaluation, in crafting a shared vision for the and identifying steps that will help us achieve this vision. 

The results of the session will be documented for all participants, used as input for updates to the CREA strategic plan, and may be submitted for publication. 

Presenters' Biographies:

Katherine Tibbetts, PhD., Director of Evaluation for the Queen Lili'uokalani Trust 

As the director, she is responsible for building evaluation culture and capacity. Prior to her current she conducted educational program evaluation and research at Kamehameha Schools. Her role there included technical support to Kamehameha Schools program staff and collaborators for program monitoring and evaluation, research related to the well-being of Native Hawaiians, and innovations in culturally responsive research and evaluation. She is also currently working with the Center for Culturally Responsive Evaluation and Assessment affiliate in Hawai‘i and the articulation of a Hawaiian-culture-based evaluation framework as part of a larger group of evaluators who work in Native Hawaiian contexts. She has a in Educational Psychology from the University of Hawai‘i at Mānoa and has experience evaluation- and research-related courses. Kathy is a founding member of the American Evaluation Association’s Indigenous Peoples in Evaluation TIG and of the Hawaii-Pacific Evaluation Association (an AEA Affiliate), a former chair of the AEA Diversity Committee, and one of the authors of the AEA Public Statement on Cultural Competence in Evaluation. She also contributed to chapters on indigenous perspectives on assessment and evaluation in Indigenous Educational Models for Contemporary Practice: In our Mother’s Voice and in Na Lau Lama.  She led the team responsible for producing Ka Huaka‘i: 2014 Native Hawaiian Educational Assessment. Most recently, she served as co-editor of the Indigenous Evaluation issue of New Directions for Evaluation. She also served as the AEA representative on the Joint Committee for Standards in Educational Program Evaluation from 2010 through 2018.

Fiona Cram, Ph.D., Katoa Ltd

Fiona (Ngāti Pāhauwera) is the mother of one and has a from the University of Otago (Social and Developmental Psychology). She has over 20 years of Kaupapa Māori (by Māori, for Māori) research and evaluation experience with Māori and Iwi and communities, as well as with government agencies, district health boards, and philanthropic . A large portion of this work involves the use of mixed or multiple methods in the pursuit of , tribal sovereignty, and societal transformation. Her current evaluation work includes the evaluation of a service supporting young Māori mothers into sustainable and a 10-year longitudinal study of Māori and Pasifika student success. Fiona has recently completed and published papers on the findings from a longitudinal qualitative study of young Māori mothers (E Hine and authored chapters in the 2015 Oxford Handbook of Mixed and Multiple Methods Research   

Joan LaFrance, EdD., Owner of Mekinak Consulting

Mekinak Consulting is an evaluation service in Seattle, Washington specializing in educational program evaluation, research, and management studies. She is a member of the Turtle Mountain Band of Chippewa.  Mekinak Consulting has a long history of evaluation of programs in Tribal Colleges and Universities, tribal and Indigenous communities, and for non-profit organizations.  With support from the National Science Foundation (NSF) through a grant to the American Indian Higher Education she conducted the research and co-authored the book Indigenous Evaluation Framework: Telling Our Story in Our Place and Time. In addition to her ongoing work in American Indian tribal communities, she is working projects in the United States Affiliated Pacific Islands to assess culturally relevant science and mathematics programs.  She was a founding member of the Indigenous Peoples in Evaluation AEA Topical Interest and former Chair of the AEA Diversity Committee.  She has taught research and evaluation methods in graduate programs and has experience in municipal budgeting, program development and management, and curriculum development.   She has published several articles and chapters on Indigenous evaluation and, most recently, was a co-editor for the New Directions in Evaluation volume on Indigenous Evaluation.  Joan received her doctorate Harvard and a Master of Public Administration from the University of Washington.

Transformative Mixed Methods Designs in Evaluation for Social, Environmental and Economic Justice
Tuesday Half Day
1:00 PM - 5:00 PM           

Presenter: Donna M Mertens, Professor Emeritus, Gallaudet University

Workshop Description: Developments in the use of mixed methods have extended beyond the practice of combining surveys and focus groups. The sophistication of transformative mixed methods designs in will be explained and demonstrated through illustrative culturally responsive examples taken from diverse sectors and geographical regions. Transformative mixed methods designs will include applications for determining the effectiveness of interventions and conducting policy evaluation. Participants will have the opportunity to create transformative mixed methods designs using evaluation vignettes for each type of evaluation. The workshop will be divided into 4 sections. First, the concepts of transformation; social, economic, and environmental justice, and intersectionality will be explored in the context of culturally responsive evaluations. Second, the basic vocabulary of mixed methods design will be explained and transformative mixed methods design will be put into the context of evaluation work. Third, transformative mixed methods designs will be explained and demonstrated for the purpose of determining an intervention's effectiveness. The participants will be given an evaluation vignette and asked to develop a mixed methods design. Fourth, this process will be repeated for the purpose of using transformative mixed methods designs for policy evaluation.

The workshop will be divided into 4 sections.

1. the concepts of transformation; social, economic, and environmental justice, and intersectionality will be explored in the context of culturally responsive evaluations.
2. the basic vocabulary of mixed methods design will be explained and transformative mixed methods design will be put into the context of evaluation work
3. transformative mixed methods designs will be explained and demonstrated evaluation work for the purpose of determining an intervention's effectiveness. The participants will be given an evaluation vignette and asked to develop a mixed methods design.
4. this process will be repeated for the purpose of using mixed        

Presenter's Biography: