Southern California Evaluation Association

SCEA at AEA 2011: Presentations on Evaluation Theory

socaleval — Tue, 01 Nov 2011 05:40:11 +0000

As evaluation practitioners, scholars, and students, we are often greeted by questions about the relationship between evaluation theory and practice. Our experiences suggest that development and advancement of these areas inevitably go hand-in-hand. In today’s post, we draw your attention to paper presentations about evaluation theory specifically. Those interested in this particular area might find the following presentations by SCEA members and their colleagues about theory, theory-building, validity, and values/valuing relevant and interesting.

For presentation time and location, please see this year’s AEA Conference Program.

Values & Validity
James Griffith
ABSTRACT: A panel at last year’s conference featured contemporary responses to Ernie House’s classic Evaluating with Validity. House’s original discussion and last year’s panel focused on what priority should be emphasized. This paper extends that discussion by analyzing the values inherent in validity and in preferences for truth, beauty, and justice. What values are we accepting, rejecting, or balancing when we choose between truth, justice, and beauty? Similarly, what values are inherent in refusing to make the choice? In our analysis, we draw not only from House’s writings and other evaluation theory classics, but also from more contemporary ideas, such as multicultural validity, cultural competence, and praxis. An interesting feature of this discussion is that unlike other theoretical discussions about evaluation, theory and practice are intertwined here. This discussion centers on the collision of theory with the constraints of reality, hence the evaluator’s concern about choosing when values compete.

Mixed Model Theory Development: Building a Theory-informed and Practice-informed Model of Evaluation
Michael Harnar
ABSTRACT: Understanding the application of evaluation approaches helps inform our understanding of evaluation theories. Exploring this interaction of practice and theory is at the heart of research on evaluation. Because evaluations are much like the programs we evaluate, where our activities are expected to lead to outcomes, using a program theory-driven evaluation technique of modeling evaluation practice should increase our understanding of our approaches. This presentation extends previous evaluation theory modeling by describing a process that engages evaluators in modeling their practice with theoretically derived variables. In this method, evaluators model their preferred practice in an online modeling software and the produced models are combined to create one representative model that evaluators review and comment on, improving the model’s reflection of practice. The final product is a theory- and practice-informed picture that might then be analyzed and tested for comprehensiveness and consistency in practice.

Theory Building Through Praxis Discourse: A Theory- And Practice-Informed Model Of A Transformative Participatory Approach To Evaluation
Michael Harnar
ABSTRACT: The discipline of evaluation is built upon its descriptions, its language, and its approaches towards practice. The development of one such approach – Participatory Evaluation – has focused almost entirely on a practical model, leaving a transformative model mostly neglected. Transformative Participatory Evaluation (T-PE) practitioners find little theoretical or practical guidance in the evaluation literature. This paper redresses this discrepancy by presenting a theory- and practice-informed model of T-PE that will help further discipline development by providing practitioners a model reflecting their work. LeBaron Wallace, Hansen, and Alkin (2009) used qualitative literature analysis to develop logic models reflecting Practical Participatory Evaluation and Transformative Evaluation but overlooked T-PE. The present research extends that work by developing a model of T-PE using both quantitative and qualitative methods that inform one another. The product is a theory- and practice-informed model of a transformative participatory approach to evaluation practice.

The Evaluator’s Role in Valuing: Who and with Whom
Anne Vo
Christina Christie
ABSTRACT: Evaluation scholars and practitioners have dedicated much energy and effort to shaping and defining the program evaluation profession. However, careful examination of the program evaluation literature turns up only a few resources that describe and operationalize value judgments, the ways in which they are reached, and who is involved in this aspect of the evaluation process. We argue in this paper that the valuing act may be perceived in many different ways and consider the multiple theoretic perspectives that govern an evaluator’s behavior. Based on this analysis, we develop a typology of evaluator valuing roles and suggest that value judgments are typically reached by stakeholders alone, stakeholders and evaluators in consort with each other, or by evaluators only. This heuristic helps us to gain a more explicit understanding of the valuing act and process as it occurs in the context of an evaluation.

SCEA at AEA 2011: Presentations on Network Analysis

socaleval — Mon, 31 Oct 2011 20:49:48 +0000

The Evaluation field’s growth over the years have led to the development of new tools for practice in some cases and integration of existing tools from various disciplines in other cases. Social network analysis/network mapping is an example of an analytic method traditionally used in Sociology and Anthropology to study interaction within social groups that has found a home within Evaluation.

Below is information about paper presentations that SCEA members and their colleagues will be giving on the uses of social network analysis to better understand the nature of collaboration in areas such as social welfare and systems thinking at this year’s AEA Conference.

For presentation time and location, please see this year’s AEA Conference Program.

Change Over Time in Networks of Care Providers in Monterey County and Interpretations of Systems Change
David Dobrowski
ABSTRACT: In an evaluation of First 5 services to families and children in Monterey County, an inter-organizational network mapping procedure was employed at two time periods to identify changes in levels of coordination and collaboration among service providers who receive funds from First 5 Monterey County. Across a one year period, there was an increase in the number of interactions between funded partner agencies, with the increase in ties occurring at the Networking and Coordination levels, but with slightly fewer ties at the Collaboration level. The results raise questions about how best to engage participating organizations and other stakeholders in increasing levels of collaboration, when is the right time to measure changes in systems, and whether an increase in collaboration is a necessary or sufficient condition to determine improvement in systems change outcomes.

Using Network Analysis to Understand and Improve Collaboration Among Michigan’s Integrated Improvement Initiatives and Center for Educational Networking
Jan Gaylord-Vanslyke
ABSTRACT: As part of a collaborative mixed-method evaluation of Michigan’s Integrated Improvement Initiatives and Center for Educational Networking (MI3-CEN), network analysis was used to characterize current patterns of collaboration in shared function areas among key network partners, and to identify opportunities for strategic development. Evaluation goals and methods were collaboratively identified by MI3-CEN network staff and evaluators. Panel participants will discuss this collaborative process and results of the network analysis from both an evaluator and user perspective.

Social Network Analysis as a Tool for Engaging Communities in the Assessment of Inter-organizational Collaboration
Joelle Greene
ABSTRACT: The success of place-based initiatives is largely contingent upon the ability of communities, leaders and organizations to collaborate. From the assessment of needs, to resource allocations and coordination of services, organizations are required to work towards mutual goals and outcomes. In this community assessment an organizational survey was conducted, in part, to help determine current levels of collaboration among organizations and to help the community partnership envision and set future goals related to the nature and degree of collaboration in their community. In the presentation we will provide examples of Social Network Analysis (SNA) maps used to interact with the community partnership, highlight how the data was used in the planning process, and how SNA will be used longitudinally to map changes in levels of collaboration over time in a place-based comprehensive community initiative.

Social Network Analysis as a Tool for Engaging Communities in the Assessment of Inter-Organizational Collaboration
Sae Lee
ABSTRACT: Inter-organizational network analysis is being used in several evaluations of First 5 county programs in California to measure changes in systems of care towards greater collaboration among agencies receiving First 5 funds. Another use of the methodology is presented in this paper, involving an evaluation of a participatory community assessment process as part of the comprehensive placed-based initiative developed by First 5 Los Angeles. To assess the current state of collaboration among community agencies within each of First 5 LA’s 14 communities, an inter-organizational survey was conducted which included Frey’s Levels of Collaboration Scale. The network analysis findings will be discussed in terms of how community partnerships can use the findings to understand the strengths and weaknesses of their relationships with other agencies in the target communities, and how multiple stakeholders can be engaged in bolstering community-based networks as part of a larger community building process.

From Survey Data to Network Mapping and Beyond: Describing Inter-Organizational Coordination and Collaboration Networks
Sae Lee
ABSTRACT: This session will demonstrate one method for measuring inter-organizational networks using a modified version of the Levels of Collaboration Scale as part of a Funded Provider survey. Based on our experiences evaluating First 5 programs in several California counties, we will demonstrate how survey data from agency respondents are transformed into two-dimensional inter-agency network maps using NetDraw, and then how features of the network (strength of ties, closeness, density) can be distilled from these network maps and analyzed statistically. We will show how to use the maps to describe the nature of collaboration between agencies, how to expand the survey to a broader network of agencies, and how to employ this procedure to measure change over time. Caveats and limitations of this method and how to interpret and communicate the maps to stakeholders and agencies to support program monitoring and improvement will also be discussed.

SCEA Week on the aea365 Blog

socaleval — Tue, 25 Oct 2011 00:53:58 +0000

In June 2011, we shared a call with the SCEA community for contributions to the American Evaluation Association’s aea365 Blog for SCEA Week. Members who responded to the call were asked to share a “lesson-learned” about conducting evaluations in their particular sector as it is situated in the Southern California context. They were also asked to highlight a “tip” or “rad resource” that will help conference attendees make the most of their visit to Anaheim.

With the help of contributing SCEA members, we are pleased to share that we have been able to kick off SCEA Week this past Sunday, October 23^rd, with our post on “The Landscape of Evaluation Practice in Greater Los Angeles.” In it we provided an overview of what subscribers to the blog can expect in the upcoming week, which included a list of activities in which conference attendees might engage during their visit to Southern California. We also previewed the week by highlighting posts that SCEA members contributed:

Bonnie Richards on Valuing in the Social Services
Leslie Fierro & Deanna Rossi on Evaluating State Asthma Programs
Bill Shennum & Kate LaVelle on the Multiple Uses of Multiple Data Sources
Connie Hoskins on Managing a Project Remotely
Kara Crohn on the AEA Annual Conference Fieldtrip and Using Multiple Methods to Check Your Assumptions

The aea365 Blog is a great resource for evaluation practitioners, scholars, and students alike. If you are not already following the blog, please consider doing so by navigating to the site through our “Links” page. Subscribing to the blog is also the easiest way to access the site’s updates, which includes the posts listed above.

While you are there, please also consider following the SCEA website to receive the Association’s e-mail updates by entering your e-mail address in the box under the “Stay Connected with SCEA” banner on the right side of the page.*

*Note: The Southern California Evaluation Association does not share your contact information with outside parties.

SCEA at AEA 2011: Roundtables

socaleval — Fri, 21 Oct 2011 05:28:53 +0000

Earlier this week we highlighted poster presentations that SCEA members will be giving at this year’s AEA Meeting. In today’s post, we want to draw your attention to another venue in which presenters and their audience can closely engage with each other — roundtable sessions.

Roundtables are held everyday during the conference and this year’s AEA Meeting offers approximately 45 papers shared using the roundtable format per day. This kind of session enables presenters to interact with a smaller audience about a focused topic. As with papers presented during concurrent sessions, topics covered during roundtables vary greatly, including issues related to logic modeling, how to foster stakeholder engagement, and challenges encountered when trying to consider divergent values in evaluation.

Roundtable presenters share work done in both research- and practice-based settings across sectors not limited to education, non-profit organizations, public health, and social welfare.

The following SCEA members and their colleagues, in particular, will be sharing some of their current work at a roundtable. For session location and time, please see this year’s AEA Conference Program.

Improving Evaluation Practice With Youth: A Checklist for Developmentally Sensitive Program Evaluation
Tiffany Berry
ABSTRACT: The evaluation of programs that serve youth can be complex given the multifaceted nature of child and adolescent development. These evaluations require a developmentally sensitive approach, which includes thoughtful consideration of the characteristics this unique population brings to the evaluation. In this Round Table we will describe a developmental sensitivity checklist framed within the Centers for Disease Control’s six-phase framework. Our goal is to document the pertinent considerations for typical as well as atypical youth across each stage of the evaluation process. This checklist was developed through an extensive literature review of best practices in applied research with youth and was validated by an expert panel of developmental psychologists and veteran youth-program evaluators. In this Round Table we will introduce our tool, receive feedback to refine the tool, as well as engage evaluators in a discussion about how we can continually improve the quality of program evaluations targeting youth.

Using Values to Focus the Evaluation of a Dynamic, Multi-Stakeholder Prevention System
Kristen Donovan, Shanelle Boyle
ABSTRACT: Given a finite set of resources, it can be challenging to design and implement a quality evaluation of a dynamic, multi-stakeholder system encompassing regional, initiative, and countywide tiers. Valuing plays a critical role in determining which components of an overall system to focus on regarding evaluation design, methodology, and utilization. This roundtable will discuss challenges associated with assigning value in a complex prevention system located across San Diego County and consisting of multiple programs. Presenters will share their experiences, strategies, and lessons learned about how valuing was carried out during the planning and implementation phases of a countywide multi-initiative alcohol and other drug (AOD) prevention system evaluation project. Attendees will also have the opportunity to share their challenges, experiences, and effective strategies for prioritizing and incorporating values into their evaluations of dynamic, multi-stakeholder systems.

Using Self-Report to Measure Implementation Fidelity
Kate Clavijo
ABSTRACT: Self-report measures have been heavily used in many program evaluations to assess fidelity of implementation. In this rountable session we will discuss a range of strategies and questions from online surveys designed to measure implementation fidelity of a professional development initiative. These data are an essential step to ensure the professional development is being used efficiently and effectively. But the big question, is how valid is the self-report data? We are interested in facilitating discussion about how to best measure implementation fidelity of teacher professional development, without directly observing classroom practice. We will discuss lessons learned from our experiences and soliciting suggestions from other evaluators. As more and more evaluation data is being collected on-line, what techniques have others found of value to accurately measure practice from participant self-reports?

Evaluation to the Rescue: Combining Internal and External Evaluation Policies and Practices to Sustain Peripheral Educational Programs
Ian Martin
ABSTRACT: In times of economic stress, school systems cut their budgets while aiming to maintain core academic services. Looking to eliminate services not deemed basic, districts put peripheral programs on budgetary chopping blocks. Thus, school social workers, counselors, nurses and librarians often find their programs and positions in jeopardy. Drawing on our work in school districts, we argue that those programs engaging in internal and external evaluation practices fare much better than programs or services without evaluation. They survive because they are proactive; that is, they use evaluation information to advocate their value to the district and to make on-going improvements, thus, increasing perceived effectiveness. This roundtable focuses on the role of evaluation within school counseling programs, presenting real cases that use combined internal and external evaluation. We highlight conceptual tools and evaluation capacity building activities that better position peripheral programs within state departments of education and local school districts.

Holding up a Mirror: Investing in an Internal Implementation Evaluation as a Vehicle to Document a Funder’s Process and Inform Decision-Making
Artineh Samkian
ABSTRACT: This presentation documents a funder’s process in developing a research and evaluation agenda as we move from a project-based funding strategy to a primarily place-based one. Special attention is given to an implementation evaluation, which will systematically document the process of carrying out our strategic plan so as to inform decisions. Consistent with the growing literature on utilization-focused evaluation (Patton, 1997) and the expressed need to ensure stakeholder buy-in and engagement (Preskill & Caracelli, 1997; Taut & Alkin, 2003; Johnson et al., 2009) it is critical to attend to the context in which the evaluation is taking place and to attend to the role of the evaluator. As a semi-internal, semi-external evaluation that reflects on and scrutinizes our own process as a funder, this case study poses an interesting opportunity to examine how the use of findings is affected by both the context and the evaluators responsible for providing feedback.

SCEA at AEA 2011: Poster Session

socaleval — Tue, 18 Oct 2011 00:50:07 +0000

One of our favorite events at the Annual AEA Meeting is the Poster Session, which takes place concurrently with the conference’s Opening Reception, on Wednesday evening (November 2nd).

The Poster Session is a great opportunity to access work that is currently being done in research- and practice-oriented settings in a less formal atmosphere. It is also a great chance to network with presenters along with other fellow conference attendees.

With over 170 posters being presented this year, across a number of different sectors, you are bound to find at least one poster that will be interesting and relevant to your own work.

In today’s post, we highlight posters being presented by SCEA members:

Poster 16: Incorporating Diverse Stakeholder Values Into an Enhanced Evaluation Design of an Individual Prevention Service Process
Shanelle Boyle
ABSTRACT: As evaluators with a restricted budget, it is challenging to design an initial evaluation that addresses diverse questions posed by multiple stakeholders who value different program components and outcomes. This was the case when EVALCORP partnered with the Riverside County Department of Mental Health – Substance Abuse Program to design and conduct the first formal evaluation of the Brief Risk Reduction Interview and Intervention Model (BRRIIM), an innovative prevention process that screens and educates individuals at high risk for alcohol and other drug problems. The program’s context and original budget constraints made it difficult to develop an evaluation design that met the varying needs and expectations of stakeholders. This presentation will provide an overview of the strategies EVALCORP used to resolve contextual issues and help the program secure additional funding for an enhanced evaluation plan that incorporates a more rigorous design and broader range of stakeholder values.

Poster 39: Evaluating an HIV and Substance Abuse Intervention for Homeless Youth
Mona Desai
ABSTRACT: This presentation describes a mixed method evaluation of a youth development approach to substance abuse and HIV prevention among homeless youth ages 18-24. Homeless youth at drop-in centers, shelters and transitional living programs were recruited to participate in the 9 session weekly curriculum that focused on helping them envision a future for themselves and understand how their behavior know can impact that future. Baseline, exit and 3 month follow-up surveys were completed with participants to determine changes in sexual risk and substance use behavior over time. In addition focus groups and interviews were done with participants to further understand the impact of the intervention as well as what changes needed to be made. We will discuss the statistically significant changes from baseline to follow-up and share youth comments about the impact the intervention has had on them. The session will also discuss tracking strategies for highly mobile homeless youth.

Poster 60: Character Assessment for School Age Children: A formative evaluation tool
Leslie Grier
ABSTRACT: The purpose of this research was to examine the psychometric properties of a character assessment scale (i.e., the Character Assessment for School Age Children – CASAC) based on the six pillars of character (Josephson Institute, 2009). Many youth development and character education programs utilize some if not all of the pillars of character explicitly or implicitly (e.g. Prestwich, 2004); yet there are few character assessments that integrate these components in a comprehensive way. Furthermore, as many of these programs have been utilized to augment academic achievement, there is increasing emphasis on the importance of formative evaluation (e.g., Berkowtiz & Bier, 2004; Zins, Bloodworth, Weissberg, & Walberg, 2004). Hence a brief assessment of character that was also predictive of academic achievement would be useful as a formative evaluation tool. Exploratory factor analyses resulted in a two-dimensional scale. These dimensions demonstrated internal reliability and criterion validity.

Poster 99: First-Year Evaluation for Community Connections, Community Senior Services, Claremont, California
Anna Fagergren
ABSTRACT: Research suggests increasing demands on existing transportation programs intended for older adults, nonetheless, formal evaluation reports are scarce. Responding to unmet transportation needs in the Pomona Valley area in Southern California, Community Senior Services (CSS) initiated a volunteer-driver program, ‘Community Connections’, in November 2010. Evaluators from Claremont Graduate University were enlisted to design and conduct a process evaluation. The evaluators, in collaboration with CSS, aim to record seniors’ needs for the program, because no existing literature has examined the need in Pomona Valley. In addition, ridership profiles and program use are going to be analyzed across the eight months of program operation. And finally, the growth of stakeholders’ evaluation capacity will be documented. Working with the evaluators, stakeholders will deliberate on the measurements and learn strategic evaluative thinking and practice. Final report is expected to be published in October.

Poster 106: The Role of Principals in Afterschool Program Evaluations: Expanding Values and Complexity
Tiffany Berry
ABSTRACT: A review of afterschool program literature gives limited insight into the links between the regular school day and afterschool programs. Education research suggests stronger relationships between principals and teachers lead to better academic outcomes for students, yet rarely have evaluators investigated similar links between the regular school day and the afterschool program. The purpose of this poster is to illustrate the values espoused by an underrepresented stakeholder: school principals. Using data from an on-going evaluation of After-School All-Stars, Los Angeles, a program serving disadvantaged middle and high school students, we will present results from 10 principal interviews to illustrate (1) principals’ values in relation to after school programs, (2) how principals’ values changed the nature of our evaluation work, and (3) the inherent complexity involved with incorporating multiple stakeholder groups.

For meeting location and time, please see this year’s AEA Conference Program.

SCEA at AEA 2011: Topical Interest Groups

socaleval — Tue, 11 Oct 2011 11:31:18 +0000

Since our involvement in SCEA, we have often been asked – particularly by graduate students and newly minted evaluators – how to become active in a professional organization as large as AEA. In our experience, one of the best ways to get involved is by participating in one of the organization’s 47 Topical Interest Groups (TIGs). AEA TIGs are subgroups within the association that consist of members who share common interests.

Some SCEA members are also members of AEA and have been actively contributing to AEA TIGs, including the TIG on:

College Access Programs

TIG Leader: Rita O’Sullivan

Quantitative Methods: Theory and Design

TIG Leader: Dale Berger

Research on Evaluation

TIG Leaders: Tarek Azzam & Tina Christie

Theories of Evaluation

TIG Leader: James Griffith

Consider attending a TIG Business Meeting this year to learn about what is happening within each TIG and how you might be able to get involved.

For meeting location and time, please see this year’s AEA Conference Program.

SCEA | Official Site Re-Launch

socaleval — Tue, 11 Oct 2011 09:26:43 +0000

Greetings, SCEA Members!

We are Anne Vo (University of California, Los Angeles) and John LaVelle (Claremont Graduate University) and we are excited to re-launch the Southern California Evaluation Association’s website with a series of posts that spotlight the work that members will be sharing at this year’s American Evaluation Association (AEA) Meeting in Anaheim!

Starting today, and with your support and permission, we will be posting members’ names along with titles and abstracts of their paper/session in an effort to help fellow SCEA members become aware of not only each other’s work, but also SCEA’s involvement in the conference. This information will be organized around themes not limited to:

Topical Interest Groups (TIGs)
Educational Evaluation
Evaluation Capacity Building
Evaluation in the Health Sector
Fidelity of Implementation
Inquiry Methods in Evaluation

…and many more!

We will also be sharing photos and highlights from events that have taken place during the year. These events include the UCLA Program Evaluation Symposium, the Claremont Graduate University’s Stauffer Symposium, and others!

Please stay tuned and follow the new SCEA website to receive the latest updates!

American Evaluation Association Annual Meeting – November 2-5, 2011

socaleval — Tue, 04 Oct 2011 09:10:23 +0000

Join SCEA at this year’s AEA Conference in Anaheim! We look forward to seeing you!