Holistic Review Toolkit

II: Why AMIGA?

Across the U.S. higher education administrators and faculty are leading initiatives that focus on increasing faculty diversity. At the same time, undergraduate student populations are becoming increasingly diverse. Although graduate study is a significant part of the pipeline to becoming a faculty member, graduate education and admissions specifically, have received far less attention.

In 2020 social inequality and access to resources manifested nationally with the realization that people of color are more likely to catch COVID-19. Moreover, racial injustice exposed the extent that black and brown people experience policing and other forms of institutional racism. Students, staff and faculty alike have pondered, "What can one do to foster social justice in academia?"

Demographic forecasts detail a significantly changing US population by 2050. In California, public colleges and universities are already enrolling substantial numbers of college students who are historically underrepresented in higher education. The Alliance for Multi-campus, Inclusive Graduate Admissions (AMIGA) project is focused on supporting holistic graduate admissions to build a faculty pipeline and increase access to graduate education.

To that end, the AMIGA project values equitable and inclusive holistic review methods for graduate admissions that potentially increases the number of students in graduate school who are first generation college, from underserved communities or are historically underrepresented. It is only with attention to equitable and inclusive methods that institutions can achieve their goals of increasing faculty diversity and preparing the brightest minds to tackle society’s most intractable problems.

III. The University of California Commitment to Diversity

The mission of the University of California (UC) is to serve society as a center of higher learning, providing long-term societal benefits through transmitting advanced knowledge, discovering new knowledge, and functioning as an active working repository of organized knowledge. The 2006 UC Statement on Diversity, recognizes that, “[t]he diversity of the people of California has been the source of innovative ideas and creative accomplishments throughout the state’s history into the present” and that in order to serve its core mission, UC, “must seek to achieve diversity among its student bodies and among its employees.” The UC Statement on Diversity defines diversity as “the variety of personal experiences, values, and worldviews that arise from differences of culture and circumstance. Such differences include race, ethnicity, gender, age, religion, language, abilities/disabilities, sexual orientation, gender identity, socioeconomic status, and geographic region, and more." Employing equitable and inclusive holistic review practices serves to remove barriers to the recruitment, retention, and advancement of talented students from historically underrepresented groups and support cohorts that are more likely to reflect the diversity of California and our changing U.S. population.

The UC campuses must comply with the California Constitution which prohibits consideration of race, ethnicity and gender in the public university. However, it is permissible in Graduate admissions processes to consider applicants’ potential to contribute to diversity and equal opportunity through their background, experience and potential for leadership in diversifying their field. The UC Guidelines For Addressing Race and Gender Equity in Academic Programs in Compliance with Proposition 209 provide a roadmap: "Departments can approach the selection process in terms of building a cohort that will enhance the breadth of interests, experiences, and perspectives in the department. Departments can consider a wide variety of indicators in the evaluation of candidates, including potential for leadership and significant life experiences, rather than admitting students solely on a narrow range of traditional indicators." A greater emphasis on applicant interests and experiences will lead to equitable and inclusive selection practices contributing not only to student success, but the academic excellence of student cohorts.

IV. Developing a Rubric

Equitable and inclusive review requires fair, clear and consistent evaluation across committee members. It can be accomplished broadly through faculty admission training and the use of a scoring template that admissions committee members develop and norm prior to application review. Discussing and finalizing selection criteria and scoring method/s among colleagues is critical to achieving consensus that supports validity and reliability. Consider the following issues to ensure the development of an equitable, inclusive and valid rubric for your graduate program.

• Discipline specific skills – In context, weigh whether an applicant's ability to take specific courses may be affected by other factors other than choice (e.g., a particular public institution may not offer Latin or Greek language study).

• Diversity – Discuss and consider how contribution/s to diversity will be valued and evaluated. Also, consider unique perspectives that students with diverse backgrounds contribute to the graduate experience.

• GPA – Contextualize the GPA by supporting evaluation with factors that may have been influential, such as a change of majors, personal circumstances or limited access to particular coursework.

• Recommendation letters – Consider the questions that faculty respond to for recommendation letters. Be cognizant that unfamiliar faculty, programs or institutions do not signal a less talented or prepared applicant.

• Research experience – Consider the type and depth of research experience in the context of the undergraduate institution, availability of graduate preparation programs, and necessity to work extensively to pay for college expenses.

• Personal background – Background to possibly consider includes contribution to diversity, obstacles overcome, first generation college, public college attendance, grad prep program participation and “distance traveled,” (the distance a person has traveled by "overcoming a lack of resources, family structure or support, and discrimination of any kind," Craig 2017).

• Standardized test scores – Consider the equitable use of scores, such as the GRE, GMAT and others. If temporarily or permanently suspended, how might corresponding skills be evaluated from the current application information?

V: Promising Practices for Holistic Review

1. Add comprehensive graduate application information to the department website.

   A successful holistic admission cycle begins with in-depth graduate program and application information on program websites that do not assume every prospective graduate student is knowledgeable about applying or preparing for graduate school. Prior to the admissions cycle, review the graduate program website for equitable and inclusive practices (e.g., clear expectations for writing sample, requirements for specific courses or particular level of math skills) and make changes accordingly.

2. Highlight diversity advocates.

   Identify and share on graduate program websites a contact person (preferably a faculty member) who champions graduate diversity for the program, in addition to providing contact information for other graduate diversity professionals. By so doing, a graduate program signals an internal commitment to a diverse graduate student body.

3. Review online application content.

   Review the electronic application to identify the information required from each applicant. Work with the Graduate Division to customize additional fields for a particular graduate program. Consider consulting UC Guidelines For Addressing Race and Gender Equity in Academic Programs in Compliance with Proposition 209.

4. Document existing practices.

   Consider the graduate admissions process that occurred in the preceding admissions cycle. Ask the previous year's graduate admissions chair to record and share the admissions process with the new admissions chair so that faculty will have a starting point for the present cycle.

5. Frame faculty discussions.

   Among program faculty and admissions committee members discuss the connection between graduate admission decisions, the future of the field and the academy as a whole. Research on the so-called “pipeline problem” may be generative. However, these discussions must follow governing guidelines and policies for a particular institution, e.g. in California, these discussions cannot include race/ethnicity, gender and national origin.

1. Assemble admissions committees early.

   In order for admissions committee members to participate in pre-admissions discussions and take advantage of faculty development opportunities, graduate program leadership should identify the new or continuing graduate admissions chairperson and admissions committee members by the start of the fall quarter or semester.

2. Diversify your admissions committee.

   Depending on the number of applications, consider involving more faculty, and possibly advanced graduate students, in application review. Advanced graduate students will acquire essential professional development skills through participation. Also, include faculty and graduate students from diverse backgrounds on the committee.

3. Identify a vice chair of admissions.

   If the continuing admission chair is in their final year of graduate admissions service, securing a vice chair to support and learn from the current admissions chair will facilitate a smooth and transition of established holistic review practices in the next admissions cycle.

4. Review admissions data.

   Admissions committees should review graduate program data for applications, enrollments and completions for previous years to understand past outcomes and trends and to set goals for the current admission cycle (e.g., defining admissions criteria, developing a rubric).

5. Review admissions resources.

   Review websites to identify broad and specific tools and information that support equity, inclusion and social justice in graduate admissions (e.g., AMIGA, the Inclusive Graduate Education Network, and the AAMC Diversity and Inclusion Tool Kit).

6. Offer holistic review training.

   Faculty may not know what holistic review entails. Strongly encourage admissions committee faculty to participate in holistic review training provided by the university or led by knowledgeable colleagues.

7. Utilize a rubric.

   With faculty colleagues, develop a rubric (scoring template) for the graduate admissions process. Identify key applicant preparation, interests, and experiences and determine how they will be evaluated (e.g., high, medium, low). They could include contribution to diversity, discipline specific skills, research experience, perseverance, coursework preparation, and intellectual curiosity. The admissions committee will apply the rubric to evaluate applicants' materials. Consider using more than one rubric for different parts of the admission process, e.g. a separate rubric for interviews.

1. Review commonly used terms.

   Discuss terms that are commonly used to understand an applicant’s background (e.g., diversity, first generation college, or distance traveled), to establish common understandings across faculty who may start with distinctly different reference bases.

2. Review application fields.

   Locating and triangulating applicant information (using multiple sources of application data to potentially support an applicant's background) both reduces review time and supports identifying key skills and backgrounds. Reviewers familiar with the application can, therefore, move quickly from one page of the application to another. Also, knowing where to find specific information supports equitable and inclusive practices by reviewing the full file (e.g., taking into consideration an applicant's first generation status or by understanding why an applicant had limited research experiences by reviewing the résumé that demonstrated extensive work experience to pay for college).

3. Focus on skills, not test scores.

   Given the temporary or permanent waiver of standardized test scores, identify the representative skills formerly associated with test scores and determine alternative information to evaluate those skills.

4. Meet to norm the review process.

   Prior to reviewing current applications, meet to confirm criteria among admissions and broader program faculty. Norm the review process by individually employing the developed rubric, guidelines for its use and using sample applications from previous years for practice. Gather to discuss individual scores, negotiate differences and finalize methods. When norming is used in admissions review, it results in a more consistent and transparent review process across multiple reviewers.

5. Meet after initial review.

   Plan for the committee to meet following individual scoring to discuss top applicants and to reach consensus. Committee members can share impressions, scores of individual applicants and practice the give and take that accompanies mutual decision making.

VI. Taking Stock

The following items assist both the admissions chair and committee members to form shared knowledge and procedures with which to implement holistic review. By taking stock of current admissions practices, the committee can then assess how holistic review methods can be tailored for implementing a successful admissions process that is both equitable and inclusive.

• In the previous cycle, were there written procedures and instructions that guided the admissions process?

• Were there any changes recommended to the admissions process that support equitable and inclusive practices? Outline them and think about the process of implementing those changes.

• What are the official and unofficial policies regarding the use of standardized test scores?

• Identify the level of admissions experience of each committee member. Is a committee member experienced, a newcomer or a mix of both? What admissions training has each committee member participated in?

• How many faculty members participated in last year’s application review?

• How is the workload distributed among members of the committee?

• Outline the process of how the committee decides who to admit. Are there specific predefined criteria the committee considers when admitting graduate students?

• What were the instructions given to the admissions committee previously? How were the admissions instructions disseminated?

• If you previously participated in the admission process, are there changes you would recommend be implemented?

• What are the criteria that are used to review applicant files? Do these address the competencies needed to prepare and succeed in your graduate program?

• Based upon the established selection criteria where will you look for this information within an application?

• Are you up to speed on current knowledge about graduate admissions, such as how to norm a scoring template/rubric?

• Review the admissions instructions and procedures shared by the chair. Are you comfortable with them or would you like additional information, such as implicit bias modules or background on equitable and inclusive graduate admission methods?

• Do the admission instructions sufficiently prepare committee members to review applications equitably? Consider how they may and may not prepare you. What else do you need?

VII. Faculty Development

Although it may be challenging to know where to begin when implementing an equitable and inclusive admissions process, extending one's basic knowledge is a good starting place.

CONTRIBUTIONS TO DIVERSITY

• Evaluating Contributions to Diversity

• UCOP Diversity Guidelines

Holistic Review

• Holistic Review: You're More than a Number

• Holistic Review in Graduate Admissions

Implicit Bias

• The Influence of Unconscious Bias

• UCLA Equity, Diversity and Inclusion Implicit Bias Training

Noncognitive Measures

• Noncognitive Measures for Higher Education Admissions

• The Core Competencies for Entering Medical Students

• The Role of Noncognitive Constructs and Other Background Variables in Graduate Education

Rubrics

• An Effective Rubric Norming Process

• IGEN and C-CIDE Rubric Design & Use

Social Justice

• Social Justice Education in America

• Teaching the Truth: Social Justice and Social Class in Graduate School

• What Is Social Justice Education Anyway

VIII. Recommended Reading

• Baumgartner, L. M., & Johnson-Bailey, J. (2010). Racism and white privilege in adult education graduate programs: Admissions, retention, and curricula. New Directions for Adult and Continuing Education, 2010(125), 27–40. doi: 10.1002/ace.360

• Bersola, S., Stolzenberg, E., Love, J., & Fosnacht, K. (2014). Understanding Admitted Doctoral Students’ Institutional Choices: Student Experiences versus Faculty and Staff Perceptions. American Journal of Education, 120(4), 515-543. doi:10.1086/676923

• Garces, L. (2014). Aligning Diversity, Quality, and Equity: The Implications of Legal and Public Policy Developments for Promoting Racial Diversity in Graduate Studies. American Journal of Education, 120(4), 457-480. doi:10.1086/676909

• Guterl, F. (2014, October 1). Diversity in Science: Why It Is Essential for Excellence. Retrieved October 23, 2019, from https://www.scientificamerican.com/article/diversity-in-science-why-it-is-essential-for-excellence/.

• Orfield, G. (2014). Realizing the Promise of the Civil Rights Revolution: Challenges and Consequences for Graduate Education. American Journal of Education, 120(4), 451–456. doi: 10.1086/676924Graduate Admissions

• Posselt, J. R. (2014). Toward inclusive excellence in graduate education: Constructing merit and diversity in PhD admissions. American Journal of Education, 120(4), 481-514.

• Posselt, J. R. (2016). Inside graduate admissions: Merit, diversity, and faculty gatekeeping. Harvard University Press.

• University of California. (2016). Guidelines For Addressing Race and Gender Equity in Academic Programs in Compliance with Proposition 209. Retrieved from https://www.ucop.edu/general-counsel/_files/guidelines-equity.pdf

• Fine, E., & Handelsman, J. (2012). Reviewing Applicants: Research on Bias and Assumptions. Women in Science & Engineering Leadership Institute University of Wisconsin-Madison.

• Milkman, K. L., Akinola, M., & Chugh, D. (2015). What happens before? A field experiment exploring how pay and representation differentially shape bias on the pathway into organizations. Journal of Applied Psychology, 100(6), 1678–1712. doi: 10.1037/apl0000022

• Moule, J. (2009). Understanding Unconscious Bias and Unintentional Racism. The Phi Delta Kappan, 90(5), 320-326. Retrieved from, http://www.jstor.org/stable/20446107

• Nivet, M. A. (2011). Commentary: Diversity 3.0: A Necessary Systems Upgrade. Academic Medicine, 86(12), 1487–1489. doi: 10.1097/acm.0b013e3182351f79

• Frehill, L. M., & Ivie, R. (2013). Increasing the Visibility of Women of Color in Academic Science and Engineering: Professional Society Data. New Directions for Higher Education, 2013(163), 7–21. doi: 10.1002/he.20061

• Gardner, S. K. (2013). The Challenges of First-Generation Doctoral Students. New Directions for Higher Education, 2013(163), 43–54. doi: 10.1002/he.20064

• Graham, E. (2013). The Experiences of Minority Doctoral Students at Elite Research Institutions. New Directions for Higher Education, 2013(163), 77–87. doi: 10.1002/he.20067

• Ramirez, E. (2012). Examining Latinos/as’ Graduate School Choice Process. Journal of Hispanic Higher Education, 12(1), 23–36. doi: 10.1177/1538192712452147

• Ruiz, E. C. (2013). Motivating Latina Doctoral Students in STEM Disciplines. New Directions for Higher Education, 2013(163), 35–42. doi: 10.1002/he.20063

• Hall, J. D., O’Connell, A. B., & Cook, J. G. (2017). Predictors of Student Productivity in Biomedical Graduate School Applications. Plos One, 12(1). doi: 10.1371/journal.pone.0169121

• Miller, C., & Stassun, K. (2014, June 11). A Test that Fails. Retrieved from https://www.nature.com/naturejobs/science/articles/10.1038/nj7504-303a.

• Moneta-Koehler, L., Brown, A. M., Petrie, K. A., Evans, B. J., & Chalkley, R. (2017). The Limitations of the GRE in Predicting Success in Biomedical Graduate School. Plos One, 12(1). doi: 10.1371/journal.pone.0166742

• Sedlacek, W. E. (2004) Why we should use noncognitive variables with graduate and professional students. The Advisor: The Journal of the National Association of Advisors for the Health Professions. 24 (2), 32-39.

• Selva, J., & Asmelash, L. (2020, September 02). University of California System can't use SAT and ACT tests for admissions, judge rules. Retrieved September 10, 2020, from https://www.cnn.com/2020/09/01/us/university-of-california-sat-act-trnd/index.html

• Coleman, A. L., Lipper, K. E., Taylor, T. E., & Palmer, S. R. (2014). Roadmap to Diversity and Educational Excellence: Key Legal and Educational Policy Foundations for Medical Schools. Roadmap to Diversity and Educational Excellence: Key Legal and Educational Policy Foundations for Medical Schools. Association of American Medical Colleges.

• Kent, J. D., & McCarthy, M. T. (2016). Holistic Review in Graduate Admissions: A Report from the Council of Graduate Schools. Council of Graduate Schools. Retrieved from, https://cgsnet.org/ckfinder/userfiles/files/CGS_HolisticReview_final_web.pdf

Visit the AMIGA literature resources page for in-depth holistic review resources.

IX. Project Contacts

The AMIGA Project is generously funded by the Andrew W. Mellon Foundation. The success of the Project is based on graduate faculty and Graduate Division collaborations that includes humanities and humanistic social science fields at both UC Davis and UCLA. Dr. Julie Posselt (USC), project assessment lead and advisor, also contributes invaluable advice and input. Moreover, staff at both UC Davis and UCLA provide critical support and vision. Finally, the AMIGA Project has supported multiple graduate students that offer their talent, insights and time to all aspects of the Project. Most recently, Karla Rodriguez Beltran (UC Davis) and Aireale Rodgers (USC) have applied their considerable skills to the AMIGA Holistic Review Toolkit and website, respectively, of which we are deeply grateful. ~ Josephine Moreno