Appreciative AIssessment

National, state, and district mandates for standardized assessment negatively impact learners and their teachers, and the US education system is in measurable decline as literacy rates drop, dropouts and mental health issues rise, and social justice needs accelerate. Appreciative A.I.sessment is a multi-faceted toolkit that enable appreciative inquiry, well-being, and universal design to mitigate these challenges. The toolkit will be extensible to mainstream environments across all education sectors and disciplines. 

Appreciative A.I.sessment consists of three technical components: 1. query form and rubric maker 2. chatbot, 3. learning dashboard. We outline the technology below. 

We will create discipline-specific forms that include a set of criteria for a single high-stakes assignment in each primary discipline (math, reading, science, etc.). The forms will be editable to address the areas in which the learner needs to stretch. 

1. The learner will input responses into a form with assistance from the "AIssessment peer" to refine the topic. The form is used to develop a custom assignment and rubric that meets the generalized assignment criteria but is personalized to the individual learner's interests and stretch areas. The assignment and rubric can be refined using the AI Peer before submitting it through the dashboard for approval. 

2. The AIssessment peer serves as an age-appropriate peer with whom the learner develops a personal rapport using the POGIL framework (Process Oriented Guided Inquiry Learning). Specific attention will be given to challenges with human-autonomy teaming and trust. However, we predict that learners will form meaningful bonds with their AI peer in ways that are not typically possible for differently-abled learners in mainstream environments. The AIssessment peer is not merely a tool or mentor but serves as a digital friend with whom the learner develops ancillary skills in self-efficacy, soft infrastructure, and non-cognitive skills as they work through the project lifecycle: generating the prompt, iteration, and peer review, developing a rubric, and self-reflection. 

3. The project dashboard is developed using human-centered and resilient design principles. The dashboard houses the AI peer, form and rubric maker, and communication tool for the teacher to view and respond to all aspects of the project. 

The technology will be able to assist with metacognition, divergent thinking (creativity), learning transfer, and convergent thinking (applying learned lessons within the disciplinary activities and holistically across disciplines), promote conversations (developing trust), empower students as learners to create their own learning opportunities by stepping outside the box using intelligent risk-taking. 

Up to 30% of the U.S. public school student population is neurodivergent, and differently-abled learners comprise 15% of all learners. We ask the foundation to consider intersectionality: many learners belong to two or more vulnerable populations (i.e., impoverished and differently-abled). Researchers increasingly find that AI integration can foster neuro-inclusive environments and establish trust and communication in ways that human-to-human interaction may not [2, 3, 4]. 

Our Appreciative AIssessment tool supports metacognition, divergent thinking, learning transfer, and convergence (applying within and across disciplines), promote conversations (developing trust), and empower students as autotelic learners who create their learning opportunities by stepping outside the box using intelligent risk-taking [5]. We promote Appreciative Inquiry through:

1. Generative Processes: Most K-12 learning occurs in a limited environment--a "black box" in which solutions are known and responses to prompts involve regurgitating information or recall and recognition. Generative processes engage learners in the full lifecycle and various participatory roles in projects so that they can discover where they stretch and shine and how to apply/translate their learning beyond the classroom. Learners will move through the five Ds of Appreciative Inquiry (define, discover, dream, design, deliver) with the AI peer to support. 

1. Generative Thinking and Conversations: Using a gradual release model, learners will engage in a divergent thinking paradigm that flexes their inquiry-based and design thinking skills and asks them to engage in authentic conversations that are teacher-facing, peer-to-peer, and self-introspective. The Bok Center at Harvard, Eberly Center at Carnegie Mellon, and the Center for Teaching at U-Michigan emphasize that generative AI can reduce cognitive load and foster inclusion, equity, and belonging for at-risk and marginalized students. 

1. Generative Outcomes: The learner gets tangible experience playing various roles on their team (leader, visionary, implementor, beta tester, auditor); designs and develops a personalized project that meets the general curriculum criteria (expanding their non-cognitive and soft infrastructure skills in addition to the subject matter experts), builds trust and relationship with a "peer" (AI bot) using resilient education [6]. 

1. Generative Empowerment: Establish a community of learning that takes a responsible use approach to leverage AI that empowers self-motivation in learners and teachers [1].   

Works Cited

1. Deho OB., et al. “When the past != the future: Assessing the Impact of Dataset Drift on the Fairness of Learning Analytics Models”. IEEE Transactions on Learning Technologies (2024).

2. Gollasch D., et al. “Designing Technology for Neurodivergent Self-determination: Challenges and opportunities”. In Lecture Notes in Computer Science (2023).

3. Jesse T. “Creating Neuro-Inclusive learning environments”. In Advances in higher education and professional development book series (2023). 

4. Porayska-Pomsta K. “A manifesto for a Pro-Actively Responsible AI in education”. International Journal of Artificial Intelligence in Education (2023). 

5. Salas-Pilco SZ, Xiao K and Oshima J. “Artificial Intelligence and New Technologies in Inclusive Education for Minority Students: A Systematic review”. Sustainability 14.20 (2022). 

6. Sanders CK and Scanlon E. “The digital divide is a human rights issue: Advancing social inclusion through social work advocacy”. Journal of Human Rights and Social Work 6.2 (2021). 

As Vice President of Schools at The Help Group, I oversee the cohort of 300+ educational staff, faculty, and external partners who push in to offer services. Our various campuses specialize in supporting at-risk (i.e., homeless, survivors of abuse, substance abuse disorders), neurodivergent, and differently-abled learners to receive a fair and appropriate education. 

The Help Group’s 14 specialized day schools offer pre-K through high school programs for more than 1,150 students. Its broad range of mental health and therapy services and vocational and residential programs extend its reach to more than 6,000 children and their families each year, including innovative affirming programming for LGBTQ+ youth. With more than 700 staff members, The Help Group’s state-of-the-art schools and programs are located on six campuses in the Los Angeles area. The Help Group is widely regarded for its high standards of excellence and unique scope and breadth of services.

My doctoral dissertation (2018) focused on leveraging AI for assessment. My qualitative study aimed to use an Appreciative Inquiry (AI) approach to discover generative instructional and assessment practices of teachers in the language arts classroom at a 5th and 6th grade low socioeconomic public school campus in Texas that earned five distinctions or three consecutive years. The study sought to design a framework for promoting generative teaching and instructional practices to enhance student learning and engagement utilizing the first three phases of the AI 4D Model: Discovery, Dream, and Design. 

Standardized education doesn't work well for "atypical" learners; our student population requires differentiated instruction and communication interventions that can be augmented by an AI peer and toolkit. In our annual retreat, 100 percent of our faculty included requests for communication supports and opportunities to introduce novel assessment methods that don't have a high barrier to entry or require cumbersome amounts of time to implement. 

Additionally, more than 60% of our faculty and staff serve as change agents by virtue of belonging to the communities we serve. We pride ourselves on intersectionality, particularly in offering extracurricular and prosocial support through our extensive child and TAY services programs. These are local and state-funded programs that enable us to provide a spectrum of physical, financial, mental health, and educational supports for people ages 5-26. 

The introduction of the AI peer toolkit would transform our campus culture in meaningful, observable ways. We aim to show public schools and other non-profit educational institutions how extensible the toolkit is to other environments. My staff and faculty are fully committed to completing pilot studies with the toolkit, contributing toward qualitative studies and sharing data for white papers, and contributing their professional skills as specialists who work with historically marginalized, vulnerable populations.

The project is in the concept phase because we only have mockups,  wireframes, and a project timeline to create the tangible deliverable. To-date, we have accomplished researching the current literature, reviewing various AI technology that might serve our 5D model, and garnering the support of The Help Group administration and campus faculty to ensure that they are all on board with this project. 

Assessment (Proto. -sedeo, Proto-Indo-European sed, L. assidere) translates as “to sit beside,” and a well-known adage states, “What is the shortest word in the English Language that contains the letters a, b, c, d, e, and f? Feedback. Don’t forget that feedback is one of the essential elements of good communication.” Current assessment practices fail because they 1. assess conceptual knowledge rather than applied skill, 2. do not promote critical and divergent thinking, 3. focus on extrinsic rewards and punishments of grades, 4. create inequity and are designed with bias that disadvantages underserved populations, 5. make learners passive recipients of rather than active participants in evaluating their learning, 6. create fatigue and burnout in teachers and staff. Appreciative A.I.ssessment integrates best practices to decolonize and democratize the classroom and will result in measurable learning transfer, improved retention, and overall greater satisfaction and self-efficacy by all learners.

State and district mandates for standardized learning are negatively impacting learners and teachers. Standardization doesn’t translate to applying learning beyond the activity or the classroom and isn’t an accurate reflection of how well a student will perform on that activity beyond the classroom. Research shows over and over again that extrinsic rewards and punishments such as a high or low grade reduce self-efficacy and motivation to learn, decrease collaboration due to competitiveness, and create adverse mental health effects like anxiety and depression. Interventions that focus on active learning, collaboration (working through the problem with peers), and explaining the rationale for selecting an answer generally improve performance for all learners in post-instruction re-assessment.

It is also widely agreed that standardized assessment doesn't serve neurodivergent and at-risk children. To truly develop bias-free standardized testing, a much broader understanding of diversity must be discussed, understood, and accepted (Weiss, 2021). Assessment developers must account for culture, gender, religion, gender identity, geography, neurodiversity, physical and non-physical disabilities, learning styles and learning differences, communication styles, and less discussed diversity issues. 

Appreciative A.I.ssessment mitigates all of these challenges: 

1. Decolonizing and democratizing by giving learners agency to create, assess, and oversee their learning. 

2. Removing extrinsic rewards and punishments based on metrics and grades, and replacing them with self-efficacy and self-motivation (which promotes well-being and resilient pedagogy) 

3. Promoting learning transfer and teaching learners non-cognitive and soft infrastructure skills, which are considered by most industry experts to be the two most lacking and two most important areas of learning for career success

4. Reducing high barriers to participation for learners who receive, process, and express information differently

5. Freeing up time for teachers and staff to create authentic learning opportunities

Appreciative A.I.ssessment consists of three technology components: A query form, the A.I.ssessment peer, and a learning dashboard. 

1. Query Form: We will use forms from a company like Action.ai or Jotform, which include virtual assistants to aid users in completing forms. The query form will include questions that guide the learner to identify the challenge and growth areas of a generalized assignment prompt (i.e., thesis development, showing math calculations, translating physics problems into visuals) and criteria for assessing the prompt. The data will then be shared with the learning dashboard, where the learner will work with the A.I. Peer to develop a customized assignment that meets the standard criteria and encourages the learner to work on challenge areas (subject matter, soft infrastructure, and non-cognitive). 

2. A.I.ssessment Peer. We will use generative AI emphasizing machine learning and natural language processing to support learners to conduct queries, build communication models, and validate their work. Currently, generative AI faces challenges with hallucination (sharing false or inaccurate information), can show personal biases (prejudices and opinions), use harmful language (discriminatory language or graphic content), and lacks nuance (asking the user to move beyond absolute thinking). 

Using best practices similar to Outlier.ai, we will adapt an AI tool that encourages learners to use open-ended prompts focused on brainstorming, chat, generating rubrics and project ideas, helping to manage and assess project deliverables, and supporting revision. We would like to contact Outlier.ai or we can use generative A.I. tools that will integrate with our learning dashboard (i.e., A.I. widgets that typically integrate with LMSs like Brightspace and Canvas). 

3. Learning Dashboard: The dashboard will be the place where materials are accessed and communication shared between the learner, teacher, staff supports, and the A.I. Peer. This "boutique" will also include learning supports such as scheduling time with a tutor or the library, a shopping cart where the learner can add resources and materials that need to be purchased and borrowed from the school, and feedback and project management widgets. The learning dashboard can be designed using the same open source interface that Etsy uses, which has been adopted in small instructional design groups at schools like Yale University. 

These components have worked successfully on their own and now need to be assimilated as a cohesive toolkit. We intend to have the tool assessed by the Learning Design Innovation team at Dartmouth, where Dr. Milstein works as a Learning Designer with specialization in DEIAB and assessment practices. The third-party evaluators will help us develop metrics for assessing our technology intervention's success and recommending and reviewing revisions.

We have included citations throughout our various entries above that provide an evidence-based approach to our project idea. 

We intend to create a pilot of the A.I.ssessment peer and share it to other private and public schools for testing.  We maintain strong relationships with University of Southern California, University of California--Los Angeles, and other institutions of higher education that can serve as mentors. Finally, Dr. Horan retains his relationship with his mentor and dissertation committee, and they will advise on this project. 

Dr. Milstein completed Outlier.ai certification, holds multiple advanced certificates in Instructional Design and DEIAB, is CPACC certified, and teaches social justice courses to graduates and undergraduates and offers faculty development workshops in Universal Design and WCAG as an Associate Professor at University of Southern California. Dr. Milstein also recently submitted a grant to the National Science Foundation that focuses specifically on assessment re-modeling for the Force Concept Inventory to eliminate algorithmic bias in the test format. 

Dr. Horan is Vice President of Schools for The Help Group, where we focus on DEIAB-based initiatives for special needs, at-risk, vulnerable, and underserved populations of learners K-12 and into vocational school.

We will use best practices from our collective expertise and in our work with our colleagues (particularly in Dr. Milstein's work with Computer Science faculty and staff in the data visualization library) to ensure that algorithmic bias is resolved. 

Dr. Horan leads a team of forty full-time staff and three contract workers who will participate on this project. 

We have committment from our eight campuses to introduce the pilot toolkit into each K-8 classroom to test on one formative and one summative assignment of the teacher's choosing. 

We are an early stage team and will rely on your support to use an agile process to work with a team of teachers, administrators, instructional designer, and external tech partners and mentors to maintain our timeline to the following elements:

idea evaluation and market research: completed

customer discovery: first 3 months

prototyping and validation: months 1-6

business model development: months 6-8

branding and marketing: month 9

scaling: months 10-12

The toolkit will be developed as an OER (open education resource) and all content shared through a GitHub repository. We will also create and share out white papers showing the metrics we collect and included analyses of our pilot study. We maintain relationships with several R1 higher education institutions and work closely with several public school districts in the Los Angeles area and will be working together to make the toolkit testable and extensible to those environments. 

We are a non-profit agency that operates on a $2M annual budget. Our team is committed and capable but limited in financial and technical resources. We appreciate the support of the Foundation to help us develop a toolkit that can potentially become accessible to all learning communities to improve learning nationwide.