The Power of Data for Decision Making and theAppropriate Use of Analytics in Higher EducationSettings
Michele J. Hansen, Ph.D., Assistant Vice Chancellor
Institutional Research and Decision Support, IUPUI
The Power of Data for Decision Making and theAppropriate Use of Analytics in Higher EducationSettings
Michele J. Hansen, Ph.D., Assistant Vice Chancellor
Institutional Research and Decision Support, IUPUI
Institutional Context
(IUPUI)
• Recognized for Learning Communities & the First Year
Experience (U.S. News)
• Large Urban Public Research University
• Student population of about 29,000 students
• First-Time cohort just over 3,300 and New External
Transfers just over 1,200 each year
• Over 250 degree programs from both Indiana& Purdue Universities, guided by the Principles of Undergraduate Learning
• Approximately 42% undergraduates are Federal Pell
Recipients and 35% are First Generation College Students
• About 50% of First-Year students commute to campus
• Role of data and analytics in higher education today and in the future
• Definitions and sources of data
• Use of data to improve student learning and success
• Ethics, social justice, security, and privacy
• Questions and discussion
• 
Enhance student achievement
• Plan courses and curriculum
• Recruit and retain students
• Optimizetheschedulingof classrooms
• Understandlevels ofstudent engagement
• The ability to make effective decisions is crucial if an institution of higher education is going to continuously improve student learning and success.
• Data helps decision makers evaluate alternatives, make resource allocations, and make informed choices.
• The effective and ethical use of analytics can improve the academic and support experience in ways that promote studentsuccess, equity, and institutional sustainability.
 |
• 
Must be reliable and timely.
• Thick data (quantitative and qualitative) upon which to make decisions.
• Take into account ethics and privacy.
• Effective data management, governance, and analysis techniques is of central importance.
• Must be accessible, action oriented, and easy to navigate. Many decision makers find that using data is no easy task as they find themselves inundated with nearlyoverwhelming amounts ofdata.
Ø 
Engage in efforts to understand the anatomy of decision making across campus (who makesdecisions, when, how, and what data is needed).
Ø Building data literacy and capacity across institution so that information exploration, interpretation, and analysis are used to supportevidence-based decisionmaking and improve institutional effectiveness.
Ø Deliver training and data toolsthat allow decision makers to leverage data and information.
7
• Strategies are set, resources are allocated; and actions are taken based on outcomes extracted from rigorous and continuous analysis of good data.
• Use of data is viewed positively and not punitively.
• Good analysis is viewed as a platform for collaboration and discussion.
• Positions a college or university to strengthen the quality of
the student experience in and outside of the classroom.
• Includes quantitative data and qualitative data(focus groups, town halls, social media, and more)
• Synthesizes data from multiple sources to identify institutional opportunities and challenges and barriers to student success and learning.
• Improve data processing speed.
• Allow us to summarizemultiple data points.
• Visualization platforms – Tableau.
• Improve access via self-service tools.
• More complex the task, moredifficult to replace with technology enabled tools.
• Decide on research design and even appropriate statistical testor algorithm.
• Understand complex data questions posed by decision makers.
• Consider ethical use of datafor decision making.
Statement of Aspirational Practice ForInstitutional Research – Association ofInstitutional Research (AIR)
• “Data are everywhere across institutions of higher education, and access to analytical tools and reporting software means that a wide array of higher education employees can be actively involved in converting data into decision-support information.”
• “The demand for data to inform decisions in postsecondary education is greater than ever before. Colleges and universities have significantly increased capacity to collect and store data about student and institutional performance, yet few institutions have adequate capacity for converting data into information needed by decision makers.”
Student Focus
Leadership for IR Function
IUPUI Selected as 1 of 10 Founding Institutions
Statement of Aspirational Practice
Structures and Leadership for IR
Expanded Definition of Decision Makers
Leverage Technology-DataVisualization Platforms such asTableau for Accessible Self-Service Data ToolsInstitutionalResearch and Decision Support
Contains highly interactive dashboards allow users to drill down and filter to allow detailed exploration of key indicators associated with the IUPUI Strategic Plan.
• We in data analytics professions play a key role in building equity-minded decision cultures.
• Create shared understandings and meanings. The Language we use matters.
• Critical to understand institutional contexts and foster understanding ofwhat structures, policies, implicit biases, stereotype threats and factors that are contributing to inequities.
• Important to be aware that many frameworks do not fully consider the experiences of marginalized groups or approach their experiences from a deficit perspective. We need to make sure to take time to thoughtfully select a framework to help answer proposed research questions.
• We need to pay attention to small populations. All voices are important.
• Rethink comparisons and reference groups. White students’ experiences are often held as the norm to which other groups are compared (Mayhew & Simonoff, 2015), carrying the assumption that White students’ experience is “normal” and implying this should be achieved by other student groups.
• Disaggregate data to help decision makers understand inequities in access and outcomes by student groups (e.g., first generation, gender, historically marginalized, under-resourced, low-income, nontraditional, transfer).
• Disaggregate data to help decision makers understand inequities in access and outcomes by faculty and staff groups (e.g., gender, historically marginalized, age, rank).
• Design our interactive reports so decision makers can examine intersectionality.
• Conduct various analyses and investigations that examine inequities in student access and outcomes (e.g., retention, academic performance, student engagement, learning outcomes). These analyses allow decision makers to understand and ideally address inequities.
• Institutional research is a broad category of research activitiesconducted at schools, colleges and universities to inform campus decision-making and planning in areas such as admissions, financial aid, curriculum, enrollment management, student success and learning, staffing, student life, finance, facilities, athletics, andalumni relations.
• Typically involves research conducted for internaldecision making, planning, and external accountability reporting.
• Purpose is primarily to improve institutional effectiveness and not to
generalizable research or inquiry.
 |
Centre for Higher Education Trust (CHET)
National Student Clearinghouse Academic Analytics
• Assessment is often defined as a continuous cycle of improvement and is comprised of a number of features: establishing clear, measurable expected outcomes of student learning; ensuring that students have sufficient opportunities to achieve those outcomes; systematically gathering, analyzing, and interpreting evidence to determine how well student learning matches expectations; and using the resulting information to understand and improve student learning” (Suskie, 2009, p. 4).
• Statistical methods involved in carrying out a study include planning, designing, collecting data, analyzing, drawing meaningful interpretation and reporting of theresearch findings.
• Gives meaning and often involves inferences.
• Requires and an understanding of quantitative and qualitative variables, measures of central tendency, sample size estimation, power analysis and statistical errors/assumptions.
• Requires a proper design of the study (understanding of research methods) and choice of a suitable statistical test.
• Improper statistical methods may result in erroneous conclusions
which may lead to unethical practice.
Retention While Taking Into Account Academic Preparation, Income, Registration Date, and First Generation Status
100.0%
Adjusted One-Year Retention Rates
90.0%
80.0%
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
78.6% 77.3%
75.1%
79.5%
75.2%
Campus Housing Summer Bridge Themed Learning
Communities
Participants 
Nonparticipants
Fall 2019 Cohort
• 
Faculty and staff do play a role in bolsteringadaptive mindsets about intelligence— which can powerfully shape students’ own growth mindset and, in turn, their academic outcomes
• Examples of items:
• “Ingeneral,mostpeopleat institution believe that some students are smart, while others are not”
• “In general, most people at institution seem to believe thatstudentshave acertain amount of intelligence,and they really can’t domuch tochangeit.”
• Faculty and staff do play a role in bolsteringadaptive mindsets about intelligence—which can powerfully shape students’ own growth mindset and, in turn, their academic outcomes
• Provide support for learning
• Set high standards and convey that we are motivated to help
students attain them (journey taking together)
• Give sense of purpose (applying learning experience tolife and real world problems)
• Foster growth and not fixed.
• Set clear expectations and giving constructive, clear feedback on
learning
• Communicate that college is a place that student belongs (not just a place for students)
1. Student Engagement Roster (SER) designed as acommunication tool that allows professors/instructors to efficiently provide critical and constructive feedback to students.
2. Conducted an investigation in an effort to collect students’perceptions of the SER process and how the feedback affected their behaviors, classroom engagement, and attitudes.
Student Engagement RosterInvestigation
• 81% reported that they read their Student Engagement Roster(SER)
feedback.
• 74% reported that the feedback made them feel (“Quite a Bit” or “A Great Deal”) positive about their instructors inthe following areas: cares about my learning, is approachable, is available to provide help if I have difficulties, and is committed to creating an environment.
• Students offered the following suggestions for SERimprovement:
– Provide feedback more frequently
– Require all instructors use the SER and provide feedback
– Instructors should provide personalized, individualizedmessages and not generic comments
– Provide feedback that is constructiveand provides specific recommendations about how to improve
– Make the feedback more accessible and easier to find inCanvas/One I.U.
INDIANA UNIVERSITY–PURDUE UNIVERSITY INDIANAPOLIS
• Understand data and limitations.
• Not relying on single data point for telling whole story.
• Know data definitions.
• Understand of how data collected, sources,
and research methods.
• Correlations does not mean causation.
• Data analytics is the science of drawing insights from raw data sources. Many ofthe methods and techniques used in data analytics are automated into mechanicalprocesses and algorithms that organize raw data for human decision making.
• Data analytics used to understand patterns of data that may otherwise be lost in the mass of information.
• “A set of [business intelligence]technologies that uncovers relationships and patterns within large volumes of data that can be used to predict behavior and events”.
• “Predictive analytics is forward-looking, using past events toanticipate the future.”
van Barneveld, A., Arnold, K.E., & Campbell, J.P. (2012).
• Poor Performance in first semester or Earning DWFI in a course
• Low high school or transfer in GPA (lower than 3.00)
• Under-Resourced (high levels of unmet financial aid, low-income)
• Late Registration Date
• Not having Academic Honors Diploma or Rigorous High School Curriculum
• Attending part-time and not enrolling in 15 or more credit hours
• Not Placing into Credit Bearing Math
• Transferring in with few hours with no degree
• First Generation College Student
• Not Participating in High Impact Practices and EarlyInterventions First Year (First Year Seminars, ThemedLearning Communities, Summer Bridge)
• Living Off-Campus
• Living Alone or With Others Not Attending the institution
• External commitments (working for pay off-campus,commuting, taking care of dependents and household responsibilities)
• Low growth mindset, self-efficacy, sense of belonging, commitment to institution (intent to transfer)
• Billions of data points being generated every minute of every day by humans, computers and technological devices – creating a real-time digital footprint of our lives with every credit card swipe, phone use, Google search, Facebook post, and more.
• With availability of this ocean of data, how can we use it to better understand and our world andserve our needs.
• While college and universities doing cutting edge research on Big Data and educating data scientists,not using it as much as other industries to innovateour academic institutions.
Lane & Finsel, 2014
“From the dawn of civilization to 2003, humans created five exabytes worth of data. As of 2013 humans produced this same amount of information every two minutes” (Miller &Chapin, 2013 as cited by Lane and Finsel, 2014)
• Typically field of Data Science used for processing Big Data - Data Science field used to tackle big data. an umbrella term that encompasses data analytics, data mining, machine learning, and several other related disciplines.
• Involves gathering data from multiple sources and applies machine learning, predictive analytics, and sentiment analysis to extract critical information from the collected data sets.
• Machine learning used as practice of using algorithms to learn from data and then forecast future trends for that topic. Traditional machine learning software comprised ofstatistical analysis and predictive analysis that are used to spot patterns and catch hidden insights based on perceived data (used by Facebook).
• Machine learning is a method of data analysis thatautomates analytical model building. It is a branchof artificial intelligence based on the idea thatsystems can learn from data, identify patterns and make decisions with minimal human intervention. (SAS)
• An algorithm is a set of heuristics and calculations that creates a model from data. To create a model, the algorithm first analyzes the data you provide, looking for specific types of patterns or trends.
• The algorithm uses the results of this analysisover many iterations to find the optimal parameters for creating the mining model. Theseparameters are then applied across the entire data set to extract actionable patterns and detailed statistics.
 |
“Relying on big data alone increases the chances we will miss something while giving us the illusion that we know everything.”
“Then something unknown enters the picture.”
Tricia Wang
• “The interpretation of a wide range of data produced by and gathered on behalf of students in order to assess academic progress, predict future performance, and spot potential issues.”
• “The use of predictive modeling and other advanced analytic techniques to help target instructional, curricular, and support resources to support the achievement of specific learning goals”
• Allows instructors to tailor educational opportunities to each
student’s level of need and ability.
• Can be used assess curricula, programs, and institution
• Source: van Barneveld, A., Arnold, K.E., & Campbell, J.P. (2012).
• Grades on assignments and exams – progress
• Engagement (logging on and page views)
• Attendance
• Activity
• Chat rooms
• University-and course-level learningoutcomes - scores on rubrics
• My Learning Analytics (MyLA) is a dashboard that provides students with information about their engagement with coursematerials and resources, assignments, and grades in a Canvas course.
 |
• E-texts also provide powerful markup and interaction tools.
• Highlighting, shared notes, questions, and answers.
• Research found that higher engagement with e-texts (reading and highlighting) correlated with higher course grades (Abaci, Quick, andMorrone, 2017) .
• Statement of integrity
• Overarching principles
• Guide for higher education data use
• Code of conduct
• Legal advice
• Predictive models and algorithms are increasingly the tools used to make decisions that affect people’s lives ---where they go to school, whether they get a loan, how much they pay for health insurance, what type of sentence people receive when convicted of a crime.
• Decisions being made by mathematical models rather than humans.
• Ideally, the mathematical models are unbiased and lead togreater fairness. Not True!
• Many models used today are mysterious, unregulated, and uncontested with no feedback or correction mechanisms. Can be wrong.
• Algorithms if left unchecked essentially increase inequality creating
“toxic cocktail for democracy.”
• Used for harm rather than good.
• Algorithms reinforce discrimination and widen inequality,
• Use people’s fear and trust of mathematics to prevent them from asking questions.
• Rely on proxies (proxies are easier to manipulate than complicated reality they represent).
• “Algorithms that are important, secret and destructive”.
• Affect large numbers of people, are entirelyopaque, and destroy lives.
• Models are opinions embedded in mathematics.
• Baseball
• Amazon
• Predictive modeling used tohelp provide resources for students
• US News and World Report Rankings (algorithms based on proxies and rankings become destiny).
• Marketing by for profit colleges(ads that pinpoint people in great need and sell them false and overpriced promises – predatory ads).
• Admissions decisions.
• Transparency (not relying on black boxes).
• Continuously update.
• Assumptions and conclusions clear.
• Rely on actual data rather than proxies.
• People being modeled understand the process and understand the models objective.
• Use to help rather than harm.
• Protection of Personal Information Act (POPIA)
• Institutional Review Boards (IRB) – generalizable research
• Stories of data violations
• Data Governance
• Student Data Advisory Councils and Faculty/Staff Data Advisory Councils
• Ownership of data such as course evaluations and Learning ManagementSystems
• Explaining use of data
• Data definitions and metadata
• Clear notes and sources
• Understanding limitations
• Explore prior to analyses and use
• Clear explanations of methods used to analyze or organize data
• Wealth of data available fordecision making
• Value of data-based decision making
• Data literacy
• Ethical Use
• Theory-based methods
Final Thought• In order for students to be productive citizens in a world in which lower skilled labor is being replaced by computers and robots, we need an educational shift focused and need to rebalance our curriculum to develop students with “creative mindsets and the mental elasticity to invent, discover, or create something valuable to society rather than concerned solely with “topping up students' minds with high-octane facts.”
• New skills: data literacy to manage the flow of big data, and technological literacy to know how their machines work, but human literacy, from the humanities, communication, and design, to function as a human being in a world populated with artificial intelligence advanced technologies.
Aoun (2018), author of Robot- Proof Higher Education in the Age of Artificial Intelligence,
Contact Information
MicheleJ.Hansen,Ph.D. AssistantViceChancellor mjhansen@iupui.edu317-278-2618
Institutional Research and Decision Support
irds.iupui.edu
Contact us with questions or requests for information!
IUPUI
• Aoun, J. E. .(2017). Robot-Proof: Higher Education in the Ageof Artificial Intelligence. MIT Press.
• Abaci, S., Quick, J., & Morrone, A. S. (2017) Studentengagement with e- textbooks: What the data tell us. Educause Review 52 (4)
• J. E. Lane (Ed.), (2014). Building a smarter university: Big data, innovation, and analytics. Albany, NY: SUNY Press. Foss, L. H.
• O’Neil, C. (2016). Weapons of Math Destruction: How Big DataIncreases
Inequality and Threatens Democracy. New York, NY: CrownPublishers
• Swing, R. L.& Ross, L. E. (2016). A new vision for institutional research,
Change: The Magazine of Higher Learning, 48 (2), 6-13.
• van Barneveld, A., Arnold, K.E., & Campbell, J.P. (2012).Analytics in higher education: establishing a common language [White paper]. Boulder, CO: EDUCAUSE Learning Initiative. Available at https://library.educause.edu/resources/2012/1/analytics-in-higher-education- establishing-a-common-language
• Wang, Tricia (2016, September). The human insights missing from big data. Ted Talk. https://www.ted.com/talks/tricia_wang_the_human_insights_missing_from_bi g_data?utm_campaign=tedspread&utm_medium=referral&utm_source=tedcomshare
• Webber, K.L. & Zheng, H. (Eds.) (2020). Big Data on campus: Data-informed decision making in higher education, Johns Hopkins University Press. ISBN 9781421439037.
• Designing College for Everyone. Brief written by the College Transition Collaborative.
• Leveraging Mindset Science to Design Educational Environments that Nurture People’s Natural Drive to
• Designing Supportive Learning Environments. Video created by the Mindset Scholars
Network.
• The New Science of Wise Psychological Interventions. Journal article by Gregory Walton, published in Current Directions in Psychological Science.
• Social-Psychological Interventions in Education: They’re Not Magic. Journal article by
David Yeager and Gregory Walton, published in Review of Educational Research.
• Broadening Participation in the Life Sciences with Social-Psychological Interventions. Journal article by Yoi Tibbetts, Judith Harackiewicz, Stacy Priniski, and Elizabeth Canning, published in CBE Life Sciences Education.