+
Preparing for Life: Evaluation - How it helps in real life!
Noel Kelly, Programme Manager
+ Context
I have 35 years experience of working in disadvantaged communities
I have only really engaged with evidence and evaluation in the past 12 years
For the first 23 years I trusted instinct and hoped I was getting it right
Now I am totally committed to evidence and evaluation and I know that I am making a real difference!
2
Starting Out
2004 – Ongoing concerns about child outcomes
We had tried many approaches but had not changed the outcomes.
We had no history of conducting evaluations to see if what we did worked.
We used evaluation reports as a way to get funding
Even at Government level evaluation reports were often not published especially negative reports.
+ Starting Out
4
What changed for me? A few things happened simultaneously!
I completed a Masters in Education in 2003. This forced me to engage with research and really think about what I was doing.
At the same time a funder (The Atlantic Philanthropies) began investing in child and family services in Ireland.
This funder insisted on evidence of need, rigorous planning and robust evaluation before any grants were considered. It also helped that 5 year funding grants were offered!
+ The Atlantic Philanthropies Planning - Methodology
Emphasis on prevention and early intervention in children’s lives
Focused on outcomes for children as the clear goal
Delivering tailored activities/services most likely to achieve the outcomes
Employing evidence of ‘what works’ for children to determine activities/services
Facilitating local communities and agencies to work together in an integrated way
Measuring progress rigorously through evaluation
+ Our Problem
Outcomes for children in our communities were consistently poorer than Irish norms.
In Ireland we had tried – smaller class sizes in schools, after school programmes, family and community programmes – you name it we had tried it.
We intervened to try and undo problems too late.
Ireland had little investment in children before they arrived in primary school.
We believed the solution had to include parents as the primary educators of their children
Many state agencies had given up on families in our area.
6
Engaging with Evaluation and Evidence
We evaluated the school readiness skills of children starting school in 2004.
Over 51% of the children were not ready for school.
In response we designed a home visiting programme to improve school readiness.
We sought expert help internationally.
We created a logic model to underpin our plan.
We contracted evaluators to measure outcomes.
We agreed to honestly share our research findings – good or bad
+ What people said before we started about evaluation
Evaluation is a waste of money – this was based on the poor quality evaluation we had been doing
It would be better to use the money on services for families
The evaluation tail will wag the programme dog
The evaluators will be telling us what we can and can’t do and what do they know about families
It will create too much paperwork
Staff will not agree to participate
Families will refuse to take part as it will be too intrusive
8
+ Big Questions - What will we do when we find out?
Positive Outcomes = That’s good news
Negative Outcomes = What do we do then?
we can choose not to believe the results,
we can blame the evaluator,
we can ask for further evaluation,
we can ignore the results and continue or
we can learn from evaluation,
we can stop doing what doesn’t work
we can use evidence based approaches
9
+ ‘Preparing for Life’ Programme
Preparing for Life: one of first experimental early childhood interventions in Ireland
Community-led initiative: operated in a highly disadvantaged area of Dublin
Evidence of need: Children scored below the norm on cognitive &
socioemotional skills on school entry
Aim: Improve levels of school readiness by assisting
parents in developing skills to prepare their children for
school.
Intervention
1. Home-visiting Mentoring Programme
Fortnightly home-visits from trained mentor - pregnancy to school entry
Mentor’s role: support parents about child development & parenting
Home visits structured around “Tip Sheets” focused on 5 areas of school readiness
Based on theories of attachment, social learning, & ecological development
2. Triple P Parenting Programme
(Sanders et al., 2003) Offered when children are between 2 and 3 years old
Promotes healthy parenting practices and positive parent-child attachment
Group Triple P: 5 two-hour group discussion sessions and 3 phone calls
+ Randomisation
INTERVENTION
CONTROL Random Assignment
Split sample
Assess outcomes for both groups PFL
Non-PFL
+ Evaluation Design
Impact Evaluation
1. Interviews: 7 home-based assessments (BL, 6, 12, 18, 24, 36, 48)
2. Direct assessments: Cognitive skills & executive functioning at age 4
3. Teacher reported school readiness assessment: S-EDI in junior infants
4. Maternal physiological data and Diaries
5. Administrative records: birth records and child hospital records
Implementation Evaluation
1. Implementation data
2. Focus groups: mothers & father figures
3. Semi-structured interviews: programme staff
4. Draw and talk activities & scenarios : children
+
What we Found?
14
HEALTHY CHILDREN: Protein Intake
32% 31%
33%
18% 17%
23%
0 5 10 15 20 25 30 35
24 months 36 months 48 months
% of children consuming RDA of protein
RDA Protein
PFL children Non PFL children
Important for growth and brain
development
+ HEALTHY CHILDREN: Body Mass Index
Important for health, running,
playing, sports
23%
41%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Overweight/Obese at Age 4
% of children with ovwewight/obese
PFL children Non PFL children
+ SOCIAL CHILDREN: Behavioural Problems
0%
1%
2%
9%
8%
17%
0 2 4 6 8 10 12 14 16 18
24 months 36 months 48 months
% behavioural problems using Child Behavior Checklist (CBCL)
CBCL
PFL children Non PFL children
Less likely to throw tantrums or
get into fights
Less likely to appear nervous,
sad or worried
+ SMART CHILDREN
0% 0%
9%
63%
83%
50%
0 10 20 30 40 50 60 70 80 90 100
6 months 12 months 18 months 24 months 36 months 48 months
% Significant
Assessment Point
Impact on Parent Reported Cognitive Skills
% Statistically Significant Effects
Distribution of BAS Cognitive Scores
0 5 10 15 20 25 30 35 40 45
0 20 40 60 80 100 120 140 160
% of Children in each Group
General Conceptual Ability Score
Non-PFL Children PFL Children
+ Mechanisms: Changes in Parenting Practices
PFL children lived in higher quality home environments
Stimulating, variety of activities, appropriate learning materials, less restricted, parents more involved in learning, better diets, better routines etc.
PFL parents less likely to have permissive parenting style
Found it easier to discipline children and follow through with disciplinary practices
Better screen time practices
Less time watching TV, DVDs, videos at age 3 and less time watching TV alone at age 4
+ Policy Impact
Healthier children to fully participate in play and recreation
Better cognitive and behaviour skills to engage for learning and success
Safe and stimulating home environments to grow and develop
Starting school equipped with skills to learn, engage and succeed in
education and employment Better socio-emotional skills to engage in positive relationships with
friends, teachers, and others
+ What we are doing with the results?
Updated our programme based on evaluation results
Offering the programme to more families
Expanding our work to now include working with health, childcare and school staff to improve their practice
Media – using all media forms to share the results & learning
Influencing policy makers & politicians
Supporting the implementation of our programme in other areas
Lobbying for more investment in prevention and early intervention in Ireland
Research team are sharing our results internationally
22
Delivering Ante-Natal Care and Education in
the Community
Home Visiting Programme to support
Child Development and Parenting
Early Years Practice Programme – Quality, Speech and Language
and Transitions
Schools Programme Literacy, Play and Self-
Regulation
Triple-P Parenting Programme
All our Interventions
+ Staff and Evaluation
What does engagement with evaluation mean for our
organisation and staff
24
+ Staff Characteristics we look for
Staff Selection – really important to get the right people
Staff must believe in what they are doing
Staff must be open to coaching, feedback, evaluation and supervision to ensure quality
Staff must have the ability to see successes even small ones
Staff need to be resilient and optimistic
Staff have to be able to build and sustain relationships
25
+ Enabling Staff to do the best job possible
Staff get appropriate training, supervision and coaching
Staff actively seek out training opportunities
Coaching including role play, observations and video coaching
Regular well managed team meetings focused on practice
Regular structured meetings with evaluators
Consistent record keeping
Valuing staff inputs and including staff in decision making.
26
What we did as an organisation to build evaluation into everyday practice
Show a genuine commitment to evaluation – willingness to use the learning from evaluation
Embrace failure – learn from mistakes and see mistakes as learning opportunities not failures. Staff should not fear talking about failures.
Provide resources to carry out timely, useful and relevant evaluations.
Create a learning culture where all staff see the benefits of collecting information and using it to inform their work.
Sharing, discussing and analysing evaluation findings.
+ Practical activities to support an evaluation culture
Evaluation practices we use in everyday work:
- learning communities; (internal or across organisations)
- peer reviews & peer coaching;
- self-reflection & supervision
- development of implementation/practice guides,
- Journal clubs – staff members prepare a short presentation on a relevant piece of research/learning
- Practice based team meetings where staff members present on their work – and pool expertise to problem solve.
- Use of evidence based approaches & programmes
- Building strong relationships with research bodies i.e. universities.
28
+ What do our staff say about
evaluation now?
We need to know that what we are doing works.
We needed to be sure we aren’t doing harm
Having evaluation evidence gives us assurance and confidence
We all feel better when we know we are making a difference
We can celebrate successes
We are not afraid to admit we make mistakes – we learn from them
We enjoy working with external evaluators
Families enjoy meeting evaluators and talking about their children
29
+ For more information visit our websites
www.preparingforlife.ie or
http://geary.ucd.ie/preparingforlife
30
