Evaluation instrument
From LSI
Text analysis of reported LSI success factors and effects
The phase "Exploring the field" resulted in a rough table with success factors and effects. This table is going to form a more substantial part of my research than I thought before. The table turns out to be the beginning of an evaluation instrument. It is a base for reconstructing and evaluating LSIs, as well as an instrument for assessment of conditions beforehand.
How come? In the summer of 2008 I started analyzing texts of reported LSI cases, from books and articles. At first this was meant to be an addition on the findings of workshops and interviews. Reading the texts I realised I needed a method to analyse the texts, so I "invented" my own text analyses work form: with a 5 colour marker I skimmed the texts.
- Green = pre-conditions
- Orange = working elements
- Blue = first order effects (non sustainable effects)
- Yellow = second order effects (sustainable effects)
While reading I came across other helpful information, so I added a colour:
- Pink = trends, research issues, synonyms for LSI and LGI methods
I analyzed transcriptions of interviews and workshop recordings of the field exploration, upto 50 reported LSI cases in published articles, emails and websites of global networks. See Sources for text analyses. I copied the marked items onto small sticky-notes (38x51 mm) with similar colours (since I could not find small blue notes, I chose light yellow for the non-sustainable effects and neon-yellow for the sustainable effects). Each note got a code, so the item can be tracked back to the original source. Then I started categorizing and clustering the notes. Preset categories were success factors (conditions, working elements of the intervention), effects (first order effects, second order effects) and useful information (trends, general remarks and synonyms for LSI).
Rough results of text analyses
In total, I clustered 680 notes, comprising 599 notes for success factors and effects and 81 with other remarks useful for this research. You can see the notes and the clusters on the photos of the A3-sheets (click on a photo to enlarge, click again to read the notes):
 |  |  |  |  |  |  |
 |  |  |  |  |  |  |
 |  |  |  |  |  |  |
 |  |  |
Towards an evaluation instrument
The remarks from the text analysis were wrapped up in a table with success factors and effects in de first column and indicators in de second column. Later on, I added a column "evaluation method", with questions to ask and observables to look for. The questions were derived from interviews, in particular the interview with Marvin Weisbord brought some crucial questions to ask "after the dance", and furthermore from my experience as an auditor of management systems (ISO 9000, ISO 14001, Total Quality systems).
The table has to be (and was) checked and improved in the case study. See the latest version of the table evaluation instrument.
Summary of success factors and effects
1. Context: LSI is the right approach
1.1. The task is important
1.2. Stakeholders need each other to succeed with this task
1.3. Complex and/or uncertain situation
2. Client: Leaders support the LSI approach
2.1. Leaders are willing to collaborate, to share power
2.2. Leaders are willing to spent time and money to do it "by the book"
2.3. Political climate: enough trust to start
3. Consultant: Facilators are skilled to conduct an LSI
3.1. Facilitators make and keep a clear contract with the client
3.2. Facilitators gain credibility, managing expectations
3.3. Facilitators are aware of their own role
3.4. Facilitaors have skills to work with large groups in a "hands-off" approach
3.5. Facilitators believe in the principles of LSI
4. Intervention: LSI is performed right
4.1. LGI is part of a larger effort
4.2. All essential decisions regarding design, management and logistics are made with a planning group
4.3. Design is coherent with context, task, relations and directions
4.4. Participants: getting the right people in the room
4.5. Representation: consideration to those who are not present
4.6. Conference enables everyone's contribution (inclusiveness, building trust)
4.7 Divergence: explore the whole before fixing any part, engaging new connections, building a common database
4.8. Leadership is distriuted by shared responsibility and self-management
4.9. Convergence: Tapping into collective intelligence, looking for common grond
4.10. Conference setting facilitates the process, the setting symbolizes the principles of LSI
4.11. Action planning for next steps is done in the conference, or soon after
4.12. Reflection on conditions and principles during the conference
4.13. Building of capacity to work participatory
4.14. Event is managed well by the facilitators
4.15. Building of a post-event support structure
5. Non-sustainable effects: LSI contributes to getting more and better work done
5.1. Short term objectives are met
5.2. Increased awareness and understaning of the system and context
5.3. Commitment and energy for change, better implementation
5.4. New relationships, more potential for innovation and learning
5.5. Elements of LSI are transferred
5.6. Increased efficiency
6. Effectiveness: Sustainable effects; LSI contributes to transformed capability for change and learning
6.1. Collective learning and changing continue, increased capability to change
6.2. Increased reflective self awareness
6.3. More permeable boundaries: Opening up the organization, inviting diversity; focus on how good the system is
6.4. New structures sustain ongoing participation in change
6.5 Communication is more direct and constructive
Sources of text analysis, looking for criteria and indicators
Prototypes of the evaluation instrument
The prototype evaluation instrument consists of three documents:
- A table evaluation instrument with the success factors and effects, with indicators and were to look for evidence
- An audit matrix derived from the table with questions and key persons/places/documents to look for evidence
- A detailed score table, to score and evaluate the audit results; there is a detailed version (for consultants) and an overview score table for clients and potential clients.
Towards a practical guide for effective use of LSI
All the results are wrapped up in a draft of "A practical guide for effective use for LSI". It contains "Client Information Leaflet" and guidelines for Experts (consultants and researchers), how to assess and evaluate the process before, during and after an LSI. See practical guide
