Research conferences
From LSI
Two interactive research conferences are held as member check, to validate the evaluation instrument and to discuss the utility and usability of a practical guide for effective use of LSI. Why member checks? Why interactive conferences?
Need for member checks in qualitative research
The prototype of the practical guide identifies success factors and effects with indicators for observable evidence. These topics are derived from the exploring the field phase, including:
- Text analyses of approximately fifty articles with reported LSIs
- Interviews and workshops engaging 35 stakeholders (facilitators and participants of LSI, researchers of interactive interventions)
- Prolonged engagement with practitioners through membership of world wide networks
From the success factors, indicators and questions an evaluation instrumentis developed, comparable to commonly used audit tools in quality management. The evaluation instrument is tested in a case study with an evaluation of three past LSIs. All guidelines and tools are assembled into a prototype of a practical guide. The format of the guide is borrowed from practical guide for a medical treatment of overweight and obesity. I choose this format for two reasons: medicine is a discipline with massive experience in evidence based information leaflets for clients and experts, and the obesity guide is about a complex problem, as is the use of LSI.
Input for the guide came from a lot of sources and people, but the development of the guide is done by a single researcher, applying qualitative research methods. So far, the researcher is the only one who did the analyzing and concluding. A member check is necessary to check the credibility of research process and outcome and to enhance applicability of the guide in practice.
Interactive conferences as member check
In qualitative research, a member check is mostly held by sending report drafts for correction to the people who were interviewed for the research project, or by organizing a meeting with key players to discuss and correct the report. Sometimes “layman” are involved to get more insight in the view of ordinary people, in order to enhance the usability of research outcomes. For instance, in medical research consensus conferences are used. In a consensus conference a panel of lay people discusses the outcomes with experts until they reach consensus, registered in a consensus statement.
For several reasons a regular member check seemed not appropriate for this research project on effective use of LSI:
- A large number of stakeholders, practioners, clients as well as their target groups were involved. It seemed right to invite them to discuss and verify the success factors and effects mentioned in the guide. Are they complete, correct, relevant, and observable?
- For future evaluations of effectiveness of LSI, it is important that the evaluation instrument is adequate and usable as a tool. So researchers with an interest in evaluation research must be invited too.
- The phenomenon of an evidence based “information leaflet” is rather new in consultancy. What is needed to make the guide work? What do potential users see as advantages or disadvantages?
Thus, involvement of a large group of clients, practitioners and researchers is necessary to discuss content as well as usability of the guide. On the motto “practice what you preach”, why not choose large group methods to involve these stakeholders in interactive conferences as member check? A one day live conference with the format of a Search Conference, is held with a group of 40 participants (researchers, consultants and practitioners). For those who were not able to attend the live conference an online conference is organized. For the online conference we used
The concept of the guide for discussion in the member check comprises an Executive Summary with when and how to use LSI, and an Expert Section with facilitation and evaluation guidelines
Planning and design of the live ReSearch conference
The design of the live conference is build on principles and work forms of the large group methods Search Conference, Future Search, Real Time Strategic Change and Open Space Technology. The conference has also a research purpose, so we named it “ReSearch Conference”. This is how main design principles of large group methods are filled in for the ReSearch conference:
- Involve stakeholders in the planning: Preparation by a planning group of academic researchers as well as practitioners, seven people in total
- Invite the whole system in the room: A diversity of stakeholders with an interest in LSI is personally invited: clients of an LSI, researchers of LSI or other interactive intervention approaches, practioners of LSI
- Participants do their own real time data assembling and analysis, building a commonly shared data base: Overviews are made collectively, all outcomes and conclusions are presented on large overviews on the wall; self-management of small groups; plenary reporting of small groups
- Every participant is able to contribute, addressing a diversity of personal capacities: Interactive as much as possible; alternation of individual, small group and plenary work; a diversity of work forms (rational, associative and creative); no long presentations; a comfortable and inviting atmosphere, flowers on the table, beverages permanent available, good food
- Looking for common ground and shared understanding: Conclusions are drawn collectively as much as possible
The ReSearch conference, 17 September 2009
The program steps and outcomes are illustrated in two online photo albums:
photos ReSearch conference morning part
photos ReSearch conference afternoon part
Online conference on 1 October 2009
Under construction
Analyses of results
Analyzing method
The results are processed with a qualitative data analysis method as used to form grounded theory, following the steps as shown in this figure:
Analyzing procedure; adapted from the figure “The Stages of Learning Histories” of Art Kleiner
All data are translated into “building blocks”, units of text that can stand on their own, representing a specific meaning. The building blocks are labelled with a code of their source and assembled in a digital table, using a first rough categorization. In several rounds the building blocks are compared and arranged around emerging themes. Conclusions and memos are added in the right column of the table. The analyses produced 423 building blocks, spread over 24 categories. The table is included in Appendix 2 of the report of the conferences.
The process of distillation leads to conclusions and recommendations, detailed corrections and additions of the guide and the list of success factors and effects, see the report of the conferences. The suggested corrections and additions are will be integrated in the next version of the guide.
Conclusions research conferences
Main themes and conclusions
- The role of the leaders and expectation management by consultants are seen as the most important success factors
- The paradox of sustainable change and planned follow up: Sustainable change is an oxymoron, a figure of speech that combines normally contradictory terms. You try to create a transformation, to create a system you believe in, at the same time you have to iterate that interaction in each present as repetition or habit.
- There is no consensus about planning of follow up beforehand and the role of the consultant in the follow up; advocates of planning follow up state that the whole LSI and the follow up should be embedded in a continuous development vision; opponents state that the LSI is the change, a turning point; if people are empowered they will take responsibility and organize the follow up themselves
- We have to live and work with a paradoxically causality. In LSI working participatory is both a condition and a result. There is a paradox in the conditions you need for LSI (enough trust to show up and communicate with each other), the complex situations LSI is indicated for (inherent conflicts and different interests) and the intended effects (more trust and better communication).
- This paradoxical causality does not match with linear academic research traditions and linear measuring of cause-effect relations; sophisticated qualitative research methods are needed to deal with non-linear processes, acknowledging multi-variable complexity, and involve facts as well as ideas and feelings.
- Do we need a scientific approach? Some say No, we already know that it works, others say Yes, it offers a framework for evaluation and conversations during the trajectory; see table 2 for a summary of Pros and Contras of an evidence based guide.
- Evaluation should be done with more than one person. Who should perform the evaluation of an LSI and how depends on the goal of the evaluation; evaluation should be discussed beforehand and built into the contract
- The guide is a tool to make explicit what a good facilitator does implicitly, thus helping those who are less familiar with LSI in facilitating; it can also be used in other scientific fields.
- How extensive should the guide be? The guide is meant as a framework, to be filled in for specific methods and situations. How long the guide should be depends on what it is used for.
- For embedding of the guide, different forms and types (regarding look and feel) for different purposes and user groups are recommended. Suggestion: Offer the guide as “open source” on the web, so everyone can contribute with stories, How To, best and bad practices etc.
Pros and contras of a practical guide for effective use of LSI
Pro: The practical guide establishes common ground across Large Group methods, through convergence at the level of principles
- Facilitates dialogue about LSI and its effectiveness
- We can use it to relate to other fields
- Brings research thinking and practice together
- An attempt to validate the field
Pro: The practical guide has potential to help both client and facilitator to improve the quality of an LSI
- Helps to convince others
- Framework for assessment of preconditions, to develop a good contract
- Helps in expectation management
- Framework for evaluation
- Tool for training of consultants
- Facilitates learning of past projects, to improve our practice
- Tool for assessment of facilitators
Contras:
- We don’t need the rigor of a scientific approach:
- Reduces openness and flexibility when the guide is misused as too steering, too normative
- Kill of enthusiasm by a too cold scientific approach
- Risk of losing the essence of the art of facilitating
- We already know it works
- Clients and consultants might not be interested in evaluation, because they have an interest in an image of success
Dilemma: The history of science is in the way. Research is often seen as exclusive for linear cause-effect processes:
- It must be very clear what causality is adopted in this research, including the worldview it is based on and the consequences for evaluation of effectiveness
- Measuring is a linear process, that doesn’t fit the spiral/circular change process, and success always has more fathers; effects should always be described in terms of “the LSI contributed to ...."
Conclusions on conferences as member check
What do participants think of the conferences? A brief summary presented as lessons learned:
Lessons from the live conference:
- The way we are doing this is an innovation for science
- We’re in this together, we’re learning all together
- We practice what we preach by doing this, learning from stakeholders, a form of professionalizing
- This conference makes discussion possible.
- What an almost impossible task we asked participants of the research conference. We should have given much more attention to introduce them into the assumption and choices made, the considerations, arguments. More space for explanation of context and goals of the research.
- In general take more time to learn to know each other, to connect to each other and to the subject. More time for plenary discussion, less directions and controls
- In addition to collective data gathering: do the data analysis also collectively; give more voice to participants in converging, less role and control by facilitators
- Ask more explicit for commitment to participation after confirming (what does it mean when you say yes and don’t show up without any notification)
- Invite more clients, they were relatively under represented
Lessons from the online conference:
- The terrific organization is appreciated, a model for what is needed to support good collective thinking
- It is an intensive and tiring experience, but although it is not as good as a three day live meeting: it works.
- Plan more time for topic selection and/or ask participants to prepare their topics in advance
- We have to learn how to “chat”: use short sentences, abbreviations
- It is hard sometimes to follow the line of thought, with several threads intertwining
- Organize convergence in another way
Full report of the research conferences
Download the report of the conferences
