Data collection and Analysis The three concerns about
data collection are: what data should we collect- how
should we collect it and analyze it; and who should do it. If the community or
organization has approached the consultant for help there will be a “presenting
problem” expressed. This problem, which may or may not be the real problem, is
the logical place to start in framing issues that should be investigated in the
data collection process. But if it is an intervener initiated process then the
ICC model suggests that the target system identify the data that should be
collected. The consultant will assist m suggesting profitable areas to
investigate or supply several usual tools for stimulation (see the community,
group and organization scan guides in this section) but the final selection of
issues will come from the participants. Figure 10 reviews Methods of Data Collection and contrasts some of their strengths and weaknesses. The most frequently used methods of collecting data are individual interview and survey questionnaires. However group interview methods are more compatible with the ICC model as they are the most collaborative and involving. They also provide immediate feedback for the members of the interview group, and do some team building with them at the same time. Group interviews get participants sharing data and analyzing its importance very quickly and are thus a great way to start modeling the ICC method.
A further review of the
'how to collect data' methods in Figure 10 suggests how many of the methods can
be combined to strengthen the usefulness of data collected from only one
method. For example, adding interviews to the usual questionnaire method
increases depth and ownership. And adding a document and record analysis
provides a historical perspective based on publicly available material. Of
course checking whatever you have found though the formal data collection with
your own experiences so far in the organization-gives an outsider's
perspective. Tuning into your feelings about your own experiences also helps
you empathize with what others in the system are likely feeling. Combining any
three methods or sources of data is called Triangulation and is thought to be
one of the best ways of increasing data credibility. Collaboration is the key
word in deciding who should be involved m collecting data. The concept of
optional involvement suggests balancing. Everyone should be involved in both
collecting the data, with providing the data with what is practical, credible
and cost effective. Collaboration-wise it is great for everyone to be
interviewing each other and then summarizing it together. But with large
numbers of people with no interviewing skills, and who don’t trust each other
enough to say anything of importance, whatever is collected would be suspect.
It might be of little help in identifying the real issues and moving the
intervention forward. There are a number of procedures which seek optimal
levels of involvement through organization, structure, and training
participants in data collection skills. A few of the methods that are the best
compromises of involvement with practicality are described below. Selected
procedures
1. SYSTEMS
IMPROVEMENT RESEARCH The systems Improvement Research Model ( Dimock, 1978,
1979, 1981) is based on a collaborative approach to data collection which joins
the external consultants with the internal participants as a data collection
and analysis team. The team proposes the areas they want studied or the
questions they want answered and using the consultants as special re-sources
prepare the data collection tool (questionnaire, interview schedule or
observation guide). The team then divides up to collect the data usually
pairing an insider and an outsider (consultant) to conduct an interview,
observe a meeting, or distribute a questionnaire. The consultants may take on a
training/coaching role in framing the insiders in data collection methods. But
the primary reason for pairing an insider and outsider is to join the strength
of the consultant's objective point of view and conceptual framework to help
organize the data with the insider's understanding of group norms and ability
to understand hidden meanings or subtle points in the data. Thus outside
objectivity and perspective is coupled with the relevant biases and historical
interpretations of the insider. Some respondents find it easier to talk to an
insider while others prefer an outsider and pairing data collectors is a useful
way of meeting some of both sets of needs. The other important aspect of the SIR method, with
relevance to data collection, is the powerful impact of immediate feedback of
the data into the system. The immediate feedback capitalizes on the energy
generated by the data collection process and eliminates any possibility that
the data can be discounted by labeling it stale or out of date. Often, the
immediate sharing of data facilitates a continuing discussion and exploration
of the areas under study which adds to the quality of the data available. This
is a technique adopted from interview methods where the interviewer summarizes
the points of view of the respondent and usually finds that the summary
generates further thinking and responses from the interviewee.
2. INTEREST
CENSUS The most basic group method of data collection is the interest census where people come together in their natural groups (preferably between 5 and 60 or 80 people) and are asked about their interests or problems related to the overall goals of the intervention. This process is usually facilitated by asking people at the opening of such a meeting to take a moment and list their interest. Sample questions include:
As people complete a brief list they are asked to join
a small group (5 to 7 people), explore the ideas of the participants,
brainstorm new interests, and finally make up a list of high priority items to
share with the other groups. As the recorder/reporters from each of the small
groups presents 3 or 4 items of greatest importance to their group a master
list is prepared (usually on flip chart sheets which are taped on the wall). When all the groups have reported, the consultant and
the group summarize the items under appropriate headings or categories. These
may be color coded or given a specific category title. When the categories are
generally agreed upon and a check is made that the input of all the participants
shows up somewhere, the group establishes the top priority categories for
immediate consideration and possible action planning. The master list is
reproduced and distributed to be used later as a checking point for progress.
3. SWOT
ANALYSIS SWOT stands for Strengths, Weaknesses, Opportunities
and Threats. It is an extension of the force-field analysis model described as
the eighth assumption about changing social systems. Describing the present
strengths and weaknesses of the internal system is preceded by describing the
perceived opportunities and threats in the external system. It is harder to
list the opportunities and threats to your community or agency from outside
forces such as economic conditions and government regulations so it is done first.
A systematic external and internal environmental scan provides a powerful
framework within which to set goals and plan relevant actions. A SWOT analysis clarifies the tensions that exist
between the present performance of a system and its goals. This is done by
contrasting two basic dimensions: good vs. bad (strengths and opportunities
with weaknesses and threats) and present vs. future (strengths and weaknesses
with opportunities and threats). The procedure used to do the SWOT analysis can
be the same as the one used in the Interest Census (#2 above) but each of the
four components should be made a separate question and activity. A summary of
the completed activity can show the top three items for each of the four
questions on the following template put on a large piece of newsprint or an
overhead projector. This identifies the combinations most likely to produce
results and what needs to be done to make the "consider" and
"maybe" combinations possible. Figure 11 puts the data collected from the SWOT analysis into perspective. It suggests the most likely activities for immediate success—the combination of strengths and opportunities appearing in the top, right hand go quadrant. And it clarifies that the strengths listed in the consider quad-rant have external restraints that are likely to reduce their success. These restraints such as changes in government priorities, changes in consumer interests, or an economic downturn, are best handled by planning action strategies to reduce their impact. The present weaknesses to implementing the opportunities found in the maybe quadrant may be overcome with additional personnel, equipment or facilities. Or, as would be the case in a change in service delivery from individual treatment to group and community oriented prevention, a change in the kind of staff needed and probably new locations and layouts for facilities.
4. STAKEHOLDER ANALYSIS The second most important assumption mentioned earlier
about changing social systems was: "Those people affected by a change
should be involved in making that change." This sounds great in the
abstract but who are these people and how might they be impacted by the program
you are planning? A stakeholder is any person, group or organization who has
claim on (or serious interest in) the activities and well-being of the system
targeted for change.
In a community, stakeholders include all full time and
part-time residents, all business and absentee landlords, related governing
bodies, unions, and political parties. The stakeholders of a community serving
organization include present and former consumers of services, founders,
employees' unions, boards (trustees), related governments, organizations
providing parallel or complimentary services and suppliers of equipment and
services. The phases of a stakeholder's analysis are to identify
and list the stakeholders, note their stake in the system's activities and
well-being, hunch the criteria they would use to assess the system's
effectiveness, and then make a guesstimate of what their evaluation would be.
If the focus is more specifically on the proposed intervention or development
program the last question would ask for a guesstimate of what their likely
reaction to the project would be at this time. A stakeholder analysis concludes with a review of the
data focusing on:
The final step asks for an action plan to head off or
respond to key stakeholders' concerns, or to identify new ways to involve them
in the planning process so that their 'stake' can receive consideration.
Analyzing a System's Culture
The focus of change using the ICC method is the
culture of the system targeted for change. The culture of a system has been
described as its 'usual ways of doing things' and consists of norms or
standards, values and beliefs, and the mythology and folklore built up over
time. Culture is nebulous, hard to get a handle on, and is revealed like an
onion, one layer at a time. Interveners, if they are new to the system, cannot
understand the subtle meanings of the insiders
descriptions of culture—they are left with the surface layers. Also, insiders
may not trust interveners with a whole picture even if they could describe it
themselves. Often basic 'usual ways of doing things' are taken for granted and
are difficult to think of as specific data. The newcomer can usually uncover the first layer by
observing the usual ways things happen here and contrasting it with other
systems. Unusual norms or idiosyncrasies of the system are much more easily
identified by the newcomer than the old-timer who takes it all for granted. As
the intervener gains trust and acceptance from the insiders they can work
together to bring into consciousness the forgotten understandings of why things
that puzzle the intervener happen the way they do. Group norms are statements of common and acceptable
ways of doing things. People who have high status and acceptance in the system
usually model these norms. For anyone investigating the culture of a system
these people (usually they are leaders) are worthy of systematic observation
and study.
The Culture Analysis
Questionnaire or Interview Guide and the Observation Guide
shown above will collect a great deal of information, especially if they
are used for group participation. If the questionnaire approach seems the way
to go, general meetings of stakeholders can be organized. At these meetings
participants complete the questionnaire as individuals, then
form groups of 4-6 people to analyze and summarize the results. As in the
Interest Census technique described as Selected Procedures #2 above,
recorder/reporters are assigned or selected by each group to report the most
significant findings to the total group. While a complete list may be made of
the work group reports for careful study later, the most frequently identified
significant findings are summarized for the total group. Perhaps too, the total
group winds up the session by discussing why these findings are the most
important and what they are really saying about the system's culture.
If the questionnaire is to
be used as an interview guide, participation and ownership of the data can be
optimized by using the group interview technique (interviews with 3-6 people
together). Or teams of interviewers, perhaps working in pairs, can do the
interviews and then get together to discuss and summarize the learnings using
the format of the general meeting described for the questionnaire. And lastly,
if the identified stake-holder's number is less than 100, they can be paired up
with each person interviewing the other and then have the general meeting with
small groups to discuss, and summarize the information.
Similar 'maximizing participation' procedures can be
used with the observation guide with a large team of observers, or everyone
being an observer for a week and then coming to a general meeting where they
form small groups. Or pairs of observers can be formed using the Systems
Improvement Research method with one insider paired with an outsider, or an
old-timer paired with a newcomer.
Process Goals in Data Collection Involving the people of the identified system in the
process of collecting and analyzing data increases their commitment to the
change program. But the process of interviewing people or framing
questionnaires and discussing the findings with others is also a very powerful
educational experience. Participants learn new ways of looking at the problem
and may change some of their own attitudes and beliefs. And, best of all, since
attitudes are sustained by the culture of the groups to which a person belongs,
the discussions with the group can lead to modifications of its culture. The
effective re-education of an individual requires shifts in the norms of closely
related groups. These insights formed the
original concept of Action Research which integrated personal re-education and
social change into the same process. Changes in the usual ways of doing things
came about as the stakeholder in the problem worked in a cooperative action research
method to study the problem and tried simultaneously some new ways of working
that might alleviate it. For example, if the identified problem was the low
status of women in the community or organization, their inability to be heard
and taken seriously by men, we can contrast two approaches to illustrate action
research. First, let's look at the usual research approach which posits that
accurate information will change males' attitudes and behaviour toward women.
Women in the system would be interviewed about their experiences by other
women. As women, especially in the system under study, talk differently to each
other, they would be more likely to reveal personal experiences and thus make
the data collected more credible. The action research
approach would start working on the problem as part of the data collection
process, believing that changes in attitudes and behaviour come about through
experiential activities focusing on group norms. Both men and women would
interview women and then compare and contrast the results. General meetings
would convene to discuss the results and the differences. Small groups
separating men and women would start the analysis at the general meeting and
then reform with half men and women to compare notes. All would be working
together to understand the role and status of women and trying to figure out
what was supporting and restraining present practices. In the process, women
would be talking to men about real issues, and the men would be working at
understanding in new ways their input. Roles and relationships, and the norms
supporting present practices, would change as action research activities put
the new attitudes and ways of working into practice.
Participative Data Analysis and Reporting There may be programs where the intervener feels it
would not be practical for the stakeholders to be involved in the collection of
the data necessary. Or, existing data may be motivation for the program such as
a recent census showing a rapid growth trend for the community. However, as the
program moves into analyzing the meaning of the new information and informing
others about it, it is imperative that participation be optimized or the
collaboration component of the ICC method will be lost. The intervener or
intervention team will likely do their own analysis of the information
collected as an outside perspective is usually helpful. Most analysis, however,
will be done by individual consumers, study groups, and community meetings
called for that purpose. Stakeholders will report their findings to one another
with the intervener facilitating the process. In no way will the consultant lay
on an analysis of the data with a diagnosis of the problem. Rather the analysis
and suggestion of related factors will be done by the program participants. The usual data analysis
procedures consist of the participants summarizing the data and then looking
for themes among them. Usually it is helpful to put the data into categories or
at least dusters of similar information. With my bias for the force field
analysis technique, data would divide into helping and hindering factors,
strengths and weaknesses pros or cons, or winners and losers dichotomies.
Following this division of the data, the participants would examine the two
halves looking for general characteristics which were true for one half but not
the other Once these characteristic differences between the halves were
identified the group would hunch about any generalizable principles which
accounted for the differences. And lastly, analysis procedures would seek to
establish the consistency and confirmability of the data by the triangulation
method. Data from three different sources would be looked for to establish the
credibility of any major theme. (These analysis procedures are described more
fully in Dimock, 1987b.)
... Next Section - Goal setting
|