Data collection and Analysis

The three concerns about data collection are: what data should we collect- how should we collect it and analyze it; and who should do it. If the community or organization has approached the consultant for help there will be a “presenting problem” expressed. This problem, which may or may not be the real problem, is the logical place to start in framing issues that should be investigated in the data collection process. But if it is an intervener initiated process then the ICC model suggests that the target system identify the data that should be collected. The consultant will assist m suggesting profitable areas to investigate or supply several usual tools for stimulation (see the community, group and organization scan guides in this section) but the final selection of issues will come from the participants.

Figure 10 reviews Methods of Data Collection and contrasts some of their strengths and weaknesses. The most frequently used methods of collecting data are individual interview and survey questionnaires. However group interview methods are more compatible with the ICC model as they are the most collaborative and involving. They also provide immediate feedback for the members of the interview group, and do some team building with them at the same time. Group interviews get participants sharing data and analyzing its importance very quickly and are thus a great way to start modeling the ICC method.    

 

A further review of the 'how to collect data' methods in Figure 10 suggests how many of the methods can be combined to strengthen the usefulness of data collected from only one method. For example, adding interviews to the usual questionnaire method increases depth and ownership. And adding a document and record analysis provides a historical perspective based on publicly available material. Of course checking whatever you have found though the formal data collection with your own experiences so far in the organization-gives an outsider's perspective. Tuning into your feelings about your own experiences also helps you empathize with what others in the system are likely feeling. Combining any three methods or sources of data is called Triangulation and is thought to be one of the best ways of increasing data credibility.

Collaboration is the key word in deciding who should be involved m collecting data. The concept of optional involvement suggests balancing. Everyone should be involved in both collecting the data, with providing the data with what is practical, credible and cost effective. Collaboration-wise it is great for everyone to be interviewing each other and then summarizing it together. But with large numbers of people with no interviewing skills, and who don’t trust each other enough to say anything of importance, whatever is collected would be suspect. It might be of little help in identifying the real issues and moving the intervention forward. There are a number of procedures which seek optimal levels of involvement through organization, structure, and training participants in data collection skills. A few of the methods that are the best compromises of involvement with practicality are described below.

 

Selected procedures

1.         SYSTEMS IMPROVEMENT RESEARCH

The systems Improvement Research Model ( Dimock, 1978, 1979, 1981) is based on a collaborative approach to data collection which joins the external consultants with the internal participants as a data collection and analysis team. The team proposes the areas they want studied or the questions they want answered and using the consultants as special re-sources prepare the data collection tool (questionnaire, interview schedule or observation guide). The team then divides up to collect the data usually pairing an insider and an outsider (consultant) to conduct an interview, observe a meeting, or distribute a questionnaire. The consultants may take on a training/coaching role in framing the insiders in data collection methods. But the primary reason for pairing an insider and outsider is to join the strength of the consultant's objective point of view and conceptual framework to help organize the data with the insider's understanding of group norms and ability to understand hidden meanings or subtle points in the data. Thus outside objectivity and perspective is coupled with the relevant biases and historical interpretations of the insider. Some respondents find it easier to talk to an insider while others prefer an outsider and pairing data collectors is a useful way of meeting some of both sets of needs.

The other important aspect of the SIR method, with relevance to data collection, is the powerful impact of immediate feedback of the data into the system. The immediate feedback capitalizes on the energy generated by the data collection process and eliminates any possibility that the data can be discounted by labeling it stale or out of date. Often, the immediate sharing of data facilitates a continuing discussion and exploration of the areas under study which adds to the quality of the data available. This is a technique adopted from interview methods where the interviewer summarizes the points of view of the respondent and usually finds that the summary generates further thinking and responses from the interviewee.

 

2.         INTEREST CENSUS

The most basic group method of data collection is the interest census where people come together in their natural groups (preferably between 5 and 60 or 80 people) and are asked about their interests or problems related to the overall goals of the intervention. This process is usually facilitated by asking people at the opening of such a meeting to take a moment and list their interest. Sample questions include:

  • What interests do you have that you think would increase the effectiveness of your work?

  • What are the major issues in this community that we should give attention to in the next two years?

  • List the ideas you have to deal with the budget cutbacks that have been thrust upon us.

  • What problems would you like to work on in the forthcoming intervention program?

  • List the changes that would please you personally and also benefit the organization.

As people complete a brief list they are asked to join a small group (5 to 7 people), explore the ideas of the participants, brainstorm new interests, and finally make up a list of high priority items to share with the other groups. As the recorder/reporters from each of the small groups presents 3 or 4 items of greatest importance to their group a master list is prepared (usually on flip chart sheets which are taped on the wall).

When all the groups have reported, the consultant and the group summarize the items under appropriate headings or categories. These may be color coded or given a specific category title. When the categories are generally agreed upon and a check is made that the input of all the participants shows up somewhere, the group establishes the top priority categories for immediate consideration and possible action planning. The master list is reproduced and distributed to be used later as a checking point for progress.

  

3.         SWOT ANALYSIS

SWOT stands for Strengths, Weaknesses, Opportunities and Threats. It is an extension of the force-field analysis model described as the eighth assumption about changing social systems. Describing the present strengths and weaknesses of the internal system is preceded by describing the perceived opportunities and threats in the external system. It is harder to list the opportunities and threats to your community or agency from outside forces such as economic conditions and government regulations so it is done first. A systematic external and internal environmental scan provides a powerful framework within which to set goals and plan relevant actions.

A SWOT analysis clarifies the tensions that exist between the present performance of a system and its goals. This is done by contrasting two basic dimensions: good vs. bad (strengths and opportunities with weaknesses and threats) and present vs. future (strengths and weaknesses with opportunities and threats). The procedure used to do the SWOT analysis can be the same as the one used in the Interest Census (#2 above) but each of the four components should be made a separate question and activity. A summary of the completed activity can show the top three items for each of the four questions on the following template put on a large piece of newsprint or an overhead projector. This identifies the combinations most likely to produce results and what needs to be done to make the "consider" and "maybe" combinations possible.

Figure 11 puts the data collected from the SWOT analysis into perspective. It suggests the most likely activities for immediate success—the combination of strengths and opportunities appearing in the top, right hand go quadrant. And it clarifies that the strengths listed in the consider quad-rant have external restraints that are likely to reduce their success. These restraints such as changes in government priorities, changes in consumer interests, or an economic downturn, are best handled by planning action strategies to reduce their impact. The present weaknesses to implementing the opportunities found in the maybe quadrant may be overcome with additional personnel, equipment or facilities. Or, as would be the case in a change in service delivery from individual treatment to group and community oriented prevention, a change in the kind of staff needed and probably new locations and layouts for facilities.

      

4.         STAKEHOLDER ANALYSIS

The second most important assumption mentioned earlier about changing social systems was: "Those people affected by a change should be involved in making that change." This sounds great in the abstract but who are these people and how might they be impacted by the program you are planning? A stakeholder is any person, group or organization who has claim on (or serious interest in) the activities and well-being of the system targeted for change.

FIGURE 12

 

Stakeholder Analysis

Stakeholder

(person, group, organization)

Give/Get

Likely Evaluation of us now

Likely reaction to proposed development program

Taxpayers (residents)

Physical education, recreation, protection & social services [Country Style – moderate taxes]

Good to very good. Want more services but no more taxes.

Mixed. Desire systematic plan. Want rural life but want to sever lots to sell.  Don’t want commerce or gravel or land fill near by.

Seasonal Residents

Roads, recreation, protection

Very good. Love untaxed status.

Trailer park seasonals want to stay all year and pay little or no tax. No sewerage upgrading.

Churches

Protection of property. [Opportunity to influence community well-being.]

Very good Churches holding own member-wise.

Freedom to expand retreat centres and graveyards. Insist on rural character. Against any development.

Etc.  . . .

 

 

 

Abridged sample of stakeholder analysis done by Town Council of rural community of Puslinch, Ontario while planning a long range program for land use in township.

 

In a community, stakeholders include all full time and part-time residents, all business and absentee landlords, related governing bodies, unions, and political parties. The stakeholders of a community serving organization include present and former consumers of services, founders, employees' unions, boards (trustees), related governments, organizations providing parallel or complimentary services and suppliers of equipment and services.

The phases of a stakeholder's analysis are to identify and list the stakeholders, note their stake in the system's activities and well-being, hunch the criteria they would use to assess the system's effectiveness, and then make a guesstimate of what their evaluation would be. If the focus is more specifically on the proposed intervention or development program the last question would ask for a guesstimate of what their likely reaction to the project would be at this time.

A stakeholder analysis concludes with a review of the data focusing on:

  1. Who are major stakeholders—the minor ones?

  2. What is it they want from us (or what can we give them)?

  3. How do they likely evaluate our operation at this time?

  4. How will they react to the proposed intervention or development program.

The final step asks for an action plan to head off or respond to key stakeholders' concerns, or to identify new ways to involve them in the planning process so that their 'stake' can receive consideration.

 

Analyzing a System's Culture

The focus of change using the ICC method is the culture of the system targeted for change. The culture of a system has been described as its 'usual ways of doing things' and consists of norms or standards, values and beliefs, and the mythology and folklore built up over time. Culture is nebulous, hard to get a handle on, and is revealed like an onion, one layer at a time. Interveners, if they are new to the system, cannot understand the subtle meanings of the insiders descriptions of culture—they are left with the surface layers. Also, insiders may not trust interveners with a whole picture even if they could describe it themselves. Often basic 'usual ways of doing things' are taken for granted and are difficult to think of as specific data.

The newcomer can usually uncover the first layer by observing the usual ways things happen here and contrasting it with other systems. Unusual norms or idiosyncrasies of the system are much more easily identified by the newcomer than the old-timer who takes it all for granted. As the intervener gains trust and acceptance from the insiders they can work together to bring into consciousness the forgotten understandings of why things that puzzle the intervener happen the way they do.

Group norms are statements of common and acceptable ways of doing things. People who have high status and acceptance in the system usually model these norms. For anyone investigating the culture of a system these people (usually they are leaders) are worthy of systematic observation and study.

 

FIGURE 13

Questionnaire and interview Guide:
Culture Analysis of a Community, Croup or Organization

  1. Your 2 best friends are joining this system tomorrow. What 3 things  would you tell them they have to do to get ahead?

  2. What 3 things would you tell them not to do in order to get ahead and gain acceptance?

  3. List 3 adjectives (descriptive words) that are most descriptive of this  system (community, group or organization).

  4. What are the 3 most prevalent behaviours of the leaders here?

  5. What are the rituals and celebrations here (how are new people introduced into the system, achievements celebrated, ghosts and failures buried)?

  6. What activities here have the highest status?

  7. What is the political culture of this system? Attitudes regarding, mental and physical ability, sex, race and religion. Styles of power and control; (autocratic, paternalistic, participative) political orientation (radical, posturing liberal, conservative, reactionary).

  8. Culture is embedded in history so it is helpful to know the folklore and history related to your system. Acceptance of this folklore often blocks change projects.
    Who were the good guys and the bad guys, the winners and the losers and what is the folklore about what they did? What were the outcomes of previous change projects?

The Culture Analysis Questionnaire or Interview Guide and the Observation Guide shown above will collect a great deal of information, especially if they are used for group participation. If the questionnaire approach seems the way to go, general meetings of stakeholders can be organized. At these meetings participants complete the questionnaire as individuals, then form groups of 4-6 people to analyze and summarize the results. As in the Interest Census technique described as Selected Procedures #2 above, recorder/reporters are assigned or selected by each group to report the most significant findings to the total group. While a complete list may be made of the work group reports for careful study later, the most frequently identified significant findings are summarized for the total group. Perhaps too, the total group winds up the session by discussing why these findings are the most important and what they are really saying about the system's culture.

 

FIGURE 14

Culture Analysis Observation Guide

The goal is to identify the unwritten rules that determine the usual ways in which things work in this community, group or organization.

  • What behaviour or activity gets rewarded? What gets demerits?
  • What are the major sources of anxiety and concern?
  • What are the norms for dress? Promptness? Attendance? Performance? Deadlines?
  • What are the practices for handling routines, lateness, deadlines, absences, and poor performance?
  • What are expectations for people to participate in and contribute to the system's well-being? Hi    Med    Lo
  • Amount of support and encouragement given to people here?
                         Hi    Med    Lo
  • Openness/secrecy regarding: income level, competence, performance, promotions or awards, and future plans.
  • What's talked about privately that isn't addressed publicly?
  • How is unacceptable behaviour punished (sarcasm, freezing out, whispering, confrontation, rejection)?
  • How are people gotten rid of here?
  • How is conflict, aggressive competition, and major disagreement handled?
  • How is space used to symbolize status and power, to maintain privacy, to identify sub-groups?
  • How does the system deal with the external environment—the larger community of which it is part?
  1. Reactive (systems lets itself be controlled)

  2. Proactive (system fries to control its destiny)

  3. Harmonizing (system interacts to work out mutually beneficial results)

  • What appears to be the common hidden agendas of the system's culture?
  • 3 biggest culture supports of the system's goals and well-being are:
  • 3 biggest culture restraints to goal achievement and well-being are:

 

  

If the questionnaire is to be used as an interview guide, participation and ownership of the data can be optimized by using the group interview technique (interviews with 3-6 people together). Or teams of interviewers, perhaps working in pairs, can do the interviews and then get together to discuss and summarize the learnings using the format of the general meeting described for the questionnaire. And lastly, if the identified stake-holder's number is less than 100, they can be paired up with each person interviewing the other and then have the general meeting with small groups to discuss, and summarize the information.

Similar 'maximizing participation' procedures can be used with the observation guide with a large team of observers, or everyone being an observer for a week and then coming to a general meeting where they form small groups. Or pairs of observers can be formed using the Systems Improvement Research method with one insider paired with an outsider, or an old-timer paired with a newcomer.

 

Process Goals in Data Collection

Involving the people of the identified system in the process of collecting and analyzing data increases their commitment to the change program. But the process of interviewing people or framing questionnaires and discussing the findings with others is also a very powerful educational experience. Participants learn new ways of looking at the problem and may change some of their own attitudes and beliefs. And, best of all, since attitudes are sustained by the culture of the groups to which a person belongs, the discussions with the group can lead to modifications of its culture. The effective re-education of an individual requires shifts in the norms of closely related groups.

These insights formed the original concept of Action Research which integrated personal re-education and social change into the same process. Changes in the usual ways of doing things came about as the stakeholder in the problem worked in a cooperative action research method to study the problem and tried simultaneously some new ways of working that might alleviate it. For example, if the identified problem was the low status of women in the community or organization, their inability to be heard and taken seriously by men, we can contrast two approaches to illustrate action research. First, let's look at the usual research approach which posits that accurate information will change males' attitudes and behaviour toward women. Women in the system would be interviewed about their experiences by other women. As women, especially in the system under study, talk differently to each other, they would be more likely to reveal personal experiences and thus make the data collected more credible.

The action research approach would start working on the problem as part of the data collection process, believing that changes in attitudes and behaviour come about through experiential activities focusing on group norms. Both men and women would interview women and then compare and contrast the results. General meetings would convene to discuss the results and the differences. Small groups separating men and women would start the analysis at the general meeting and then reform with half men and women to compare notes. All would be working together to understand the role and status of women and trying to figure out what was supporting and restraining present practices. In the process, women would be talking to men about real issues, and the men would be working at understanding in new ways their input. Roles and relationships, and the norms supporting present practices, would change as action research activities put the new attitudes and ways of working into practice.

 

Participative Data Analysis and Reporting

There may be programs where the intervener feels it would not be practical for the stakeholders to be involved in the collection of the data necessary. Or, existing data may be motivation for the program such as a recent census showing a rapid growth trend for the community. However, as the program moves into analyzing the meaning of the new information and informing others about it, it is imperative that participation be optimized or the collaboration component of the ICC method will be lost.

The intervener or intervention team will likely do their own analysis of the information collected as an outside perspective is usually helpful. Most analysis, however, will be done by individual consumers, study groups, and community meetings called for that purpose. Stakeholders will report their findings to one another with the intervener facilitating the process. In no way will the consultant lay on an analysis of the data with a diagnosis of the problem. Rather the analysis and suggestion of related factors will be done by the program participants.

The usual data analysis procedures consist of the participants summarizing the data and then looking for themes among them. Usually it is helpful to put the data into categories or at least dusters of similar information. With my bias for the force field analysis technique, data would divide into helping and hindering factors, strengths and weaknesses pros or cons, or winners and losers dichotomies. Following this division of the data, the participants would examine the two halves looking for general characteristics which were true for one half but not the other Once these characteristic differences between the halves were identified the group would hunch about any generalizable principles which accounted for the differences. And lastly, analysis procedures would seek to establish the consistency and confirmability of the data by the triangulation method. Data from three different sources would be looked for to establish the credibility of any major theme. (These analysis procedures are described more fully in Dimock, 1987b.)                       

 

... Next Section - Goal setting