C. J. Lokan

School of Computer Science

Australian Defence Force Academy

University of New South Wales

Canberra, Australia

+61 2 6268 8060

c-lokan@adfa.edu.au

In function point analysis (FPA), fourteen "general systems characteristics" (GSC’s) are used to construct a "value adjustment factor" (VAF), with which a basic function point count is adjusted. This stage of function point analysis is controversial.

The general systems characteristics are intended to measure general aspects of size, as opposed to application-specific size [1]. Intuitively, they are used to adjust a function point count so as to better predict effort.

Criticisms of the GSC’s and VAF are both theoretical and practical. Theoretical criticisms are that the construction of the VAF involves operations that are inadmissable according to measurement theory [2]; since "complexity" appears in computing unadjusted function points and again in the GSC’s, there is some double counting [3,4]; and the GSC’s are inter-related [3,5].

Practical criticisms are that not all of the right things are counted as GSC’s [4]; when computing the VAF it is not appropriate to give all of the GSC’s the same weight [4]; the VAF does not provide enough variation [4]; and using the VAF does not improve effort estimation [3,6,7,8].

The adjustment phase of FPA is widely used despite these concerns. It makes intuitive sense to practitioners, whose experience is that the GSC’s are things which influence project difficulty and effort.

An informed decision is needed on whether the GSC’s and VAF are really worth using. If they bring no practical benefits, they should be abandoned. If there are practical benefits, the decision arises of whether or not to use them in spite of theoretical objections. Empirical research is needed to inform this decision.

A study by Desharnais [5] found that only some of the GSC’s were relevant in the adjustment process. He proposed a modified adjustment method, in which different GSC’s have different weights. Several authors have investigated the VAF, finding consistently that it is not effective for improving effort estimation [3,6,7,8].

This paper describes a new empirical study of the FPA adjustment process. A collection of 235 projects is analyzed. The initial focus is on the GSC’s alone, without regard to their use in the VAF. Questions investigated are: how the GSC’s are used overall; how their use varies in different types of projects; and what relationships exist between them. Attention then turns to the adjustment process, to see whether the findings of previous researchers are confirmed in this much larger data set.

Section 2 describes the data set analyzed here. Section 3 looks at the GSC’s individually, describing the distribution of each GSC and noting how the distributions vary for different types of projects. Section 4 investigates relationships between the GSC’s. Section 5 evaluates the VAF. Section 6 presents conclusions.

The projects analyzed here come from the International Software Benchmarking Standards Group ("ISBSG") Repository [9]. This is a public repository of data about completed software projects. The projects cover a wide range of applications, development techniques and tools, implementation languages, and platforms. ISBSG believes that they are representative of better software development projects worldwide.

At the time of this analysis, the repository contained data on 421 projects. After those which were unsuitable for study were excluded (for example, some are measured with Mark II function points [10], which do not use the GSC’s), a sample of 235 projects remained.

The following data was available for each project in the sample: size, measured using the IFPUG standard [1] for function points; the effort expended by the development team; all fourteen GSC values; and project attributes such as application type, development platform, programming language, and date of implementation.

Table 1 summarizes the ranges of function points, effort, and delivery rate.

Each characteristic is scored on a six-point ordinal scale. A score of zero means "not relevant", and scores of 1 through 5 (5 is high) indicate the importance of the characteristic. IFPUG provides guidelines for determining the scores 1].

Table 2 shows the distribution of scores observed for each characteristic. These distributions give some impression of the nature of the projects. A range of profiles can be seen: mostly high values (eg on-line data entry, data communication); mostly low values (eg distributed data processing); mostly average values (eg complex processing); and a fairly even spread (eg transaction rate).

One might expect different characteristics to matter more for some types of software than others. The distributions of GSC values for different types of projects are presented in Tables 5-10 (in the appendix).

Projects are classified by type of organization, business area, application type, project type, programming language level, and development platform. These particular classifications were chosen for two reasons. First, they cover the full range of project attributes, including context, nature, and implementation. Second, in our view the GSC’s are an attempt to explain variations in productivity, and these are the classifications that have been found to be related to productivity in the ISBSG data set [9]. The projects were also classified by date of implementation; the few trends that emerged are noted below.

The tables can be interpreted in two ways. For each characteristic, one can see the types of projects for which they are more or less important (Section 3.1). For each group of projects, one can see which GSC’s matter more and which matter less (Section 3.2). Some interesting trends can be seen from each perspective. These are summarized in Section 3.3

Each section below characterizes the overall distribution of a single GSC (see Table 2).

The distributions of GSC values shown in Tables 5-10 are then analyzed. The chi-squared test was used to identify statistically significant differences between distributions (at the 0.05 significance level). Those differences are described.

Teleprocessing is now almost universal. Only 10% of projects report a "below average" score of 2 or less; 56% have an "above average" score of 4 or 5.

The scores tend to be lower for banking projects, and for projects developed on personal computers. There has been a gradual decline, from above average towards average, from 1991 to 1996.

Of all the GSC’s, this has one of the greatest percentages of "below average" values. The distribution is bimodal: systems tend to be either monolithic, or to have distributed processing as a characteristic of above average importance.

There tends to be more distributed processing in engineering systems. Distributed processing is more common on midrange platforms than on other platforms. It is more common in transaction/production systems and office information systems than in management information systems and decision support systems. It is more important in new developments than in enhancement projects.

This characteristic has a broad spread: scores are below average for 32% of projects, average for 30%, and above average for 38%.

Performance is more important for transaction/production systems than for management information systems. It is more important for new developments than for enhancement projects.

Scores for this characteristic are generally low: 66% below average, 20% average, and 14% above average.

Scores are lower for transaction/production systems and office information systems than for management information systems and decision support systems. They are lower for new developments than for enhancement projects; higher for midrange projects than for other platforms; and higher for engineering systems. Scores increase from 3GL projects to 4GL projects to application generator projects.

Scores for transaction rate are shared across the range from 0 to 4; there are few scores of 5.

Transaction rates are more important than usual for banking systems, and less for engineering systems. They are more important on mainframes than on other platforms. Although one might expect them to be more important for transaction/production systems, there are no significant differences between application types. Scores for this characteristic rise steadily from 1991 through 1996.

This characteristic has by far the highest scores of all, and the least variation. Fully 60% of projects score this characteristic at 5, the maximum value possible.

According to IFPUG’s guidelines [1], a score of 5 means that more than 30% of transactions involve interactive data entry. Perhaps 30% is too low a threshold nowadays; a higher value might provide more useful discrimination.

The scores are lower (generally 3’s) for a block of Cobol/mainframe/banking projects from one organization. The score is 5 for just about everything else.

This characteristic has a broad spread, with a slight tendency towards higher scores: 34% are below average, and 43% are above average.

End user efficiency is more important for management information systems than for transaction/production systems. The scores are lower, and have a flatter distribution, for new developments than for enhancement projects. Again, scores increase from 3GL projects to 4GL projects to application generator projects.

Scores for on-line update tend to be high (half are above average), but they are mostly 3’s and 4’s rather than 5’s.

Scores tend to be higher for transaction/production systems. They are lower on personal computers than on other platforms. Once again, scores increase from 3GL projects to 4GL projects to application generator projects.

This characteristic has a normal (bell-shaped) distribution, centered about the average score of 3. There are few scores of 0 or 5.

Scores for complex processing are highest on mainframes and lowest on microcomputers; highest in 3GL projects and lowest in 4GL projects. The scores are higher, and have a flatter distribution, for new developments than for enhancement projects. Processing complexity increases steadily from 1991 through to 1996.

The importance of reusability is generally low, with 59% below average and only 14% above average, but it is a very mixed bag.

Reusability is slightly more of a concern for decision support systems than for other types of software. It is less of a concern on personal computers than on mainframes.

This characteristic has one of the broadest distributions. Scores are low overall (54% below average, 22% above average), but both extremes are well represented. Ease of installation is of no concern in 20% of projects, and extremely important in 15%.

Scores are higher for enhancement projects than for new developments; higher for mainframe projects than for other platforms; and higher for engineering systems than for other business areas.

Ease of operation is rarely much of an issue. Scores for this characteristic are nearly the lowest overall: 62% are below average, and only 14% are above average. Scores are spread from 0-3, with 2 the most common score.

The only significant difference emerges when projects are grouped by application type: scores are higher for management information systems and decision support systems than for transaction/production systems and office information systems.

This characteristic has the lowest scores of all: 68% are below average, and 33% have the minimum possible score of 0.

Scores are very low for legal systems, and high for engineering systems. They are higher for new developments than for enhancements or redevelopments; higher for 3GL projects than for others; and higher on midrange computers than on mainframes. Once again, scores are higher for management information systems and decision support systems than for transaction/production systems and office information systems.

Each score for this characteristic is well represented, but generally scores are low: 53% are below average, and 20% above average. The distribution is bimodal, with the two common scores being 0 (of no concern) and 3 (average concern).

Scores are low for 3GL projects, and high for 4GL projects. They are low for new developments; low for mainframe projects; and high for engineering projects. Not surprisingly, this characteristic is more important for management information systems and decision support systems, and less important for transaction/production systems.

In this section the projects are broken down into different groups, on various criteria. For each group of projects, we can see which GSC’s matter more and which matter less.

Type of organization (see Table 5)

Manufacturing projects score low for complexity, distribution, and user focus, and high for data communication. Projects developed by utilities (electricity, gas, water) are lower in complexity, higher in performance constraints and distribution, and place more focus on the user (end user efficiency, ease of installation).

The banking projects are dominated by one organization. Almost all projects from this organization had the same GSC values. They score lower than usual in data communication, on-line data entry, and user focus; and higher in processing complexity.

Business area (see Table 6)

Banking systems score lower than usual for on-line data entry; and higher for transaction rate, processing complexity, and reuse (although reuse is still not very important). Engineering projects feature more distributed processing, and focus more than usual on user efficiency; ease of operation is less important. Legal systems focus on the end user; software developed for multiple sites is rare, and ease of installation is rarely a concern.

Application type (see Table 7)

Management information systems and decision support systems often vary together, as do transaction/production systems and office information systems. Management information systems usually vary in opposition to transaction/production systems.

Management information systems and decision support systems tend to be low in complexity, and high in facilitating change by the user. Transaction/production systems and office information systems have more constraints on performance, and are less concerned with facilitating change by the user. Decision support systems also score higher than usual (although still low overall) on distributed processing.

Project type (see Table 8)

New developments score higher than enhancement projects for complex processing, performance constraints, distributed processing (although this is still low overall), and multiple sites. They have less concern for the user (facilitating change, end user efficiency).

The GSC distributions for new developments tend to be flatter. Those for enhancement projects have sharper peaks.

Programming language (see Table 9)

Third generation languages tend to be used more for software intended for multiple sites. Processing complexity scores are also higher for 3GLs, while scores for on-line update are lower. 4GL projects score highly on facilitating change, and low on processing complexity. Application generator projects score highly on data communication, heavily used configuration, end user efficiency, and on-line update.

Development platform (see Table 10)

Mainframe projects score high for processing complexity and transaction rate, and low for user efficiency, data communication, and on-line update. Midrange projects score high for distribution, multiple sites, heavily used configuration, and facilitating change by the user. Microcomputer projects have low scores for on-line update, processing complexity, and reuse.

From 1991 to 1996, there is a trend down (from more 5’s to more 3’s) for on-line data entry and data communication. There is a trend towards increasing importance for transaction rate, complex processing, and multiple sites.

The characteristics with the highest scores relate to data communication and on-line entry. These are almost universal. Redefinition of these characteristics may be advisable if they are not to lose their discriminative value.

Redefinition of data communication and on-line entry may already be happening informally. A downward trend in the scores for these characteristics might reflect changing perceptions, as what was once seen as extreme is now seen as normal. An alternative explanation is that this trend is due to the increasing use of personal computers, where communication appears to be less of an issue.

The characteristics with the lowest scores relate to distributed processing, multiple sites, ease of installation and operation, and reusability. The first two might be expected to increase, as intranets and the Internet become more important.

More demands are being placed on software. Processing complexity and transaction rates are each increasing with time.

Some GSC’s vary in clearly recognizable ways for different types of software. Most trends make sense. For example, performance is important for transaction/production systems; systems developed on personal computers are more likely to be stand-alone systems with little concern for communication and on-line entry; facilitating change by the user is most important for decision support systems.

The fewest patterns are observed for heavily used configuration, end user efficiency, ease of installation, ease of operation, and multiple sites. The lack of pattern could mean that these have the greatest chance of giving useful variation between projects, and hence helping to explain variations in productivity. On the other hand, these tend to have the lowest scores, and so are the least important issues in a project.

Where patterns are observable across different types of project, they are generally most strongly associated with the application type. Application type is associated with consistent variations in the importance of performance, distribution, multiple sites, on-line update, and (especially) facilitating change.

Several patterns can also be seen when projects are grouped by project type. Fewer patterns emerge when projects are grouped by platform, language, or business area.

Dramatic variations can be seen when projects are grouped by type of organization, but there are few consistent patterns or easy explanations. In this data set, most categories are dominated by a few organizations. Differences between organizations, rather than between types of organization, probably cause the variation. An example is the apparent link between ease of installation and organization type. There is no obvious reason for the variations that were observed (for example, why ease of installation should be important in engineering but not in banking). This is likely to be a matter of organizational attitude rather than anything else.

All of the differences described above, between GSC distributions for different groups of projects, are statistically significant in this data set. Care is needed nevertheless in interpreting them and drawing conclusions from them, for several reasons.

The projects are classified above on one attribute at a time. Sometimes the effect of several attributes may be confounded. For example, there are two blocks of projects which tend to vary together: one of Cobol/mainframe/banking projects, and one of legal/public administration projects. This can mean that a single underlying pattern is reported several times.

A second concern is that some groups are small. For example, when projects are grouped by business area most groups contain less than 20 projects. Small samples limit the strength of conclusions that can be drawn.

When attempting to draw general conclusions, how representative the projects are is a concern. For example, the group of banking projects appears quite large, with 66 projects. Many come from a single organization though, so the sample may not be representative of banking projects generally. The same comment applies to the legal projects.

Examination of Tables 5-10 reveals some relationships between GSC’s. For example, transaction rate and complex processing tend to vary together. To a lesser extent, performance varies with them. These relationships suggest that a smaller number of underlying dimensions might be involved, with some being captured several times over by different GSC’s. To investigate this, factor analysis was used to study the common variation within the GSC’s.

Principal components analysis suggested six factors, as did factor analysis using the maximum likelihood method for estimating factor loadings. When six factors were used, however, no GSC loaded significantly on the sixth (loadings of less than 0.5 were not considered significant). The analysis was therefore performed using five factors.

Table 3 shows the factor loadings for each GSC. The highest loading for each GSC is highlighted in bold, provided its value is at least 0.5.

Between them the five factors account for just over half the total variation in the GSC’s. The first three factors each account for about 13% of the variation. The fourth and fifth factors explain less variation.

Several GSC’s load on each of the first three factors:

1. distributed data processing, multiple sites, heavily used configuration.
2. transaction rate, complex processing.
3. data communication, on-line entry, on-line update, end user efficiency.

Each of these factors has a common theme: distribution, the difficulty of the problem to be solved, and interactivity respectively.

Three characteristics do not load significantly on any factor. Performance loads most strongly on factor two, strengthening that factor’s interpretation as one of problem difficulty. Ease of operation loads most strongly on factor four (ease of installation), suggesting ease of use as an interpretation for that factor. Reusability loads most strongly on factor one; this is difficult to interpret.

On-line update loads almost equally on factors two (problem difficulty) and three (interactivity). This makes sense, since high scores for this factor represent not just that most updates are performed on-line, but that complex processing is involved (eg for backups).

Separate factor analyses were performed for each of the different project groupings reported in Section 3. Details are not given here, because the results are unchanged. Although the order of importance of the factors varies from group to group, the same GSC’s always load on the same factors.

Sections 3 and 4 looked at the GSC’s in isolation, without regard to their purpose within function point analysis. Now we turn our attention to that purpose.

The GSC’s are used to construct a value adjustment factor ("VAF"). The result is a number from 0.65 to 1.35. It is used as a multiplier to adjust the basic function point count.

VAF = 0.65 + 0.01 x (sum of GSC’s)

In our view, the adjustment process in FPA is an attempt to explain variations in productivity - in other words, to improve the relationship between function points and project effort. In this section we investigate the relationships between function points and project effort, before and after adjustment with the VAF, to see whether the VAF serves this purpose.

Table 4 summarizes the range of value adjustment factors, and Figure 1 shows their distribution.

Table 4: Summary of VAF

Figure 1: Distribution of VAF

The distribution is skewed slightly to the right, and peaks sharply at 1.00. Tests for normality indicate that the distribution is normal.

The median is exactly 1.00, and the mean 1.01.

The maximum VAF represents an increase of 29% in the function point count; the minimum represents a decrease of 27%. Since the variation is limited to plus or minus 35%, the extremes observed here approach the limits of possibility. Few adjustments are this great; half involve an adjustment of only about 5%.

Although the overall range of variation is quite large, for most projects the VAF does not result in much change to the function point value.

To see if the adjustment process is effective effective for effort prediction, the strength of the relationship between function points and effort was investigated, using both adjusted and unadjusted function points as the independent variable.

Regression models were derived that relate each function point count to effort. The sample was divided randomly into a set of 157 projects (two thirds of them) from which the models were derived, and a set of 78 projects with which the models were evaluated. The accuracy of the models was assessed using R² (which indicates the amount of variation in effort explained by function points), mean magnitude of relative error ("MMRE"), and the proportion of estimates that were within 25% of the actual effort ("Pred(.25)").

When adjusted or unadjusted function points were plotted against effort, the graphs were fan-shaped and dominated by small values for function points and effort. They were not suitable for linear regression. Taking the logarithms of both dependent and independent variable gave values much better suited to linear regression. The regressions were accordingly done on the transformed values. The residuals were normally distributed, and showed no pattern and no heteroscedasticity.

The resulting equation was: Effort = 17.2 x UFP ^.863

With this equation, the correlation between estimated and actual effort was 0.76. R² was thus 0.58, meaning that unadjusted function points explained 58% of the variation in effort. MMRE was 1.28, meaning an average relative error of 128%. The largest relative error of all was 920%. 21% of estimates were within 25% of the actual values.

The equation derived using adjusted function points was: Effort = 21.0 x AFP ^.826

With this equation, the correlation between estimated and actual effort was 0.77, giving a value of 0.59 for R². MMRE was 1.25, the maximum relative error was again 920%, and 17% of estimates were within 25% of the actual values.

The R² values of 58-59% are comparable to Kitchenham’s 50-54% [3], Kemerer’s 55% [6], and Jeffery’s 58% [8].

Using adjusted function points instead of unadjusted function points as a predictor of effort makes almost no difference in any of R², MMRE, or Pred(.25). Pred(.25) even gets slightly worse.

In 108 of the 235 projects, VAF points in the right direction for correcting estimation errors (eg effort is underestimated from unadjusted function points, but VAF is greater than 1.00 and so indicates that an upwards adjustment is needed). But in 73 projects VAF points in the wrong direction, and in 54 projects VAF is 1.00 and provides no clue. Thus one cannot even use VAF as a simple indicator of whether effort will be over-estimated or under-estimated from unadjusted function points.

In this data set, the VAF provides no assistance for estimating effort. This confirms the findings of other researchers [3,6,7,8]. We have studied a larger sample of projects, and used a wider set of criteria to assess the relationship, and still obtained the same results. This strengthens the statistical evidence that the adjustment process in function point analysis is not effective for effort prediction.

A collection of 235 projects has been analyzed, to see how the function point GSC’s are used in practice.

The first part of the analysis was largely descriptive. The GSC values were studied in isolation, without regard to their purpose within function point analysis.

The individual GSC’s may be viewed as categorical information about projects. Interesting trends can be seen when GSC values are compared for different collections of projects. Some GSC’s are more important than others for different types of projects.

Some GSC’s are defined in terms more suited to 1979-1984 than to today’s on-line world. They have less discriminative value now, and should perhaps be redefined.

Factor analysis indicates that the GSC’s reflect a smaller number of underlying dimensions: distribution, problem difficulty, interactivity, ease of use, and focus on the end user. Most of the fourteen GSC’s can be assigned to these factors in obvious ways. One characteristic (on-line update) appears to combine two issues, and should perhaps be redefined to separate the two concerns.

The last part of the analysis evaluated the VAF. The VAF was found not to improve the relationship between function points and effort. This confirms the findings of other researchers.

It seems clear that the value adjustment factor should not be used. Theoretical objections to how it is constructed might be overlooked if it had some practical benefit, but the evidence is that it does not.

Recording the values of the individual GSC’s may still be useful, both for historical comparisons and for understanding differences between projects.

Two avenues for future work are clear. A new formulation is needed if the GSC values are to be used in some way to improve the explanation of project effort. Some of the GSC’s themselves could be profitably redefined. The five underlying five factors identified here (distribution, problem difficulty, interactivity, ease of use, user focus) could provide a starting point for doing so.

I am grateful to Alain Abran for several valuable discussions early in this research, and for his comments on this paper.

Manufacturing (17 projects) Utilities (17 projects)

Public admin (36 projects) Fin/Prop/Business (33 projects)

Banking (39 projects) Other (65 projects)

Financial (21 projects) Personnel (11 projects)

Manufacturing (18 projects) Banking (66 projects)

Legal (13 projects) Engineering (16 projects)

Decision Support Management information

(11 projects) (47 projects)

Office Information Transaction/production

(23 projects) (74 projects)

New development (142 projects)