Artiﬁcial neural networks for streamﬂow prediction Utilisation des réseaux des neurones artiﬁcielles pour la prédiction des écoulements OSCAR R. DOLLING, Assistant Professor Department of Hydraulics, Universidad Nacional de San Juan, San Juan, Argentina EDUARDO A. VARAS, Professor, School of Engineering, Pontiﬁcia Universidad Católica de Chile, Santiago, Chile ABSTRACT This paper presents monthly streamﬂow prediction using artiﬁcial neural networks (ANN) on mountain watersheds. The procedure addresses the selection of input variables, the deﬁnition of model architecture and the strategy of the learning process. Results show that spring and summer monthly streamﬂows can be adequately represented, improving the results of calculations obtained using other methods. Better streamﬂow prediction methods should have signiﬁcant beneﬁts for the optimal use of water resources for irrigation and hydroelectric energy generation. RÉSUMÉ Cet article présente une méthodologie pour prédire les débits mensuels dans les basin versant de montagne, en employant la technique des réseaux des neurones artiﬁcielles. La méthode permet le choix des variables d‘entrée et la déﬁnitions de l‘architecture des modèle et des procédés d‘apprentissage. Les résultats montrent qu‘il est possible prédire les débits mensuels des saisons de printemps et d‘été avec une approximation satisfaissante, meilleure que celle qu‘on peut obtenir avec les méthodes traditionelles. Ceci permet d‘avoir une utilisation optimale des ressources en eau dans les basins, soit pour irrigation ou pour la production dénergie hydroelectrique.

I. Introduction Several researchers have suggested different mathematical and statistical methods to predict streamﬂows. Approaches such as hydrologic simulation models (Crawford and Linsley, 1962), snowmelt models (Martinec and Rango, 1992), multiple regression models, transfer functions models (Tripodi, 1999) and empirical models applied to log-transformed ﬂows (Karunanithi et al, 1994) can be mentioned. A recent study to predict streamﬂow in the upper Maule River basin (Fernandez and Tripodi, 1999), illustrates the difﬁculty of predicting monthly streamﬂow with reasonable accuracy. The authors report the importance of incorporating the inﬂuence of global phenomena such as ENSO in the hydrologic behavior of watershed in the southern Paciﬁc Region. This same fact has also been reported by other researchers.(Salazar et al 1998), (Rossel et al., 1998), (Flamenco and Berri, 1997), (Aceituno and Garreaud, 1995) . Artiﬁcial Neural Networks have a structure where non linear functions are present and the parameter identiﬁcation process is based on techniques which search for global maximums in the space of feasible parameter values, and hence can represent the non linear effects present in the rainfall-runoff processes. ANN were developed as a information storage models and their parameters are calculated in a manner that resembles natural processes (McCulloch and Pitts, 1943). Details of their properties and the computational process have been presented by Hopﬁeld (1982) and the learning process of ANN is described by Rumelhart and McClelland (1986). The use of ANN techniques in water resources and streamﬂow prediction is relatively new and has been reported by French et al (1992) , Zurada (1992), Hall and Minns (1993), Zealand et al (1999), Abrahart et al (1998), Zhu and Fujita (1994), Hsu et al (1993), Abrahart and See (1998), Minns (1998) and Salazar et al (1998), among others. An important advantage of ANN compared to classical stochastic

models is that they do not require variables to be stationary and normally distributed (Burke, 1991). Non stationary effects present in global phenomena, in morphological changes in rivers and others can be captured by the inner structure of ANN (Dandy and Maier, 1996). Furthermore, ANN are relatively stable with respect to noise in the data and have a good generalization potential to represent input-output relationships. (Zealand et al, 1999). II. Artiﬁcial neural networks An ANN is a mathematical model which has a highly connected structure similar to brain cells. They consist of a number of neurons arranged in different layers, an input layer, an output layer and one or more hidden layers. (Figure 1) The input neurons receive and process the input signals and send an output signal to other neurons in the network. Each neuron can be connected to the other neurons and has an activation function and a threshold function, which can be continuous, linear or non linear functions. The signal passing through a neuron is transformed by weights which modify the functions and thus the output signal that reaches the following neuron. Modifying the weights for all neurons in the network, changes the output. Once the architecture of the network is deﬁned, weights are calculated so as to represent the desired output through a learning process where the ANN is trained to obtain the expected results. Information available is used to deﬁne a learning or training data set and a validation data set. (Rumelhart et al., 1986). Several different architectures and topologies have been proposed for ANN. They differ in terms of architecture, in the learning process and in the training strategy. (Nussbaum et al, 1996). A linear model can be represented adequately by a single layer network, while a non linear model is generally associated with a multiple layer network. (LSWC, 1999). In this work, only three layer networks were considered since it has been shown that they have a

Revision received May 24, 2001. Open for discussion till February 28, 2003.

JOURNAL OF HYDRAULIC RESEARCH, VOL. 40, 2002, NO. 5

547

1

1 2 3 4 5 6 7

1

2

: : :

2

: 20

30

Fig. 1. Artiﬁcial neuron network fully connected Multi Input Multi Output structure (MIMO) ANN(30,20,7)

good potential to represent linear or non linear outputs. (Minns, 1998). According to Elaine and Knight (1996) multiple layer networks can represent any function, which implies that the design process has to focus on the deﬁnition of the number of neurons and in the learning strategy. Possible model types are feedforward networks (BP1) with backpropagation momentum learning process, hyperbolic activation function for the hidden layer and linear function for the output layer; BP2 networks which are the same as the previous ones but with sigmoidal activation function (output and hidden layers); SM network which are equal to the previous one but with stochastic learning process; Elman recurrent network with backpropagation momentum learning process (ELM) and radial basis function networks (RBN). A detailed description of these model types are presented by Dibike et al (1999), Zell el al – SNNS – (1995), Mitchell (1997), Hassoun (1995) and Freeman and Skapura (1992). III. Problem formulation Application of ANN to streamﬂow predictions requires a decision regarding four main aspects: selection of the variables that best explain runoff, design of the optimal network architecture, selection of the best strategy for the learning process and selection of the system that best represents the streamﬂow universe. Since there are several ways in which the researcher can fulﬁll the mentioned tasks, it is important to develop a systematic procedure that can produce an ANN that captures most of the predictable information present in the data and that can be safely generalized to represent realizations different to the ones present in the training episodes. This paper proposes a method to obtain an ANN to predict monthly streamﬂow in mountain watersheds, subject to rainfall and snowmelt in conditions of scarce hydrologic information. The models should be able to predict streamﬂow one to seven months in advance during the spring and summer periods. Input variables are ENSO index for Zone 3 of the Paciﬁc, monthly temperature, precipitation and snow course information. Two different approaches were tested. The ﬁrst approach developed a different ANN for each month and the second approach deﬁned one ANN

548

with 7 different output variables which represented monthly predictions for the ﬂows of spring and summer months. IV. Proposed methodology There is no unique and systematic methodology for the design and validation of an ANN model. This paper presents a procedure developed using current technical literature, heuristics and experience of experts in artiﬁcial intelligence. The steps of the method are included in the block diagram presented in Figure 2 and the principal stages which are information analysis and model identiﬁcation, are described below. IV.1 Information analysis This part includes the preliminary data analysis, the selection of the most pertinent inputs and output and the choice of the learning, testing and validation data sets. a) Information handling: The ﬁrst step in every modeling process is to collect and analyze the relevant information available in order to have accurate, representative and relevant information for the problem of interest. The importance of this stage cannot be overlooked, since the rest of the process depends on the data. (Linsley et al, 1988). b) Selection of output variables: For this particular problem, two different approaches can be followed. One can predict the streamﬂow of one month at a time using the same type of model but with different parameters or alternatively one can build a model with 7 output variables each of them corresponding to the streamﬂow value of each month between July through January. In the ﬁrst approach, the same input variables and model architecture are maintained, estimating parameters to represent the ﬂow of each of the seven months. This model has 1 output variable. In the second case, a model to estimate 7 output variables, which represent the monthly streamﬂows for the period of interest. The design procedure in both cases is the same, even though both models are different, since the second model has to represent and interpret several phenomena simultaneously.

JOURNAL OF HYDRAULIC RESEARCH, VOL. 40, 2002, NO. 5

Information Analysis Streamflow and data series

Statistical data handling

Selection of neurons in output layer

Selection of neurons in input layer

Identification of pairs for training, validation and testing sets

Model candidates

Data consistency Problem

Evaluation of model behavior

Selection input Problem

Statistic calculation and analysis of errors

Identification Problem

Model validation

Learning Problem

Learning and training strategy

Selection input-output pairs Problem

Specification of model types

Comparison of validation results

Selection of optimal model

Optimal model

Fig. 2 stages in ann model formulation

c) Selection of input variables: To choose the most adequate group of input variables a sensitivity analysis should be performed so as to be able to identify which are the most relevant variables to represent the output or streamﬂow. This stage deﬁnes

JOURNAL OF HYDRAULIC RESEARCH, VOL. 40, 2002, NO. 5

the number of neurons of the input layer. To select the variables a multivariate analysis using principal components is recommended. One can also use an ANN for this step but the possible combinations tend to grow in a signiﬁcant manner, so the re-

549

quired time to train and validate the ANN models might be non practical. Using a different type of model, as suggested, makes this stage independent of model design and leaves the possibility of performing a sensitivity analysis of the output in the validation stage of the process. This sensitivity analysis might even suggest the advantage of altering the selected input variables. This step is specially critical since leaving out variables which explain a signiﬁcant part of the variance present in the streamﬂow data should be avoided. On the other hand it is convenient to have a parsimonious model. This stage has a direct effect on the learning time of the model and the accuracy of the ﬁnal result. (Dean et al,1995), (Zealand et al, 1999). d) Selection of learning, validation and testing sets. Once the input and output variables are deﬁned, it is convenient to classify the complete data set in three categories: dry, normal and wet, depending on the ﬂow volumes on each month. This classiﬁcation was obtained by using an ANN model with the same input and output variables, excluding the hydrologic response index. The inclusion of an index of this type, has been suggested in order to improve system response. (Zealand et al, 1999). The selection of the input-output pairs which form the validation, training and testing sets is not random, in order to have a model with adequate predictive capability for the whole range of the data (Mitchell, 1997). The learning and validation sets should have at least 40% of the pairs representing each of the hydrologic response categories. The testing set should include 20% of the pairs representing each hydrologic response. IV.2 Model identiﬁcation This step of the process includes the deﬁnition of model architecture, the learning or training process, the validation of the model and the selection of the optimal model. a) Selection of the number of neurons in the hidden layer: To select the number of neurons of the hidden layer the Cascade-Correlation algorithm proposed by Fahlman and Lebiere (1990) is recommended (Zell et al – SNNS –, 1995). The sets used for training, validation and testing are the same for each of the 5 different types of models mentioned previously. This algorithm combines the increment of the number of neurons in the hidden layer with the learning process, analyzing the effect on the sum of square errors in the output. This is a faster algorithm than the backpropagation algorithm (Karunanithi et al, 1994) and is specially useful to identify the best number of neurons. For the case of the Elman network the Recurrent-Cascade-Correlation (RCC) algorithm is recommended. Afterwards, model architectures are subject to connection pruning, using the magnitude based algorithm, in order to obtain more sparse models (Abrahart et al, 1998). Once the number of neurons in the hidden layer is deﬁned, the models are completely speciﬁed and the selection of the best model can be approached.

550

b) Learning process of model candidates. The weights necessary to represent the required output of the ANN models, are determined through a learning process using the information represented by the input variables. The best set of weights represent the values that minimize an objective function, such as the mean square error. The following learning methods are suggested. Backpropagation momentum for feedforward networks; Recurrent Cascade Correlation algorithm for Elman networks; Radial Basis Learning algorithm for radial Basis Function. (Zell el al – SNNS –, 1995). The initialization function and the way in which weights are modiﬁed during the learning process depend on the algorithm used in the learning process. (Zell el al – SNNS –, 1995). An important consideration is the criteria to terminate the learning process, in order to avoid overﬁtting. For feedforward networks trained using the backpropagation momentum algorithm, the strategy proposed by Mitchell (1997) is recommended. c) Validation and evaluation of predictive capability: The validation process consists in the analysis of errors, deﬁned as the difference between observed and estimated streamﬂow for each set of outputs (learning, validation and test sets). A useful index is the number of cycles needed for training the network. The statistics used for the objective function are the ones presented in Table 1, and measure the goodness of ﬁt of the model, the ability of the network to generalize or extrapolate the results outside of the range of the learning set, the presence of overﬁtting problems, the sensibility of the network to initial conditions and the errors due to the use of a speciﬁc combination of learning and validation sets. In particular, S4E is more sensible to maximum values, MAE measures the ﬁt to mean values. RMSE evaluates the variance of errors independently of the sample size (Abrahart and See, 1998). A high value of RMSE will usually indicate a deﬁciency in generalization of the network due to a bad selection of the number of hidden neurons or a weak learning process. In order to evaluate the consistency of the input-output pairs the following strategy is suggested. First the network is trained with all the pairs selected for the learning process. Then, one of the pairs is eliminated from the set and the network is trained with the rest of the pairs and the isolated pair is used as the validation set. The network is trained until a minimum sum of squared errors is obtained. This process is repeated for the rest of the pairs of the learning set. Those pairs which present convergence problems are examined in detail to detect possible inconsistencies with the rest of the pairs. If an atypical behavior is detected the possibility of eliminating this pair should be studied. d) Selection of the optimal model: The results obtained with the validation set for each of the selected model architectures are analyzed in order to choose the best model for the required streamﬂow prediction. To judge which model has the best performance, graphical and analytical comparisons can be used. One can compare time series graphs of observed and predicted monthly streamﬂows and dispersion diagrams of observed and calculated values. Errors or residues should be analyzed to test them for normality, independence, autocorrelation and cross cor-

JOURNAL OF HYDRAULIC RESEARCH, VOL. 40, 2002, NO. 5

Table 1. Statistics for model comparison Concept

Name

Sum of square errors

SSE

Formula j

∑ (obs − calc)

2

i

Fourth order error

S4E

j

∑ (obs − calc)

4

i

Mean absolute error

MAE

j

∑ (obs − calc) i

N Root mean square error RMSE

j

∑ (obs − calc)

2

i

N Efﬁciency Coefﬁcient

COE

Mean relative variance

ARV

(Variance residues) E = 1− (data Variance) j

∑ (obs − calc )

2

i

j

∑ (obs-avg.obs)

2

i

Coefﬁcient of determination

R2

j

∑ (calc-avg.obs) i j

2

∑ (obs-avg. obs)

2

i

relation. Both numerical and graphical results should be considered with respect to predetermined criteria to select the best model. V. Application to San Juan river basin This methodology was applied to obtain a model for streamﬂow prediction for San Juan River basin, Argentina, using climatological data from Pachón meteorological station located at 1900 m of

altitude. Flows were measured at Km 47.3 Station which controls a 20.000 square kilometer watershed. The neural network model was used to predict monthly ﬂows for the period from July through January, using meteorological information gathered during April, May and June. Input variables used were the month number, mean temperature, relative humidity, sunshine hours, wind velocity, snow depth, number of cloudy days, precipitation, ENSO index and runoff volumes for the three preceding months. Output variables were the 7 monthly ﬂows for the period July through January. The San Juan River basin is located in a mountainous region of the Andes Mountain Ranges. The area was formed at the end of the Tertiary and beginning of the Pleistocean, period in which the Andes Mountain Ranges complete their ascent and acquire their current morphology. The upper basin area extends from latitude 30° 33´S (Olivares Mountains) to 32° South (northern slope of the Aconcagua), and from the summits of the Tigre Mountain Range and the Western ﬂank of the Precordillera reaching the summits that separate the slopes of the Atlantic and of the Paciﬁc. The upper part of the basin, which represent approximately 50% of the total area, is located in the central part of the mountain ranges of the Andes, where the altitude ranges from 3,500 m above sea level to a maximum of 7,000 m. The average elevation for the basin is 3,750 m. Precipitation on the basin is mostly snow. The liquid precipitation on the lower basin hardly reach the 100 mm per year. Although there are no rainfall stations on the upper basin, above 3,000 m. studies carried out by Zakalik (1990) characterize the pluvial regime of the different sub-basins. Castaño river with 600 mm per year, Patillos River with 400 mm per year and Teatinos River with 300 mm per year. A study of the CFI, (1961) classiﬁes precipitation in the Castaño River basin as snow, with occasional rains; Los Patos Superior River, as glaciers and little snow; and Blanco River, with prevalence of permanent snow and with greater inﬂuence in the regime of the San Juan River. The annual average ﬂow of the basin of the San Juan River is 2078 Mm3. With the 18 years of available data input-output pairs were prepared. Each pair consisted in 30 input variables and 7 output variables, that is, a total of 666 data elements were used for training and testing the network. From these pairs, 17 were used for train-

SSE

r = 0.2 r = 0.8 cycles Fig. 3. Training curves for different strategies using backpropagation momentum method.

JOURNAL OF HYDRAULIC RESEARCH, VOL. 40, 2002, NO. 5

551

Table 2. Sum of square errors as a function of training cycles Cycles

SSE

500

0.03772

1300

0.04307

2000

0.01719

4000

0.01482

4000 IOS

0.00900

ing and the last pair was used for testing the network. As mentioned the 30 input variables correspond to the observed values at Station Pachón for the months of April, May and June, of number of the month, IOS (Index of Oscillation of the South), monthly precipitation, average temperature, relative average humidity, effective sunshine hours, average monthly wind, maximum snow depth, number of cloudy days and the ﬂow volumes measured at Km 47.3 gaging station. Both stations are dependent of the Department of Hydraulics of the County of San Juan. The 7 variables correspond to the values of the monthly ﬂow volumes measured at the Km 47.3 gaging station, for the months of July to January. A feed-forward ANN model having an architecture 30-20-7 (30 input neurons, 20 hidden neurons and 7 output neurons) trained

by means of back-propagation momentum and using the learning strategy suggested by Mitchell (1997) for scarce data had a good predictive behavior. Connection pruning was carried out to obtain a total of 320 connections. All values included in the pairs were scaled to ± 0.7 with the purpose of homogenizing the magnitudes of the different variables, to be able to use a sigmoid function as an activation function at the network exit, and to extrapolate output values greater than the ones used in the training of the network. The network was initialized with random values between ± 1. The back-propagation algorithm of convergence was studied and different learning rates were tested with different momentum coefﬁcients. The values ﬁnally adopted were 0.9 and 0.7 respectively. An example of two training curves is presented in Figure 3. The sum of squared errors (SSE) is shown in Table 2 for the prediction of streamﬂow as a function of the number of training cycles. Results show that the network has a good generalization performance using 4000 training cycles. Table 3 presents the absolute values of prediction errors in percent and the frequency of different error magnitudes for the 17 year period. It can be observed that errors are only in two cases greater than 14%. Figure 4 shows the mean errors for the 7 month prediction period for two different number of training cycles. This curves indicate the ability of the neural network to generalize its performance. The graph shows that if 4000 cycles are used for

Table 3. Absolute values of prediction errors

552

Year

July

August

September

October

November

December

January

1981

2

6

5

5

4

3

1

1982

1

1

1

0

0

0

0

1983

0

2

2

0

0

0

0

1984

1

2

1

0

0

0

0

1985

5

14

11

4

1

1

0

1986

4

2

4

1

0

0

0

1987

1

2

1

1

0

0

0

1988

0

0

0

1

0

1

0

1989

2

10

7

4

1

2

1

1990

4

4

2

1

0

0

0

1991

0

2

1

1

0

0

0

1992

4

3

8

5

1

1

1

1993

0

3

7

7

1

3

1

1994

6

11

8

4

0

0

1

1995

0

2

2

0

0

0

0

1996

8

6

6

9

1

4

4

1997

6

11

3

22

4

1

17

Error

I. Introduction Several researchers have suggested different mathematical and statistical methods to predict streamﬂows. Approaches such as hydrologic simulation models (Crawford and Linsley, 1962), snowmelt models (Martinec and Rango, 1992), multiple regression models, transfer functions models (Tripodi, 1999) and empirical models applied to log-transformed ﬂows (Karunanithi et al, 1994) can be mentioned. A recent study to predict streamﬂow in the upper Maule River basin (Fernandez and Tripodi, 1999), illustrates the difﬁculty of predicting monthly streamﬂow with reasonable accuracy. The authors report the importance of incorporating the inﬂuence of global phenomena such as ENSO in the hydrologic behavior of watershed in the southern Paciﬁc Region. This same fact has also been reported by other researchers.(Salazar et al 1998), (Rossel et al., 1998), (Flamenco and Berri, 1997), (Aceituno and Garreaud, 1995) . Artiﬁcial Neural Networks have a structure where non linear functions are present and the parameter identiﬁcation process is based on techniques which search for global maximums in the space of feasible parameter values, and hence can represent the non linear effects present in the rainfall-runoff processes. ANN were developed as a information storage models and their parameters are calculated in a manner that resembles natural processes (McCulloch and Pitts, 1943). Details of their properties and the computational process have been presented by Hopﬁeld (1982) and the learning process of ANN is described by Rumelhart and McClelland (1986). The use of ANN techniques in water resources and streamﬂow prediction is relatively new and has been reported by French et al (1992) , Zurada (1992), Hall and Minns (1993), Zealand et al (1999), Abrahart et al (1998), Zhu and Fujita (1994), Hsu et al (1993), Abrahart and See (1998), Minns (1998) and Salazar et al (1998), among others. An important advantage of ANN compared to classical stochastic

models is that they do not require variables to be stationary and normally distributed (Burke, 1991). Non stationary effects present in global phenomena, in morphological changes in rivers and others can be captured by the inner structure of ANN (Dandy and Maier, 1996). Furthermore, ANN are relatively stable with respect to noise in the data and have a good generalization potential to represent input-output relationships. (Zealand et al, 1999). II. Artiﬁcial neural networks An ANN is a mathematical model which has a highly connected structure similar to brain cells. They consist of a number of neurons arranged in different layers, an input layer, an output layer and one or more hidden layers. (Figure 1) The input neurons receive and process the input signals and send an output signal to other neurons in the network. Each neuron can be connected to the other neurons and has an activation function and a threshold function, which can be continuous, linear or non linear functions. The signal passing through a neuron is transformed by weights which modify the functions and thus the output signal that reaches the following neuron. Modifying the weights for all neurons in the network, changes the output. Once the architecture of the network is deﬁned, weights are calculated so as to represent the desired output through a learning process where the ANN is trained to obtain the expected results. Information available is used to deﬁne a learning or training data set and a validation data set. (Rumelhart et al., 1986). Several different architectures and topologies have been proposed for ANN. They differ in terms of architecture, in the learning process and in the training strategy. (Nussbaum et al, 1996). A linear model can be represented adequately by a single layer network, while a non linear model is generally associated with a multiple layer network. (LSWC, 1999). In this work, only three layer networks were considered since it has been shown that they have a

Revision received May 24, 2001. Open for discussion till February 28, 2003.

JOURNAL OF HYDRAULIC RESEARCH, VOL. 40, 2002, NO. 5

547

1

1 2 3 4 5 6 7

1

2

: : :

2

: 20

30

Fig. 1. Artiﬁcial neuron network fully connected Multi Input Multi Output structure (MIMO) ANN(30,20,7)

good potential to represent linear or non linear outputs. (Minns, 1998). According to Elaine and Knight (1996) multiple layer networks can represent any function, which implies that the design process has to focus on the deﬁnition of the number of neurons and in the learning strategy. Possible model types are feedforward networks (BP1) with backpropagation momentum learning process, hyperbolic activation function for the hidden layer and linear function for the output layer; BP2 networks which are the same as the previous ones but with sigmoidal activation function (output and hidden layers); SM network which are equal to the previous one but with stochastic learning process; Elman recurrent network with backpropagation momentum learning process (ELM) and radial basis function networks (RBN). A detailed description of these model types are presented by Dibike et al (1999), Zell el al – SNNS – (1995), Mitchell (1997), Hassoun (1995) and Freeman and Skapura (1992). III. Problem formulation Application of ANN to streamﬂow predictions requires a decision regarding four main aspects: selection of the variables that best explain runoff, design of the optimal network architecture, selection of the best strategy for the learning process and selection of the system that best represents the streamﬂow universe. Since there are several ways in which the researcher can fulﬁll the mentioned tasks, it is important to develop a systematic procedure that can produce an ANN that captures most of the predictable information present in the data and that can be safely generalized to represent realizations different to the ones present in the training episodes. This paper proposes a method to obtain an ANN to predict monthly streamﬂow in mountain watersheds, subject to rainfall and snowmelt in conditions of scarce hydrologic information. The models should be able to predict streamﬂow one to seven months in advance during the spring and summer periods. Input variables are ENSO index for Zone 3 of the Paciﬁc, monthly temperature, precipitation and snow course information. Two different approaches were tested. The ﬁrst approach developed a different ANN for each month and the second approach deﬁned one ANN

548

with 7 different output variables which represented monthly predictions for the ﬂows of spring and summer months. IV. Proposed methodology There is no unique and systematic methodology for the design and validation of an ANN model. This paper presents a procedure developed using current technical literature, heuristics and experience of experts in artiﬁcial intelligence. The steps of the method are included in the block diagram presented in Figure 2 and the principal stages which are information analysis and model identiﬁcation, are described below. IV.1 Information analysis This part includes the preliminary data analysis, the selection of the most pertinent inputs and output and the choice of the learning, testing and validation data sets. a) Information handling: The ﬁrst step in every modeling process is to collect and analyze the relevant information available in order to have accurate, representative and relevant information for the problem of interest. The importance of this stage cannot be overlooked, since the rest of the process depends on the data. (Linsley et al, 1988). b) Selection of output variables: For this particular problem, two different approaches can be followed. One can predict the streamﬂow of one month at a time using the same type of model but with different parameters or alternatively one can build a model with 7 output variables each of them corresponding to the streamﬂow value of each month between July through January. In the ﬁrst approach, the same input variables and model architecture are maintained, estimating parameters to represent the ﬂow of each of the seven months. This model has 1 output variable. In the second case, a model to estimate 7 output variables, which represent the monthly streamﬂows for the period of interest. The design procedure in both cases is the same, even though both models are different, since the second model has to represent and interpret several phenomena simultaneously.

JOURNAL OF HYDRAULIC RESEARCH, VOL. 40, 2002, NO. 5

Information Analysis Streamflow and data series

Statistical data handling

Selection of neurons in output layer

Selection of neurons in input layer

Identification of pairs for training, validation and testing sets

Model candidates

Data consistency Problem

Evaluation of model behavior

Selection input Problem

Statistic calculation and analysis of errors

Identification Problem

Model validation

Learning Problem

Learning and training strategy

Selection input-output pairs Problem

Specification of model types

Comparison of validation results

Selection of optimal model

Optimal model

Fig. 2 stages in ann model formulation

c) Selection of input variables: To choose the most adequate group of input variables a sensitivity analysis should be performed so as to be able to identify which are the most relevant variables to represent the output or streamﬂow. This stage deﬁnes

JOURNAL OF HYDRAULIC RESEARCH, VOL. 40, 2002, NO. 5

the number of neurons of the input layer. To select the variables a multivariate analysis using principal components is recommended. One can also use an ANN for this step but the possible combinations tend to grow in a signiﬁcant manner, so the re-

549

quired time to train and validate the ANN models might be non practical. Using a different type of model, as suggested, makes this stage independent of model design and leaves the possibility of performing a sensitivity analysis of the output in the validation stage of the process. This sensitivity analysis might even suggest the advantage of altering the selected input variables. This step is specially critical since leaving out variables which explain a signiﬁcant part of the variance present in the streamﬂow data should be avoided. On the other hand it is convenient to have a parsimonious model. This stage has a direct effect on the learning time of the model and the accuracy of the ﬁnal result. (Dean et al,1995), (Zealand et al, 1999). d) Selection of learning, validation and testing sets. Once the input and output variables are deﬁned, it is convenient to classify the complete data set in three categories: dry, normal and wet, depending on the ﬂow volumes on each month. This classiﬁcation was obtained by using an ANN model with the same input and output variables, excluding the hydrologic response index. The inclusion of an index of this type, has been suggested in order to improve system response. (Zealand et al, 1999). The selection of the input-output pairs which form the validation, training and testing sets is not random, in order to have a model with adequate predictive capability for the whole range of the data (Mitchell, 1997). The learning and validation sets should have at least 40% of the pairs representing each of the hydrologic response categories. The testing set should include 20% of the pairs representing each hydrologic response. IV.2 Model identiﬁcation This step of the process includes the deﬁnition of model architecture, the learning or training process, the validation of the model and the selection of the optimal model. a) Selection of the number of neurons in the hidden layer: To select the number of neurons of the hidden layer the Cascade-Correlation algorithm proposed by Fahlman and Lebiere (1990) is recommended (Zell et al – SNNS –, 1995). The sets used for training, validation and testing are the same for each of the 5 different types of models mentioned previously. This algorithm combines the increment of the number of neurons in the hidden layer with the learning process, analyzing the effect on the sum of square errors in the output. This is a faster algorithm than the backpropagation algorithm (Karunanithi et al, 1994) and is specially useful to identify the best number of neurons. For the case of the Elman network the Recurrent-Cascade-Correlation (RCC) algorithm is recommended. Afterwards, model architectures are subject to connection pruning, using the magnitude based algorithm, in order to obtain more sparse models (Abrahart et al, 1998). Once the number of neurons in the hidden layer is deﬁned, the models are completely speciﬁed and the selection of the best model can be approached.

550

b) Learning process of model candidates. The weights necessary to represent the required output of the ANN models, are determined through a learning process using the information represented by the input variables. The best set of weights represent the values that minimize an objective function, such as the mean square error. The following learning methods are suggested. Backpropagation momentum for feedforward networks; Recurrent Cascade Correlation algorithm for Elman networks; Radial Basis Learning algorithm for radial Basis Function. (Zell el al – SNNS –, 1995). The initialization function and the way in which weights are modiﬁed during the learning process depend on the algorithm used in the learning process. (Zell el al – SNNS –, 1995). An important consideration is the criteria to terminate the learning process, in order to avoid overﬁtting. For feedforward networks trained using the backpropagation momentum algorithm, the strategy proposed by Mitchell (1997) is recommended. c) Validation and evaluation of predictive capability: The validation process consists in the analysis of errors, deﬁned as the difference between observed and estimated streamﬂow for each set of outputs (learning, validation and test sets). A useful index is the number of cycles needed for training the network. The statistics used for the objective function are the ones presented in Table 1, and measure the goodness of ﬁt of the model, the ability of the network to generalize or extrapolate the results outside of the range of the learning set, the presence of overﬁtting problems, the sensibility of the network to initial conditions and the errors due to the use of a speciﬁc combination of learning and validation sets. In particular, S4E is more sensible to maximum values, MAE measures the ﬁt to mean values. RMSE evaluates the variance of errors independently of the sample size (Abrahart and See, 1998). A high value of RMSE will usually indicate a deﬁciency in generalization of the network due to a bad selection of the number of hidden neurons or a weak learning process. In order to evaluate the consistency of the input-output pairs the following strategy is suggested. First the network is trained with all the pairs selected for the learning process. Then, one of the pairs is eliminated from the set and the network is trained with the rest of the pairs and the isolated pair is used as the validation set. The network is trained until a minimum sum of squared errors is obtained. This process is repeated for the rest of the pairs of the learning set. Those pairs which present convergence problems are examined in detail to detect possible inconsistencies with the rest of the pairs. If an atypical behavior is detected the possibility of eliminating this pair should be studied. d) Selection of the optimal model: The results obtained with the validation set for each of the selected model architectures are analyzed in order to choose the best model for the required streamﬂow prediction. To judge which model has the best performance, graphical and analytical comparisons can be used. One can compare time series graphs of observed and predicted monthly streamﬂows and dispersion diagrams of observed and calculated values. Errors or residues should be analyzed to test them for normality, independence, autocorrelation and cross cor-

JOURNAL OF HYDRAULIC RESEARCH, VOL. 40, 2002, NO. 5

Table 1. Statistics for model comparison Concept

Name

Sum of square errors

SSE

Formula j

∑ (obs − calc)

2

i

Fourth order error

S4E

j

∑ (obs − calc)

4

i

Mean absolute error

MAE

j

∑ (obs − calc) i

N Root mean square error RMSE

j

∑ (obs − calc)

2

i

N Efﬁciency Coefﬁcient

COE

Mean relative variance

ARV

(Variance residues) E = 1− (data Variance) j

∑ (obs − calc )

2

i

j

∑ (obs-avg.obs)

2

i

Coefﬁcient of determination

R2

j

∑ (calc-avg.obs) i j

2

∑ (obs-avg. obs)

2

i

relation. Both numerical and graphical results should be considered with respect to predetermined criteria to select the best model. V. Application to San Juan river basin This methodology was applied to obtain a model for streamﬂow prediction for San Juan River basin, Argentina, using climatological data from Pachón meteorological station located at 1900 m of

altitude. Flows were measured at Km 47.3 Station which controls a 20.000 square kilometer watershed. The neural network model was used to predict monthly ﬂows for the period from July through January, using meteorological information gathered during April, May and June. Input variables used were the month number, mean temperature, relative humidity, sunshine hours, wind velocity, snow depth, number of cloudy days, precipitation, ENSO index and runoff volumes for the three preceding months. Output variables were the 7 monthly ﬂows for the period July through January. The San Juan River basin is located in a mountainous region of the Andes Mountain Ranges. The area was formed at the end of the Tertiary and beginning of the Pleistocean, period in which the Andes Mountain Ranges complete their ascent and acquire their current morphology. The upper basin area extends from latitude 30° 33´S (Olivares Mountains) to 32° South (northern slope of the Aconcagua), and from the summits of the Tigre Mountain Range and the Western ﬂank of the Precordillera reaching the summits that separate the slopes of the Atlantic and of the Paciﬁc. The upper part of the basin, which represent approximately 50% of the total area, is located in the central part of the mountain ranges of the Andes, where the altitude ranges from 3,500 m above sea level to a maximum of 7,000 m. The average elevation for the basin is 3,750 m. Precipitation on the basin is mostly snow. The liquid precipitation on the lower basin hardly reach the 100 mm per year. Although there are no rainfall stations on the upper basin, above 3,000 m. studies carried out by Zakalik (1990) characterize the pluvial regime of the different sub-basins. Castaño river with 600 mm per year, Patillos River with 400 mm per year and Teatinos River with 300 mm per year. A study of the CFI, (1961) classiﬁes precipitation in the Castaño River basin as snow, with occasional rains; Los Patos Superior River, as glaciers and little snow; and Blanco River, with prevalence of permanent snow and with greater inﬂuence in the regime of the San Juan River. The annual average ﬂow of the basin of the San Juan River is 2078 Mm3. With the 18 years of available data input-output pairs were prepared. Each pair consisted in 30 input variables and 7 output variables, that is, a total of 666 data elements were used for training and testing the network. From these pairs, 17 were used for train-

SSE

r = 0.2 r = 0.8 cycles Fig. 3. Training curves for different strategies using backpropagation momentum method.

JOURNAL OF HYDRAULIC RESEARCH, VOL. 40, 2002, NO. 5

551

Table 2. Sum of square errors as a function of training cycles Cycles

SSE

500

0.03772

1300

0.04307

2000

0.01719

4000

0.01482

4000 IOS

0.00900

ing and the last pair was used for testing the network. As mentioned the 30 input variables correspond to the observed values at Station Pachón for the months of April, May and June, of number of the month, IOS (Index of Oscillation of the South), monthly precipitation, average temperature, relative average humidity, effective sunshine hours, average monthly wind, maximum snow depth, number of cloudy days and the ﬂow volumes measured at Km 47.3 gaging station. Both stations are dependent of the Department of Hydraulics of the County of San Juan. The 7 variables correspond to the values of the monthly ﬂow volumes measured at the Km 47.3 gaging station, for the months of July to January. A feed-forward ANN model having an architecture 30-20-7 (30 input neurons, 20 hidden neurons and 7 output neurons) trained

by means of back-propagation momentum and using the learning strategy suggested by Mitchell (1997) for scarce data had a good predictive behavior. Connection pruning was carried out to obtain a total of 320 connections. All values included in the pairs were scaled to ± 0.7 with the purpose of homogenizing the magnitudes of the different variables, to be able to use a sigmoid function as an activation function at the network exit, and to extrapolate output values greater than the ones used in the training of the network. The network was initialized with random values between ± 1. The back-propagation algorithm of convergence was studied and different learning rates were tested with different momentum coefﬁcients. The values ﬁnally adopted were 0.9 and 0.7 respectively. An example of two training curves is presented in Figure 3. The sum of squared errors (SSE) is shown in Table 2 for the prediction of streamﬂow as a function of the number of training cycles. Results show that the network has a good generalization performance using 4000 training cycles. Table 3 presents the absolute values of prediction errors in percent and the frequency of different error magnitudes for the 17 year period. It can be observed that errors are only in two cases greater than 14%. Figure 4 shows the mean errors for the 7 month prediction period for two different number of training cycles. This curves indicate the ability of the neural network to generalize its performance. The graph shows that if 4000 cycles are used for

Table 3. Absolute values of prediction errors

552

Year

July

August

September

October

November

December

January

1981

2

6

5

5

4

3

1

1982

1

1

1

0

0

0

0

1983

0

2

2

0

0

0

0

1984

1

2

1

0

0

0

0

1985

5

14

11

4

1

1

0

1986

4

2

4

1

0

0

0

1987

1

2

1

1

0

0

0

1988

0

0

0

1

0

1

0

1989

2

10

7

4

1

2

1

1990

4

4

2

1

0

0

0

1991

0

2

1

1

0

0

0

1992

4

3

8

5

1

1

1

1993

0

3

7

7

1

3

1

1994

6

11

8

4

0

0

1

1995

0

2

2

0

0

0

0

1996

8

6

6

9

1

4

4

1997

6

11

3

22

4

1

17

Error