The New Complex-Valued Wavelet Neural Network

A new complex-valued wavelet neural network is proposed in this paper, by introducing a modified complex-valued back propagation algorithm, in which a new error function is to be minimized by the algorithm. The improvement performance is further confirmed by the simulation results, which show that the modified algorithm is simpler than the conventional algorithm, and has better convergence, better stability and faster running speed.


Introduction
Wavelet analysis theory is considered to be a breakthrough in the Fourier analysis and has been applied in many research areas.Wavelet transform can effectively extract the local information of the signal by scaling and translation to analyze the signal [1].Combining the wavelets with the artificial neural network (ANN), the wavelet neural network (WNN) has been developed [2]- [4].The ANN has many important properties such as learning, generalization, and parallel computation, although it need a large number of neurons in hidden layer and cannot converge quickly.WNN has inherited the good properties of the ANN.Moreover it can converge quickly and give high precision with reduced network size because of the time-frequency localization properties of wavelets [5].
There are two types of WNN structure.The first WNN is pre-wavelet neural network, and the architecture is shown in Figure 1.The network firstly process the input signal using the orthogonal wavelet matrix, then the network put into learning and discriminating.The second WNN is called embedded wavelet neural network, the architecture of which is shown in Figure 2, in which the wavelet transform algorithm is integrated into the feed-forward neural network.In embedded wavelet neural network, wavelet functions are used in the hidden layer of the network as activation functions instead of local functions in time such as Gaussian and sigmoid functions.
Li et al. [6] proposed complex-valued wavelet artificial network (CVWNN) using Haar wavelet as the hidden layer activation function (AF) in complex-valued artificial neural network (CVANN).The complex-valued wavelet neural network is the complex version of the real-valued wavelet neural network, which has complex inputs, outputs, connection weights, dilation and translation parameters, but the nonlinearity of the hidden nodes remains a real-valued function (real-valued wavelet function).CVWNN has expanded its applications in fields dealing with complex numbers such as biomedical image processing [7], telecommunications [8], [9], carotid arterial Doppler ultrasound signals classifying [5], speech recognition [10], signal and image processing with the Fourier transformation [11].
The core algorithm of the CVWNN is complex-valued BP algorithm, which is based on gradient descent often suffers from a local minima problem and has slow convergence.Many methods [12], [13] have been proposed to improve the performance, such as the convergence and the local stability.These methods usually applied adaptive activation function and added a term to the conventional error function to speed up the convergence and prevent the learning from sticking into the local minima.Unfortunately, the local minimal problem and some errors are closely related to the neuron saturation of the activation function.When the actual output approaches the extreme value, the neurons in the output layer and the hidden layer are sensitive to input signals and the propagation chain will almost be blocked.

WNN
Wavelet is a new powerful tool for representing nonlinearity.A function can be represented by the superposition of daughters of a mother wavelet , where can be expressed as (1) and are, respectively, called dilation and translation parameters.The continuous wavelet transform of is defined as And the function can be reconstructed by the inverse wavelet transform (3)

Hidden layer Input layer
Output layer The continuous wavelet transform and its inverse transform are not directly implementable on digital computers.When the inverse wavelet transform (3) is discretized, has the following approximate wavelet-based representation form: (4) where , and are weight coefficients, translations and dilations for each daughter wavelet.This approximation can be expressed as the neural network of Figure 2, which contains wavelet nonlinearities in the artificial neurons rather than the standard sigmoidal nonlinearities.

Traditional complex-valued BP neural network 3.1 The forward propagation process
In this paper, the classical three-layer network [14] is introduced, the architecture of which is shown in Figure 2. The input vector is , which is applied to the input layer of the network.Then the input units distribute the values to the hidden layer units.The net input to the jth hidden unit is (4) where is the complex-valued connection weight from the ith input unit to the jth unit, and is the bias terms in the jth unit.The "h" superscript is the quantities on the hidden layer.The output of the hidden neuron is (5) where "R" superscript and "I" superscript are the quantities on the real part and the imaginary part of the values respectively."F" is the complex-valued activation function, which is (6) where refers to the formula (4).The net input and the output of the jth output unit are (7) (8) where the "o" superscript is the quantities on the output layer.The backward propagation refers to the error signal backward propagation.is defined as the error at a single output unit, where "p" refers to the pth training vector, and "k" refers to the kth unit.The error is minimized by the complex algorithm.Since the size of complex-valued number cannot be compared, the error energy function is as follows, which is the sum of the squares of the errors of all output units， (9) where the "*" is means complex conjugation and M is the node number of the output layer.In order to determine the weight changing direction, it is necessary to calculate the negative of the gradient of the according to the real and imaginary part of the coefficients.The weights can be written as (10) Firstly, the adaption rule of the output layer is considered.According to the steepest descent rule, the weights can be updated as (11) (12) where is learning step, which is a positive constant.Combining (11) and ( 12), we can have (13) Finally，according to the error function formula, we can get (14) The weight update equations can be summarized by defining a quantity (15) When the active function (AF) is sigmoid function, such as (16) which is one of the most widely used AF for the artificial neural network.The first-order differential of the AF is

The backward propagation process
w o kj;I (t + 1) = w o kj;I (t) ¡ ´@E p @w o kj;I (t) where .
In this section, it is explained the results of research and at the same time is given the comprehensive discussion.Results can be presented in figures, graphs, tables and others that make the reader understand easily [2], [5].The discussion can be made in several sub-chapters.

New complex-valued wavelet neural networks
The complex-valued WNN algorithm as described above has the following questions.When the actual value approaches the extreme value, i.e., 0 or 1, the factors and makes the error signal very small.This means that an output unit can be maximally wrong without producing a strong error signal with which the synaptic weight should be significantly adjusted.The search for a minimum in the error will be retarded.M.Jiang et al. have introduced a modified error function for the complexvalued BP neural network to overcome the above shortcomings and avoid the delay of the convergence of the network.Instead of minimizing the squares of the differences between the actual outputs and desired outputs [16], in which the error function to be minimized is as follows, (22) where M is the total number of the output neurons, are the real part and the imaginary part of the actual outputs of the kth output neuron respectively.are , so" true" error is measured.

The adaptation rule of the hidden layer
The adaptation rule of the hidden layer still is (28) Similiarly, we can have where .Obviously the above formula is the same with the traditional CVBP algorithm in form.But there are no and , the "true" error can be measured.In order to verify the validity and practicability of the proposed method, the paper carries out the processing of the XOR problem.The learning pattern is called the similar XOR problem, which is shown in Table 1.The real part of the output can be seen as the XOR of the input's real and input's imaginary part, and the imaginary part of the output is equal to the real part of the input.
This problem has been simulated with a 1-3-1 complex-value network in [12,15].In this paper the new complex-valued WNN and conventional CVBPNN and CVWNN are applied to resolve the similar XOR problem.For the above methods, learning rate and maximum iteration number are chosen as 0.1 and 5,000, respectively.The architecture of the complex-valued neural network is 1-2-1.When the minimal error is 0.1, 0.01 and 0.001, the learning curve for the similar XOR problem is shown is Figure 3, Figure 4, and Figure 5 respectively.The success rate and average learning epochs are shown in Table 2.When the error criteria was used as the Ep = 0.1, the new CVWNN has 100% success rate, and its average learning epoch is only 153.This is a third of the conventional CVWNN's 475 epochs.When the error criteria is Ep = 0.01 and Ep = 0.001, the improved algorithm is faster than conventional, which can be seen from the Figure 3 and Figure 4 and Table 2  A comparison of the proposed method and the conventional CVWNN is illustrated in Figure 3, 4, and 5, from which we can see that the proposed method has better stability convergence performance, and faster running speed than the conventional CVWNN.

Conclusion
In this paper, a new CVWNN proposed, whose inputs, outputs and weights are all complex-valued, and the nonlinear activation function remains real-valued.The back propagation learning algorithm for training the complex-valued wavelet network is modified by introducing a new error function.The performance of the proposed CVWNN is illustrated with application to the XOR.The simulation results demonstrate that the CVWNN has better stability convergence performance, and faster running speed than the conventional CVWNN.Anymore， in signal processing and communication areas, there are a large number of complex-vuaed number to be dealt with,, thus the proposed CVWNN provides a powerful tool for such cases.

Figure 1 .Figure 2 .
Figure 1.The architecture of the pre-wavelet neural network the output layer activation function , the weight-update equation in the output layer can be written (19) Similarly we can get the adaption rule of the hidden layer, (20)

TELKOMNIKA 4 . 1
Vol. 12, No. 3, September 2014: 613 -622 618 the real part and the imaginary part of the desired outputs of the kth output neuron.Meanwhile the back propagation will be changed, and the forward propagation remains unchanged.The adaptation rule of the output layer The gradients of modified error function with respect simply the update equation, we also introduce the error term propagation is now directly propagated o the difference between the desired value and the actual value.Formula (27) lacks the factors and

5 2 
The new complex-valued WNN Complex-valued wavelet artificial neural network used Mexican hat wavelet and Haar wavelet function as hidden layer AF instead of logarithmic sigmoid activation function.In this paper Morlet wavelet (or Gabor wavelet) function is chosen as the theee hidden layer AF, which is defined by (32) in which via experimentation.Like new complex-valued neural network, activation function of output layer is chosen as logarithmic sigmoid in proposed CVWANN structures.Error function is chosen as the modified error function in Eqs.(22).Mathematical formulations of proposed CVWNN structures are obtained by using wavelet function instead of using logarithmic sigmoid function.CVWNN architectures used in this paper is shown in Figure 2. 6. Simulation results Ã Morlet = cos(ax)e ¡bx ISSN: 1693-6930 TELKOMNIKA Vol. 12, No. 3, September 2014: 613 -622 620

TELKOMNIKA 621 Figure 3 .Figure 4 .
Figure 3.The comparison of the learning curve between the proposed CVWNN and the conventional CVWNN, when the minimal error is 0.01 Figure 4.The comparison of the learning curve between the proposed CVWNN and the conventional CVWNN, when the minimal error is 0.001

Table 1 .
. Learning pattern for similar XOR problem

Table 2 .
Simulation results for similar XOR problem