text:
Category Archives: Classification
A Gentle Introduction to Artificial Neural Networks
Though many phenomena in the world can be adequately modeled using linear regression or classification, most interesting phenomena are generally nonlinear in nature. In order to deal with nonlinear phenomena, there have been a diversity of nonlinear models developed. For example parametric models assume that data follow some parameteric class of nonlinear function (e.g. polynomial, power, or exponential), then fine-tune the shape of the parametric function to fit observed data. However this approach is only helpful if data are fit nicely by the available catalog of parametric functions. Another approach, kernel-based methods, transforms data non-linearly into an abstract space that measures distances between observations, then predicts new values or classes based on these distances. However, kernel methods generally involve constructing a kernel matrix that depends on the number of training observations and can thus be prohibitive for large data sets. Another class of models, the ones that are the focus of this post, are artificial neural networks (ANNs). ANNs are nonlinear models motivated by the physiological architecture of the nervous system. They involve a cascade of simple nonlinear computations that when aggregated can implement robust and complex nonlinear functions. In fact, depending on how they are constructed, ANNs can approximate any nonlinear function, making them a quite powerful class of models (note that this property is not reserved for ANNs; kernel methods are also considered "universal approximators"; however, it turns out that neural networks with multiple layers are more efficient at approximating arbitrary functions than other methods. I refer the interested reader to more in-depth discussion on the topic.).
In recent years ANNs that use multiple stages of nonlinear computation (aka "deep learning") have been able obtain outstanding performance on an array of complex tasks ranging from visual object recognition to natural language processing. I find ANNs super interesting due to their computational power and their intersection with computational neuroscience. However, I've found that most of the available tutorials on ANNs are either dense with formal details and contain little information about implementation or any examples, while others skip a lot of the mathematical detail and provide implementations that seem to come from thin air. This post aims at giving a more complete overview of ANNs, including (varying degrees of) the math behind ANNs, how ANNs are implemented in code, and finally some toy examples that point out the strengths and weaknesses of ANNs.
Single-layer Neural Networks
The simplest ANN (Figure 1) takes a set of observed inputs , multiplies each of them by their own associated weight , and sums the weighted values to form a pre-activation .Oftentimes there is also a bias that is tied to an input that is always +1 included in the preactivation calculation. The network then transforms the pre-activation using a nonlinear activation function to output a final activation .
There are many options available for the form of the activation function , and the choice generally depends on the task we would like the network to perform. For instance, if the activation function is the identity function:
which outputs continuous values , then the network implements a linear model akin to used in standard linear regression. Another choice for the activation function is the logistic sigmoid:
which outputs values . When the network outputs use the logistic sigmoid activation function, the network implements linear binary classification. Binary classification can also be implemented using the hyperbolic tangent function, , which outputs values (note that the classes must also be coded as either -1 or 1 when using . Single-layered neural networks used for classification are often referred to as "perceptrons," a name given to them when they were first developed in the late 1950s.
To get a better idea of what these activation function do, their outputs for a given range of input values are plotted in the left of Figure 2. We see that the logistic and tanh activation functions (blue and green) have the quintessential sigmoidal "s" shape that saturates for inputs of large magnitude. This behavior makes them useful for categorization. The identity / linear activation (red), however forms a linear mapping between the input to the activation function, which makes it useful for predicting continuous values.
A key property of these activation functions is that they are all smooth and differentiable. We'll see later in this post why differentiability is important for training neural networks. The derivatives for each of these common activation functions are given by (for mathematical details on calculating these derivatives, see this post):
Each of the derivatives are plotted in the right of Figure 2. What is interesting about these derivatives is that they are either a constant (i.e. 1), or are can be defined in terms of the original function. This makes them extremely convenient for efficiently training neural networks, as we can implement the gradient using simple manipulations of the feed-forward states of the network.
Code Block 1: Defines standard activation functions and generates Figure 2:
% DEFINE A FEW COMMON ACTIVATION FUNCTIONS gLinear = inline('z','z'); gSigmoid = inline('1./(1+exp(-z))','z'); gTanh = inline('tanh(z)','z'); % ...DEFINE THEIR DERIVATIVES gPrimeLinear = inline('ones(size(z))','z'); gPrimeSigmoid = inline('1./(1+exp(-z)).*(1-1./(1+exp(-z)))','z'); gPrimeTanh = inline('1-tanh(z).^2','z'); % VISUALIZE EACH g(z) z = linspace(-4,4,100); figure set(gcf,'Position',[100,100,960,420]) subplot(121); hold on; h(1) = plot(z,gLinear(z),'r','Linewidth',2); h(2) = plot(z,gSigmoid(z),'b','Linewidth',2); h(3) = plot(z,gTanh(z),'g','Linewidth',2); set(gca,'fontsize',16) xlabel('z') legend(h,{'g_{linear}(z)','g_{logistic}(z)','g_{tanh}(z)'},'Location','Southeast') title('Some Common Activation Functions') hold off, axis square, grid ylim([-1.1 1.1]) % VISUALIZE EACH g'(z) subplot(122); hold on h(1) = plot(z,gPrimeLinear(z),'r','Linewidth',2); h(2) = plot(z,gPrimeSigmoid(z),'b','Linewidth',2); h(3) = plot(z,gPrimeTanh(z),'g','Linewidth',2); set(gca,'fontsize',16) xlabel('z') legend(h,{'g''_{linear}(z)','g''_{logistic}(z)','g''_{tanh}(z)'},'Location','South') title('Activation Function Derivatives') hold off, axis square, grid ylim([-.5 1.1])
Multi-layer Neural Networks
As was mentioned above, single-layered networks implement linear models, which doesn't really help us if we want to model nonlinear phenomena. However, by considering the single layer network diagrammed in Figure 1 to be a basic building block, we can construct more complicated networks, ones that perform powerful, nonlinear computations. Figure 3 demonstrates this concept. Instead of a single layer of weights between inputs and output, we introduce a set of single-layer networks between the two. This set of intermediate networks is often referred to as a "hidden" layer, as it doesn't directly observe input or directly compute the output. By using a hidden layer, we form a mult-layered ANN. Though there are many different conventions for declaring the actual number of layers in a multi-layer network, for this discussion we will use the convention of the number of distinct sets of trainable weights as the number of layers. For example, the network in Figure 3 would be considered a 2-layer ANN because it has two layers of weights: those connecting the inputs to the hidden layer (), and those connecting the output of the hidden layer to the output layer().
Multi-layer neural networks form compositional functions that map the inputs nonlinearly to outputs. If we associate index i with the input layer, index j with the hidden layer, and index k with the output layer, then an output unit in the network diagrammed in Figure 3 computes an output value given and input via the following compositional function:
Here is the is the pre-activation values for units for layer , is the activation function for units in that layer (assuming they are the same), and is the output activation for units in that layer. The weight links the outputs of units feeding into layer to the activation function of units for that layer. The term is the bias for units in layer .
As with the single-layered ANN, the choice of activation function for the output layer will depend on the task that we would like the network to perform (i.e. categorization or regression), and follows similar rules outlined above. However, it is generally desirable for the hidden units to have nonlinear activation functions (e.g. logistic sigmoid or tanh). This is because multiple layers of linear computations can be equally formulated as a single layer of linear computations. Thus using linear activations for the hidden layers doesn't buy us much. However, as we'll see shortly, using linear activations for the output unit activation function (in conjunction with nonlinear activations for the hidden units) allows the network to perform nonlinear regression.
Training neural networks & gradient descent
Training neural networks involves determining the network parameters that minimize the errors that the network makes. This first requires that we have a way of quantifying error. A standard way of quantifying error is to take the squared difference between the network output and the target value:
(Note that the squared error is not chosen arbitrarily, but has a number of theoretical benefits and considerations. For more detail, see the following post) With an error function in hand, we then aim to find the setting of parameters that minimizes this error function. This concept can be interpreted spatially by imagining a "parameter space" whose dimensions are the values of each of the model parameters, and for which the error function will form a surface of varying height depending on its value for each parameter. Model training is thus equivalent to finding point in parameter space that makes the height of the error surface small.
To get a better intuition behind this concept, let's define a super simple neural network, one that has a single input and a single output (Figure 4, bottom left). For further simplicity, we'll assume the network has no bias term and thus has a single parameter, . We will also assume that the output layer uses the logistic sigmoid activation function. Accordingly, the network will map some input value onto a predicted output via the following function.
Now let's say we want this simple network to learn the identity function: given an input of 1 it should return a target value of 1. Given this target value we can now calculate the value of the error function for each setting of . Varying the value of from -10 to 10 results in the error surface displayed in the left of Figure 4. We see that the error is small for large positive values of , while the error is large for strongly negative values of . This not surprising, given that the output activation function is the logistic sigmoid, which will map large values onto an output of 1.
Things become more interesting when we move from a single-layered network to a multi-layered network. Let's repeat the above exercise, but include a single hidden node between the input and the output (Figure 4, bottom right). Again, we will assume no biases, and logistic sigmoid activations for both the hidden and output nodes. Thus the network will have two parameters: . Accordingly the 2-layered network will predict an output with the following function:
Now, if we vary both and , we obtain the error surface in the right of Figure 4.
We see that the error function is minimized when both and are large and positive. We also see that the error surface is more complex than for the single-layered model, exhibiting a number of wide plateau regions. It turns out that the error surface gets more and more complicated as you increase the number of layers in the network and the number of units in each hidden layer. Thus, it is important to consider these phenomena when constructing neural network models.
Code Block 2: generates Figure 4 (assumes you have run Code Block 1):
% VISUALIZE ERROR SURFACE OF SIMPLE ANNS E = {}; [w1,w2] = meshgrid(linspace(-10,10,50)); g = gSigmoid; target = 1; net1Output = g(w1.*target); net2Output = g(w2.*g(w1.*target)); E{1} = (net1Output - target).^2; E{2} = (net2Output - target).^2; figure for ii = 1:2 set(gcf,'Position',[100,100,960,420]) subplot(1,2,ii) surf(w1,w2,E{ii}); shading faceted; colormap(flipud(hot)); caxis([0,max(max(E{ii}))]) set(gca,'fontsize',16) xlabel('w_{1}'), ylabel('w_{2}'), zlabel('E(w)') axis square; title(sprintf('Error Surface: %d-layer Network',ii)) [az, el] = view; view([az + 180, el]); set(gcf,'position',[100,100,1020,440]) drawnow end
The examples in Figure 4 gives us a qualitative idea of how to train the parameters of an ANN, but we would like a more automatic way of doing so. Generally this problem is solved using gradient descent: The gradient descent algorithm first calculates the derivative / gradient of the error function with respect to each of the model parameters. This gradient information will give us the direction in parameter space that decreases the height of the error surface. We then take a step in that direction and repeat, iteratively calculating the gradient and taking steps in parameter space.
The backpropagation algorithm
It turns out that the gradient information for the ANN error surface can be calculated efficiently using a message passing algorithm known as the backpropagation algorithm. During backpropagation, input signals are forward-propagated through the network toward the outputs, and network errors are then calculated with respect to target variables and back-propagated backwards towards the inputs. The forward and backward signals are then used to determine the direction in the parameter space to move that lowers the network error.
The formal calculations behind the backpropagation algorithm can be somewhat mathematically involved and may detract from the general ideas behind the learning algorithm. For those readers who are interested in the math, I have provided the formal derivation of the backpropagation algorithm in the following post (for those of you who are not interested in the math, I would also encourage you go over the derivation and try to make connections to the source code implementations provided later in the post).
Figure 5 demonstrates the key steps of the backpropagation algorithm. The main concept underlying the algorithm is that for a given observation we want to determine the degree of "responsibility" that each network parameter has for mis-predicting a target value associated with the observation. We then change that parameter according to this responsibility so that it reduces the network error.
In order to determine the network error, we first propagate the observed input forward through the network layers. This is Step I of the backpropagation algorithm, and is demonstrated in Figure 5-I. Note that in the Figure could be considered network output (for a network with one hidden layer) or the output of a hidden layer that projects the remainder of the network (in the case of a network with more than one hidden layer). For this discussion, however, we assume that the index k is associated with the output layer of the network, and thus each of the network outputs is designated by . Also note that when implementing this forward-propagation step, we should keep track of the feed-forward pre-activations and activations for all layers , as these will be used for calculating backpropagated errors and error function gradients.
Step II of the algorithm is to calculate the network output error and backpropagate it toward the input. Let's again that we are using the sum of squared differences error function:
where we sum over the values of all output units (one in this example). We can now define an "error signal" at the output node that will be backpropagated toward the input. The error signal is calculated as follows:
Thus the error signal essentially weights the gradient of the error function by the gradient of the output activation function (notice there is a term is used in this calculation, which is why we keep it around during the forward-propagation step). We can continue backpropagating the error signal toward the input by passing through the output layer weights , summing over all output nodes, and passing the result through the gradient of the activation function at the hidden layer (Figure 5-II). Performing these operations results in the back-propagated error signal for the hidden layer, :
For networks that have more than one hidden layer, this error backpropagation procedure can continue for layers , etc.
Step III of the backpropagation algorithm is to calculate the gradients of the error function with respect to the model parameters at each layer using the forward signals , and the backward error signals . If one considers the model weights at a layer as linking the forward signal to the error signal (Figure 5-III), then the gradient of the error function with respect to those weights is:
Note that this result is closely related to the concept of Hebbian learning in neuroscience. Thus the gradient of the error function with respect to the model weight at each layer can be efficiently calculated by simply keeping track of the forward-propagated activations feeding into that layer from below, and weighting those activations by the backward-propagated error signals feeding into that layer from above!
What about the bias parameters? It turns out that the same gradient rule used for the weight weights applies, except that "feed-forward activations" for biases are always +1 (see Figure 1). Thus the bias gradients for layer are simply:
The fourth and final step of the backpropagation algorithm is to update the model parameters based on the gradients calculated in Step III. Note that the gradients point in the direction in parameter space that will increase the value of the error function. Thus when updating the model parameters we should choose to go in the opposite direction. How far do we travel in that direction? That is generally determined by a user-defined step size (aka learning rate) parameter, . Thu,s given the parameter gradients and the step size, the weights and biases for a given layer are updated accordingly:
To train an ANN, the four steps outlined above and in Figure 5 are repeated iteratively by observing many input-target pairs and updating the parameters until either the network error reaches a tolerably low value, the parameters cease to update (convergence), or a set number of parameter updates has been achieved. Some readers may find the steps of the backpropagation somewhat ad hoc. However, keep in mind that these steps are formally coupled to the calculus of the optimization problem. Thus I again refer the curious reader to check out the derivation in order to make connections between the algorithm and the math.
Example: learning the OR & AND logical operators using a single layer neural network
Here we go over an example of training a single-layered neural network to perform a classification problem. The network is trained to learn a set of logical operators including the AND, OR, or XOR. To train the network we first generate training data. The inputs consist of 2-dimensional coordinates that span the input values values for a 2-bit truth table:
We then perturb these observations by adding Normally-distributed noise. To generate target variables, we categorize each observations by applying one of logic operators (See Figure 6) to the original (no-noisy) coordinates. We then train the network with the noisy inputs and binary categories targets using the gradient descent / backpropagation algorithm. The code implementation of the network and training procedures, as well as the resulting learning process are displayed below. (Note that in this implementation, I do not use the feed-forward activations to calculate the gradients as suggested above. This is simply to make the implementation of the learning algorithm more explicit in terms of the math. The same situation also applies to the other examples in this post).
Code Block 3: Implements and trains a single-layer neural network for classification to learn logical operators (assumes you have run Code Block 1):
%% EXAMPLE: SINGLE-LAYERED NETWORK % DEFINE DATA AND TARGETS data = [0 0; 0 1; 1 0; 1 1;]; classAND = and(data(:,1)>0,data(:,2)>0); classOR = or(data(:,1)>0,data(:,2)>0); classXOR = xor(data(:,1)>0,data(:,2)>0); % THE TYPE OF TRUTH TABLE TO LEARN (UNCOMMENT FOR OTHERS) classes = classOR % classes = classAND; % classes = classXOR; % MAKE MULTIPLE NOISY TRAINING OBSERVATIONS nRepats = 30; data = repmat(data, [nRepats, 1]); classes = repmat(classes, [nRepats, 1]); data = data + .15*randn(size(data)); % SHUFFLE DATA shuffleIdx = randperm(size(data,1)); data = data(shuffleIdx,:); classes = classes(shuffleIdx); % INITIALIZE MODEL PARAMETERS [nObs,nInput] = size(data); % # OF INPUT DIMENSIONS nOutput = 1; % # OF TARGET/OUTPUT DIMENSIONS lRate = 3; % LEARNING RATE FOR PARAMETERS UPDATE nIters = 80; % # OF ITERATIONS % DECLARE ACTIVATION FUNCTIONS (AND DERIVATIVES) g_out = gSigmoid; gPrime_out = gPrimeSigmoid; % INITIALIZE RANDOM WEIGHTS W_out = (rand(nInput,nOutput)-.5); b_out = (rand(1,nOutput)-.5); % SOME OTHER INITIALIZATIONS % (FOR VISUALIZATION) visRange = [-.2 1.2]; [xx,yy] = meshgrid(linspace(visRange(1), visRange(2),100)); iter = 1; mse = zeros(1,nIters); figure set(gcf,'Position',[100,100,960,420]) while 1 err = zeros(1,nObs); % LOOP THROUGH THE EXAMPLES for iO = 1:nObs % GET CURRENT NETWORK INPUT DATA AND TARGET input = data(iO,:); target = classes(iO); %% I. FORWARD PROPAGATE DATA THROUGH NETWORK z_out = input*W_out + b_out; % OUTPUT UNIT PRE-ACTIVATIONS a_out = g_out(z_out); % OUTPUT UNIT ACTIVATIONS %% II. BACKPROPAGATE ERROR SIGNAL % CALCULATE ERROR DERIVATIVE W.R.T. OUTPUT delta_out = gPrime_out(z_out).*(a_out - target); %% III. CALCULATE GRADIENT W.R.T. PARAMETERS... dEdW_out = delta_out*input; dEdb_out = delta_out*1; %% IV. UPDATE NETWORK PARAMETERS W_out = W_out - lRate*dEdW_out'; b_out = b_out - lRate*dEdb_out'; % CALCULATE ERROR FUNCTION err(iO) = .5*(a_out-target).^2; end mse(iter) = mean(err); % DISPLAY LEARNING clf; subplot(121); hold on; set(gca,'fontsize',16) netOut = g_out(bsxfun(@plus,[xx(:),yy(:)]*W_out, b_out)); contourf(xx,yy,reshape(netOut,100,100)); colormap(flipud(spring)) hold on; gscatter(data(:,1),data(:,2),classes,[0 0 0 ; 1 1 1],[],20,'off'); title(sprintf('Iteration %d',iter)) xlim([visRange(1) visRange(2)]),ylim([visRange(1) visRange(2)]); axis square subplot(122); set(gca,'fontsize',16) plot(1:iter,mse(1:iter)); xlabel('Iteration') ylabel('Mean Squared Error') axis square m1(iter) = getframe(gcf); if iter >= nIters break end iter = iter + 1; end
Figure 7 displays the procedure for learning the OR mapping. The left plot displays the training data and the network output at each iteration. White dots are training points categorized "1" while black dots are categorized "0". Yellow regions are where the network predicts values of "0", while magenta highlights areas where the network predicts "1". We see that the single-layer network is able to easily separate the two classes. The right plot shows how the error function decreases with each training iteration. The smooth trajectory of the error indicates that the error surface is also fairly smooth.
Figure 8 demonstrates an analogous example, but instead learning the AND operator (by executing Code Block 3, after un-commenting line 11). Again, the categories can be easily separated by a plane, and thus the single-layered network easily learns an accurate predictor of the data.
Going Deeper: nonlinear classification and multi-layer neural networks
Figures 7 and 8 demonstrate how a single-layered ANN can easily learn the OR and AND operators. This is because the categorization criterion for these logical operators can be represented in the input space by a single linear function (i.e. line/plane). What about more complex categorization criterion that cannot be represented by a single plane? An example of a more complex binary classification criterion is the XOR operator (Figure 6, far right column).
Below we attempt to train the single-layer network to learn the XOR operator (by executing Code Block 3, after un-commenting line 12). The single layer network is unable to learn this nonlinear mapping between the inputs and the targets. However, it turns out we can learn the XOR operator using a multi-layered neural network.
Below we train a two-layer neural network on the XOR dataset. The network incorporates a hidden layer with 3 hidden units and logistic sigmoid activation functions for all units in the hidden and output layers (see Code Block 4, lines 32-33).
Code Block 4: Implements and trains a two-layer neural network for classification to learn XOR operator and more difficult "ring" problem (Figures 10 & 11; assumes you have run Code Block 1):
%% EXAMPLE: MULTI-LAYER NEURAL NETWORK FOR CLASSIFICATION data = [0 0; 0 1; 1 0; 1 1;]; classXOR = xor(data(:,1)>0,data(:,2)>0); % THE TYPE OF TRUTH TABLE TO LEARN classes = classXOR; % UNCOMMENT FOR MOR DIFFICULT DATA... % data = [data; .5 .5; 1 .5; 0 .5; .5 0; .5 1]; % classRing = [1; 1; 1; 1; 0; 1; 1; 1; 1]; % classes = classRing; % CREATE MANY NOISY OBSERVATIONS nRepats = 30; data = repmat(data, [nRepats, 1]); classes = repmat(classes, [nRepats, 1]); data = data + .15*randn(size(data)); % SHUFFLE OBSERVATIONS shuffleIdx = randperm(size(data,1)); data = data(shuffleIdx,:); classes = classes(shuffleIdx); % INITIALIZE MODEL PARAMETERS [nObs,nInput] = size(data); % # OF INPUT DIMENSIONS nHidden = 3; % # OF HIDDEN UNITS lRate = 2; % LEARNING RATE FOR PARAMETERS UPDATE nIters = 300; % # OF ITERATIONS % DECLARE ACTIVATION FUNCTIONS (AND DERIVATIVES) g_hid = gSigmoid; gPrime_hid = gPrimeSigmoid; g_out = gSigmoid; gPrime_out = gPrimeSigmoid; % INITIALIZE WEIGHTS W_hid = (rand(nInput,nHidden)-.5); b_hid = (rand(1,nHidden)-.5); W_out = (rand(nHidden,nOutput)-.5); b_out = (rand(1,nOutput)-.5); iter = 1; mse = zeros(1,nIters); figure set(gcf,'Position',[100,100,960,420]) % MAIN TRAINING ALGORITHM while 1 err = zeros(1,nObs); % LOOP THROUGH THE EXAMPLES for iO = 1:nObs % GET CURRENT NETWORK INPUT DATA AND TARGET input = data(iO,:); target = classes(iO); %% I. FORWARD PROPAGATE DATA THROUGH NETWORK z_hid = input*W_hid + b_hid; % HIDDEN UNIT PRE-ACTIVATIONS a_hid = g_hid(z_hid); % HIDDEN UNIT ACTIVATIONS z_out = a_hid*W_out + b_out; % OUTPUT UNIT PRE-ACTIVATIONS a_out = g_out(z_out); % OUTPUT UNIT ACTIVATIONS %% II. BACKPROPAGATE ERROR SIGNAL % CALCULATE ERROR DERIVATIVE W.R.T. OUTPUT delta_out = gPrime_out(z_out).*(a_out - target); % CALCULATE ERROR CONTRIBUTIONS FOR HIDDEN NODES... delta_hid = gPrime_hid(z_hid)'.*(delta_out*W_out); %% III. CALCULATE GRADIENT W.R.T. PARAMETERS... dEdW_out = delta_out*a_hid; dEdb_out = delta_out*1; dEdW_hid = delta_hid*input; dEdb_hid = delta_hid*1; %% IV. UPDATE NETWORK PARAMETERS W_out = W_out - lRate*dEdW_out'; b_out = b_out - lRate*dEdb_out'; W_hid = W_hid - lRate*dEdW_hid'; b_hid = b_hid - lRate*dEdb_hid'; % CALCULATE ERROR FUNCTION err(iO) = .5*(a_out-target).^2; end mse(iter) = mean(err); % DISPLAY LEARNING clf; subplot(121); hold on; set(gca,'fontsize',16) netOut = g_out(bsxfun(@plus,g_hid(bsxfun(@plus,[xx(:),yy(:)]*W_hid, b_hid))*W_out, b_out)); contourf(xx,yy,reshape(netOut,100,100)); colormap(flipud(spring)) hold on; gscatter(data(:,1),data(:,2),classes,[0 0 0; 1 1 1],[],20,'off'); title(sprintf('Iteration %d',iter)) xlim([visRange(1), visRange(2)]),ylim([visRange(1), visRange(2)]); axis square subplot(122); set(gca,'fontsize',16) plot(1:iter,mse(1:iter)); xlabel('Iteration') ylabel('Mean Squared Error') axis square m2(iter) = getframe(gcf); if iter >= nIters break end iter = iter + 1; end
Figure 10 displays the learning process for the 2-layer network. The formatting for Figure 10 is analogous to that for Figures 7-9. The 2-layer network is easily able to learn the XOR operator. We see that by adding a hidden layer between the input and output, the ANN is able to learn the nonlinear categorization criterion!
Figure 11 shows the results for learning a even more difficult nonlinear categorization function: points in and around are categorized as "0", while points in a ring surrounding the "0" datapoints are categorized as a "1" (Figure 11). This example is run by executing Code Block 4 after un-commenting lines 9-11.
Figure 11 shows the learning process. Again formatting is analogous to the formatting in Figures 8-10. The 2-layer ANN is able to learn this difficult classification criterion.
Example: Neural Networks for Regression
The previous examples demonstrated how ANNs can be used for classification by using a logistic sigmoid as the output activation function. Here we demonstrate how, by making the output activation function the linear/identity function, the same 2-layer network architecture can be used to implement nonlinear regression.
For this example we define a dataset comprised of 1D inputs, that range from . We then generate noisy targets according to the function:
where is a nonlinear data-generating function and is Normally-distributed noise. We then construct a two-layered network with tanh activation functions used in the hidden layer and linear outputs. For this example we set the number of hidden units to 3 and train the model as we did for categorization using gradient descent / backpropagation. The results of the example are displayed below.
Code Block 5: Trains two-layer network for regression problems (Figures 11 & 12; assumes you have run Code Block 1):
%% EXAMPLE: NONLINEAR REGRESSION % DEFINE DATA-GENERATING FUNCTIONS f(x) xMin = -5; xMax = 5; xx = linspace(xMin, xMax, 100); f = inline('2.5 + sin(x)','x'); % f = inline('abs(x)','x'); % UNCOMMENT FOR FIGURE 13 yy = f(xx) + randn(size(xx))*.5; % FOR SHUFFLING OBSERVATIONS shuffleIdx = randperm(length(xx)); data = xx; targets = yy; % INITIALIZE MODEL PARAMETERS nObs = length(data); % # OF INPUT DIMENSIONS nInput = 1; % # OF INPUTS nHidden = 3; % # OF HIDDEN UNITS nOutput = 1; % # OF TARGET/OUTPUT DIMENSIONS lRate = .15; % LEARNING RATE FOR PARAMETERS UPDATE nIters = 200; % # OF ITERATIONS cols = lines(nHidden); % DECLARE ACTIVATION FUNCTIONS (AND DERIVATIVES) g_hid = gTanh; % HIDDEN UNIT ACTIVATION gPrime_hid = gPrimeTanh; % GRAD OF HIDDEN UNIT ACTIVATION g_out = gLinear; % OUTPUT ACTIVATION gPrime_out = gPrimeLinear; % GRAD. OF OUTPUT ATIVATION % % INITIALIZE WEIGHTS W_hid = (rand(nInput,nHidden)-.5); b_hid = (rand(1,nHidden)-.5); W_out = (rand(nHidden,nOutput)-.5); b_out = (rand(1,nOutput)-.5); % INITIALIZE SOME THINGS.. % (FOR VISUALIZATION) mse = zeros(1,nIters); visRange = [xMin, xMax]; figure set(gcf,'Position',[100,100,960,420]) iter = 1; while 1 err = zeros(1,nObs); % LOOP THROUGH THE EXAMPLES for iO = 1:nObs % GET CURRENT NETWORK INPUT DATA AND TARGET input = data(iO); target = targets(iO); %% I. FORWARD PROPAGATE DATA THROUGH NETWORK z_hid = input*W_hid + b_hid; % HIDDEN UNIT PRE-ACTIVATIONS a_hid = g_hid(z_hid); % HIDDEN UNIT ACTIVATIONS z_out = a_hid*W_out + b_out; % OUTPUT UNIT PRE-ACTIVATIONS a_out = g_out(z_out); % OUTPUT UNIT ACTIVATIONS %% II. BACKPROPAGATE ERROR SIGNAL % CALCULATE ERROR DERIVATIVE W.R.T. OUTPUT delta_out = gPrime_out(z_out).*(a_out - target); %% CALCULATE ERROR CONTRIBUTIONS FOR HIDDEN NODES... delta_hid = gPrime_hid(z_hid)'.*(delta_out*W_out); %% III. CALCULATE GRADIENT W.R.T. PARAMETERS... dEdW_out = delta_out*a_hid; dEdb_out = delta_out*1; dEdW_hid = delta_hid*input; dEdb_hid = delta_hid*1; %% IV. UPDATE NETWORK PARAMETERS W_out = W_out - lRate*dEdW_out'; b_out = b_out - lRate*dEdb_out'; W_hid = W_hid - lRate*dEdW_hid'; b_hid = b_hid - lRate*dEdb_hid'; % CALCULATE ERROR FUNCTION FOR BATCH err(iO) = .5*(a_out-target).^2; end mse(iter) = mean(err); % UPDATE ERROR % DISPLAY LEARNING clf; subplot(121); hold on; set(gca,'fontsize',14) plot(xx,f(xx),'m','linewidth',2); hold on; scatter(xx, yy ,'m'); % PLOT TOTAL NETWORK OUTPUT netOut = g_out(g_hid(bsxfun(@plus, xx'*W_hid, b_hid))*W_out + b_out); plot(xx, netOut, 'k','linewidth', 2) % PLOT EACH HIDDEN UNIT'S OUTPUT FUNCTION for iU = 1:nHidden plot(xx,g_hid(xx*W_hid(iU) + b_hid(iU)),'color',cols(iU,:),'Linewidth',2, ... 'Linestyle','--'); end % TITLE AND LEGEND title(sprintf('Iteration %d',iter)) xlim([visRange(1) visRange(2)]),ylim([visRange(1) visRange(2)]); axis square legend('f(x)', 'Targets', 'Network Output','Hidden Unit Outputs','Location','Southwest') % PLOT ERROR subplot(122); set(gca,'fontsize',14) plot(1:iter,mse(1:iter)); xlabel('Iteration') ylabel('Mean Squared Error') axis square; drawnow % ANNEAL LEARNING RATE lRate = lRate *.99; if iter >= nIters break end iter = iter + 1; end
The training procedure for is visualized in the left plot of Figure 12. The data-generating function is plotted as the solid magenta line, and the noisy target values used to train the network are plotted as magenta circles. The output of the network at each training iteration is plotted in solid black while the output of each of the tanh hidden units is plotted in dashed lines. This visualization demonstrates how multiple nonlinear functions can be combined to form the complex output target function. The mean squared error at each iteration is plotted in the right plot of Figure 12. We see that the error does not follow a simple trajectory during learning, but rather undulates, demonstrating the non-convexity of the error surface.
Figure 13 visualizes the training procedure for trying to learn a different nonlinear function, namely (by running Code Block 5, after un-commenting out line 7). Again, we see how the outputs of the hidden units are combined to fit the desired data-generating function. The mean squared error again follows an erratic path during learning.
Notice for this example that I added an extra implementation detail known as simulated annealing (line 118) that was absent in the classification examples. This technique decreases the learning rate after every iteration thus making the algorithm take smaller and smaller steps in parameter space. This technique can be useful when the gradient updates begin oscillating between two or more locations in the parameter space. It is also helpful for influencing the algorithm to settle down into a steady state.
Wrapping up
In this post we covered the main ideas behind artificial neural networks including: single- and multi-layer ANNs, activation functions and their derivatives, a high-level description of the backpropagation algorithm, and a number of classification and regression examples. ANNs, particularly mult-layer ANNs, are a robust and powerful class of models that can be used to learn complex, nonlinear functions. However, there are a number of considerations when using neural networks including:
- How many hidden layers should one use?
- How many hidden units in each layer?
- How do these relate to overfitting and generalization?
- Are there better error functions than the squared difference?
- What should the learning rate be?
- What can we do about the complexity of error surface with deep networks?
- Should we use simulated annealing?
- What about other activation functions?
It turns out that there are no easy or definite answers to any of these questions, and there is active research focusing on each topic. This is why using ANNs is often considered as much as a "black art" as it is a quantitative technique.
One primary limitation of ANNs is that they are supervised algorithms, requiring a target value for each input observation in order to train the network. This can be prohibitive for training large networks that may require lots of training data to adequately adjust the parameters. However, there are a set of unsupervised variants of ANNs that can be used to learn an initial condition for the ANN (rather than from randomly-generated initial weights) without the need of target values. This technique of "unsupervised pretraining" has been an important component of many "deep learning" models used in AI and machine learning. In future posts, I look forward to covering two of these unsupervised neural networks: autoencoders and restricted Boltzmann machines.
Derivation: Derivatives for Common Neural Network Activation Functions
When constructing Artificial Neural Network (ANN) models, one of the primary considerations is choosing activation functions for hidden and output layers that are differentiable. This is because calculating the backpropagated error signal that is used to determine ANN parameter updates requires the gradient of the activation function gradient . Three of the most commonly-used activation functions used in ANNs are the identity function, the logistic sigmoid function, and the hyperbolic tangent function. Examples of these functions and their associated gradients (derivatives in 1D) are plotted in Figure 1.
In the remainder of this post, we derive the derivatives/gradients for each of these common activation functions.
The Identity Activation Function
The simplest activation function, one that is commonly used for the output layer activation function in regression problems, is the identity/linear activation function:
(Figure 1, red curves). This activation function simply maps the pre-activation to itself and can output values that range . Why would one want to do use an identity activation function? After all, a multi-layered network with linear activations at each layer can be equally-formulated as a single-layered linear network. It turns out that the identity activation function is surprisingly useful. For example, a multi-layer network that has nonlinear activation functions amongst the hidden units and an output layer that uses the identity activation function implements a powerful form of nonlinear regression. Specifically, the network can predict continuous target values using a linear combination of signals that arise from one or more layers of nonlinear transformations of the input.
The derivative of , , is simply 1, in the case of 1D inputs. For vector inputs of length the gradient is , a vector of ones of length .
The Logistic Sigmoid Activation Function
Another function that is often used as the output activation function for binary classification problems (i.e. outputs values that range ), is the logistic sigmoid. The logistic sigmoid has the following form:
(Figure 1, blue curves) and outputs values that range . The logistic sigmoid is motivated somewhat by biological neurons and can be interpreted as the probability of an artificial neuron "firing" given its inputs. (It turns out that the logistic sigmoid can also be derived as the maximum likelihood solution to for logistic regression in statistics). Calculating the derivative of the logistic sigmoid function makes use of the quotient rule and a clever trick that both adds and subtracts a one from the numerator:
Here we see that evaluated at is simply weighted by 1-minus-. This turns out to be a convenient form for efficiently calculating gradients used in neural networks: if one keeps in memory the feed-forward activations of the logistic function for a given layer, the gradients for that layer can be evaluated using simple multiplication and subtraction rather than performing any re-evaluating the sigmoid function, which requires extra exponentiation.
The Hyperbolic Tangent Activation Function
Though the logistic sigmoid has a nice biological interpretation, it turns out that the logistic sigmoid can cause a neural network to get "stuck" during training. This is due in part to the fact that if a strongly-negative input is provided to the logistic sigmoid, it outputs values very near zero. Since neural networks use the feed-forward activations to calculate parameter gradients (again, see this previous post for details), this can result in model parameters that are updated less regularly than we would like, and are thus "stuck" in their current state.
An alternative to the logistic sigmoid is the hyperbolic tangent, or tanh function (Figure 1, green curves):
Like the logistic sigmoid, the tanh function is also sigmoidal ("s"-shaped), but instead outputs values that range . Thus strongly negative inputs to the tanh will map to negative outputs. Additionally, only zero-valued inputs are mapped to near-zero outputs. These properties make the network less likely to get "stuck" during training. Calculating the gradient for the tanh function also uses the quotient rule:
Similar to the derivative for the logistic sigmoid, the derivative of is a function of feed-forward activation evaluated at , namely . Thus the same caching trick can be used for layers that implement tanh activation functions.
Wrapping Up
In this post we reviewed a few commonly-used activation functions in neural network literature and their derivative calculations. These activation functions are motivated by biology and/or provide some handy implementation tricks like calculating derivatives using cached feed-forward activation values. Note that there are also many other options for activation functions not covered here: e.g. rectification, soft rectification, polynomial kernels, etc. Indeed, finding and evaluating novel activation functions is an active subfield of machine learning research. However, the three basic activations covered here can be used to solve a majority of the machine learning problems one will likely face.
Derivation: Error Backpropagation & Gradient Descent for Neural Networks
Artificial neural networks (ANNs) are a powerful class of models used for nonlinear regression and classification tasks that are motivated by biological neural computation. The general idea behind ANNs is pretty straightforward: map some input onto a desired target value using a distributed cascade of nonlinear transformations (see Figure 1). However, for many, myself included, the learning algorithm used to train ANNs can be difficult to get your head around at first. In this post I give a step-by-step walk-through of the derivation of gradient descent learning algorithm commonly used to train ANNs (aka the backpropagation algorithm) and try to provide some high-level insights into the computations being performed during learning.
Some Background and Notation
An ANN consists of an input layer, an output layer, and any number (including zero) of hidden layers situated between the input and output layers. Figure 1 diagrams an ANN with a single hidden layer. The feed-forward computations performed by the ANN are as follows: The signals from the input layer are multiplied by a set of fully-connected weights connecting the input layer to the hidden layer. These weighted signals are then summed and combined with a bias (not displayed in the graphical model in Figure 1). This calculation forms the pre-activation signal for the hidden layer. The pre-activation signal is then transformed by the hidden layer activation function to form the feed-forward activation signals leaving leaving the hidden layer . In a similar fashion, the hidden layer activation signals are multiplied by the weights connecting the hidden layer to the output layer , a bias is added, and the resulting signal is transformed by the output activation function to form the network output . The output is then compared to a desired target and the error between the two is calculated.
Training a neural network involves determining the set of parameters that minimize the errors that the network makes. Often the choice for the error function is the sum of the squared difference between the target values and the network output (for more detail on this choice of error function see):
Equation (1)
This problem can be solved using gradient descent, which requires determining for all in the model. Note that, in general, there are two sets of parameters: those parameters that are associated with the output layer (i.e. ), and thus directly affect the network output error; and the remaining parameters that are associated with the hidden layer(s), and thus affect the output error indirectly.
Before we begin, let's define the notation that will be used in remainder of the derivation. Please refer to Figure 1 for any clarification.
- : input to node for layer
- : activation function for node in layer (applied to )
- : ouput/activation of node in layer
- : weights connecting node in layer to node in layer
- : bias for unit in layer
- : target value for node in the output layer
Gradients for Output Layer Weights
Output layer connection weights,
Since the output layer parameters directly affect the value of the error function, determining the gradients for those parameters is fairly straight-forward:
Equation (2)
Here, we've used the Chain Rule. (Also notice that the summation disappears in the derivative. This is because when we take the partial derivative with respect to the -th dimension/node, the only term that survives in the error gradient is -th, and thus we can ignore the remaining terms in the summation). The derivative with respect to is zero because it does not depend on . Also, we note that . Thus
Equation (3)
where, again we use the Chain Rule. Now, recall that and thus , giving:
Equation (4)
The gradient of the error function with respect to the output layer weights is a product of three terms. The first term is the difference between the network output and the target value . The second term is the derivative of output layer activation function. And the third term is the activation output of node j in the hidden layer.
If we define to be all the terms that involve index k:
we obtain the following expression for the derivative of the error with respect to the output weights :
Equation (5)
Here the terms can be interpreted as the network output error after being back-propagated through the output activation function, thus creating an error "signal". Loosely speaking, Equation (5) can be interpreted as determining how much each contributes to the error signal by weighting the error signal by the magnitude of the output activation from the previous (hidden) layer associated with each weight (see Figure 1). The gradients with respect to each parameter are thus considered to be the "contribution" of the parameter to the error signal and should be negated during learning. Thus the output weights are updated as , where is some step size ("learning rate") along the negative gradient.
As we'll see shortly, the process of backpropagating the error signal can iterate all the way back to the input layer by successively projecting back through , then through the activation function for the hidden layer via to give the error signal , and so on. This backpropagation concept is central to training neural networks with more than one layer.
Output layer biases,
As far as the gradient with respect to the output layer biases, we follow the same routine as above for . However, the third term in Equation (3) is , giving the following gradient for the output biases:
Equation (6)
Thus the gradient for the biases is simply the back-propagated error from the output units. One interpretation of this is that the biases are weights on activations that are always equal to one, regardless of the feed-forward signal. Thus the bias gradients aren't affected by the feed-forward signal, only by the error.
Gradients for Hidden Layer Weights
Due to the indirect affect of the hidden layer on the output error, calculating the gradients for the hidden layer weights is somewhat more involved. However, the process starts just the same:
Notice here that the sum does not disappear because, due to the fact that the layers are fully connected, each of the hidden unit outputs affects the state of each output unit. Continuing on, noting that …
Equation (7)
Here, again we use the Chain Rule. Ok, now here's where things get "slightly more involved". Notice that the partial derivative in the third term in Equation (7) is with respect to , but the target is a function of index . How the heck do we deal with that!? Well, if we expand , we find that it is composed of other sub functions (also see Figure 1):
Equation (8)
From the last term in Equation (8) we see that is indirectly dependent on . Equation (8) also suggests that we can use the Chain Rule to calculate . This is probably the trickiest part of the derivation, and goes like…
Equation (9)
Now, plugging Equation (9) into in Equation (7) gives the following for :
Equation (10)
Notice that the gradient for the hidden layer weights has a similar form to that of the gradient for the output layer weights. Namely the gradient is some term weighted by the output activations from the layer below (). For the output weight gradients, the term that was weighted by was the back-propagated error signal (i.e. Equation (5)). Here, the weighted term includes , but the error signal is further projected onto and then weighted by the derivative of hidden layer activation function . Thus, the gradient for the hidden layer weights is simply the output error signal backpropagated to the hidden layer, then weighted by the input to the hidden layer. To make this idea more explicit, we can define the resulting error signal backpropagated to layer as , and includes all terms in Equation (10) that involve index . This definition results in the following gradient for the hidden unit weights:
Equation (11)
This suggests that in order to calculate the weight gradients at any layer in an arbitrarily-deep neural network, we simply need to calculate the backpropagated error signal that reaches that layer and weight it by the feed-forward signal feeding into that layer! Analogously, the gradient for the hidden layer weights can be interpreted as a proxy for the "contribution" of the weights to the output error signal, which can only be observed–from the point of view of the weights–by backpropagating the error signal to the hidden layer.
Output layer biases,
Calculating the gradients for the hidden layer biases follows a very similar procedure to that for the hidden layer weights where, as in Equation (9), we use the Chain Rule to calculate . However, unlike Equation (9) the third term that results for the biases is slightly different:
Equation (12)
In a similar fashion to calculation of the bias gradients for the output layer, the gradients for the hidden layer biases are simply the backpropagated error signal reaching that layer. This suggests that we can also calculate the bias gradients at any layer in an arbitrarily-deep network by simply calculating the backpropagated error signal reaching that layer !
Wrapping up
In this post we went over some of the formal details of the backpropagation learning algorithm. The math covered in this post allows us to train arbitrarily deep neural networks by re-applying the same basic computations. Those computations are:
- Calculated the feed-forward signals from the input to the output.
- Calculate output error based on the predictions and the target
- Backpropagate the error signals by weighting it by the weights in previous layers and the gradients of the associated activation functions
- Calculating the gradients for the parameters based on the backpropagated error signal and the feedforward signals from the inputs.
- Update the parameters using the calculated gradients
The only real constraints on model construction is ensuring that the error function and the activation functions are differentiable. For more details on implementing ANNs and seeing them at work, stay tuned for the next post.


text:
If we're not careful, we can come to think of the Christian life as a religion, or as a set of beliefs we agree on, or about what we're against and oppose. Christianity is best expressed as the imitation of Christ or as Jesus put it, "Following" Him into a life of complete submission to God lived for the benefit of others.
Following Jesus is always about right now. It's not about what you did a few years ago. It's not about what you're planning on doing some day. It's about whether or not you are following Jesus right now, in this very moment.
As the scriptures remind us:
"God again set a certain day, calling it 'Today'" (Hebrews 4:7)
"Today, if you hear his voice, do not harden your hearts as you did in the rebellion" (Hebrews 3:15)
"…then chose for yourselves this day whom you will serve…As for me and my house we will serve the Lord." (Joshua 24:14-16)
"Whoever wants to be my disciple must deny themselves and take up their cross daily and follow me." (Luke 9:23)
Like Paul wrote to us, we must "forget the past" (Philippians 3:13) and press onward to our calling to follow Jesus daily.
It's why Jesus asks us to pray for "daily bread" (Matt 6:11) and to receive God's rich Mercies that are "new every morning." (Lamentations 3:23)
We need to think of our walk with Jesus as a moment-by-moment engagement with Jesus where we are trusting Him with every heartbeat and in time with every breath we breathe.
Are you trusting Jesus as your Lord and Savior at this very moment? Is His Kingdom brand new in your heart? Have you devoted yourself to His service 24/7?
My prayer is that I could always experience this sort of constant conversion where my commitment to Jesus is as new as my latest thought and as immediate as this very moment.
Conversatio morem!

subset: commoncrawl


text:
Let's have some phun! For all tennis players in the Phoenix area, regardless of ability. We can match up based on skills and location.
Join in here to meet other players in the Phoenix / Scottsdale area. To find partners, introduce yourself below.
Location: Phoenix, AZ
Members: 283
Latest Activity: Jul 18, 2018
Start a New Discussion
Want more organized tennis? Try a flex league where you are matched with players of your level in your town. We've partnered with TennisPhoenix.com and all Tennisopolis members get 10% off your first league! Click here to join.
Hi. Mark, the Mayor of Tennisopolis here. This group has grown quite large and is very active! Congratulations. We could use another Administrator for this group. Would anyone like to volunteer? Just send me a note if so.
Started by Steve from TLN. Last reply by Mark / The Mayor Apr 25, 2016. 1 Reply 0 Likes
Started by Mark Willis. Last reply by Mark Willis Aug 13, 2015. 9 Replies 0 Likes
Started by Jordan Collins. Last reply by Sonu Sud Sep 12, 2014. 7 Replies 0 Likes
Started by Michael. Last reply by Rylan Mills Jan 12, 2014. 1 Reply 0 Likes
Started by Ron. Last reply by Grantman Dec 18, 2013. 1 Reply 1 Like
Started by Juan Pablo Gonzales. Last reply by Michael Nov 6, 2013. 1 Reply 0 Likes
© 2020 Created by Mark / The Mayor. Powered by


text:
Amarilla and J.G. gave lots 21 and 22 in Block 12 to the Methodist Episcopal Church of Pine River for $1.00 in September of 1902. Amarilla was to bequeath a lot more to the town of Pine River.
Mrs. Ammarrila Dawes & husband, to The Methodist Episcopal Church of Pine River, filed 10 October 1902 at 9 am. This indenture made this 11th day of September, 1902 between Mrs. Ammarilla Dawes & J.G. Dawes of the County of Cass and State of Minnesota parties of the first part and The Methodist Episcopal Church of Pine River of the County of Cass and State of Minnesota party of the 2nd part Witnesseth that the said parties of the first part in consideration of the sume of one dollar to them in hand paid by the parties of the 2nd part….Lots numbered twenty one (21) and twenty two (22) in Block twelve (12) in the Village of Pine River, according to the original plat on file with the Register of Deeds in and for said county and state….signed by Ammarilla Dawes and J.G. Dawes and witnessed by Hans P. Hanson and Wyman H. Davis the notary public witnessed their free act and deed.
Source: Cass County Register of Deeds, Cass County Courthouse, Walker, Minnesota, Book W, pg. 407, 11 September, 1901.
Apparently, Amarilla and J.G., were involved in the founding of this church for there appears an Article of Incorporation which was filed for record September 23, 1902. Although they are not listed as trustees it is evident that they assisted with the process by the giving of the lots of land to the church.
Know all men by these presents, that we, Robert Forbes, the presiding officer, and Samuel J. Parish, the Secretary, of a Quarterly Conference of the Methodist Episcopal Church, which was duly called, held and conducted on Monday September 1, 1902, in the school house of Pine River, County of Cass, State of Minnesota, according to the discipline and usage of the Methodist Episcopal Church and statues in such case and provided, do hereby certify that at the said Quarterly Conference the following named persons to wit: Stephen Jarvis, Frank L. White, Cramer H. Mitchell, Manly T. White and Heber S. Gilbert were duly elected trustees of The Methodist Episcopal Church of Pine River in the County of Cass, State of Minnesota, a religious Corporation said corporation to be known as The Methodist Episcopal Church of Pine River which corporate name was then and there assumed by said Trustees. Given under our hands and seals this 1st day of September, 1902. In the Presence of George O. Parrish and S.P. Hanson. Signed by Robert Forbes, President and Samuel L. Parish, Secretary. Signed by S. P. Hansen, Justice of the Peace Cass County, MN
The above was found in the clerk books after the deed for $1.00 mentioned above.
The instrument continues with an affidavit of the two men doing this of their own free act and deed followed with an affidavit certifying that Mrs. Amarilla Dawes was the same person as Mrs. G.A. Barclay, dated the 8th day of September 1902.


text:
Julianne Hough, Carrie Underwood, and Alanis Morissette To Perform On Dancing With The Stars Semi-Finals
Get Ready!! Not only is Alanis Morisssette to perform next week, but, Grammy winning Carrie Underwood is also performing. Adding to the fun (and many of you will be so happy to hear this news like I am!) JULIANNE HOUGH is also going to perform. "Macy's Stars of Dance" will also present this season's "Design-a-Dance". More fun highlights below and at ABC MediaNet.
"Episode 1409 & 1409A" – After last week's nerve-racking double elimination, the four remaining couples face off in the semifinals of "Dancing with the Stars," MONDAY, MAY 14 (8:00-9:31 p.m., ET), on the ABC Television Network. The couples will perform one new Ballroom and one new Latin dance to determine which three couples will land in the finals.
On TUESDAY, MAY 15 (9:00-10:01 p.m., ET), special musical guests Carrie Underwood and Alanis Morissette will hit the stage with unforgettable performances on "Dancing with the Stars the Results Show." Award winning Country superstar Carrie Underwood is set to perform "Good Girl," the first single off her brand new album, "Blown Away." Her exciting performance will be accompanied by Tristan Macmanus, Sharna Burgess, Henry Byalikov and Emma Slater. Multi-platinum, Grammy ® Award-winning artist Alanis Morissette will hit the stage with a world premiere performance of her new single, "Guardian," from her upcoming album, "Havoc and Bright Lights." Her performance will be accompanied by Blake McGrath and Tyne Stecklein.
During Tuesday night's results show we will also have an incredible performance from the upcoming Warner Bros Pictures feature release, "Rock of Ages." The electrifying performance will feature two-time former "Dancing with the Stars" champion Julianne Hough and her co-star, actor/singer Diego Boneta. Directed by Adam Shankman ("Hairspray"), New Line Cinema's adaptation of the smash hit Broadway musical "Rock of Ages" hit theaters June 15th. The movie musical stars Julianne Hough, Diego Boneta, Russell Brand, Paul Giamatti, Catherine Zeta-Jones, Malin Akerman, Mary J. Blige, Bryan Cranston, with Alec Baldwin and Tom Cruise.
In the final "Macy's Stars of Dance" performance this season, two yet to be named "DWTS" pros will perform a riveting Samba to "End of Time." All season long, fans have voted on the performance song, style, costumes and pro dancers for the season's Macy's Design-a-Dance.
Also and in case you haven't heard, Dancing With The Stars has been renewed for another season. YAY!!


text:
Free chat with TSWildnessFree chat with TSWildness
Hello my dearest viewers! I'm a horny little POST OPERATED TRANSSEXUAL beauty for your pleasure ;) If you are ready to try new things, you are on the best place ever! Come to my room, visit my private show, just try me ;)


text:
Canton Community Clinic Inc. has received a grant to bring health care to under-served patients in Southeast Canton.
Canton Community Clinic has received a grant to bring health care to under-served patients.
The clinic snagged a Health Center New Access Point grant from the federal government to turn its Ninth Street Family Health Center into a federally-qualified health clinic. The clinic received $108,000 of its requested $650,000 on Nov. 1, and should be awarded the remainder in January, said Eric Riley, clinic CEO.
The grant will allow the Ninth Street center, which offers medical and behavioral services, to add a full-time family physician and bump other part-time employees to full-time status, Riley said.
The center opened at 408 Ninth St. SW in February. It was operating as essentially a private medical practice and funded through the small amount of fees it collected from patients, he said.
The center is not a free clinic but doesn't turn anyone away based on their inability to pay. They accept Medicare, Medicaid and private insurance, and patients without coverage pay for services on sliding scale.
Without the center, those patients would be going to emergency room for medical care, Riley said.
Aultman Hospital cooperated with the clinic in putting in the Ninth Street center, which will hopefully divert some ER traffic to the center, he said.
There's a strong need for a health center in the area, Riley said.
Canton Community Clinic's main location, a federally-qualified health clinic at 2725 Lincoln St. E in Canton Township, can be hard to get to for some, he said. Now those who live in the southeast Canton, where the Ninth Street center is located, won't have to navigate bus changes to get health care.
The Ninth Street center, right on a popular SARTA route, is easily-accessible, Riley said.
The Department of Health and Human Services' Health Resources and Services Administration — often shortened to HRSA — considers Access Point grant applications based on the need for services in the area and the number of potential patients, among other criteria, Riley said.
The clinic's application scored 99 out of a possible 100 points, he said.
HRSA awarded 11 grants to organizations throughout Ohio and 236 nationwide. The grants are currently funded through the Affordable Care Act.
"They just don't hand these grants out," Riley said, adding that some states didn't receive any new grants.
The center has seen its patient numbers double each month just through word-of-mouth, Riley said.
Between the two locations, the clinics see about 1,600 to 1,700 patients monthly, he said.
Both clinics offer mental and behavioral health services for patients of all ages, and the main clinic also offers dental care. Now, patients at the Ninth Street center can be referred to a dentist at the main branch.
The center is accepting new patients and has the space to grow. With the grant, it also the funding to do so.
"Right now, we're the best kept secret in Stark County," said Joan Glew, a registered nurse and the center's coordinator. "We'd like to let everyone know we're here."
Reach Jessica at 330-580-8322.
On Twitter: @jholbrookREP


text:
Teacher Development (Professional Graduate Certificate in Education)
A programme designed for teachers working in schools around the world. Become a better school teacher and gain a valuable professional qualification.
Search for a course, locations you can study, or application information
Use the tabs below to see the different types of courses


text:
Eating Disorder Counselor
Rebekah Miller, MA, LMFT
Licensed Marriage & Family Therapist at Wellness Counseling Services (WCS)
In 55427 - Nearby to Chaska.
Do you struggle with low self-esteem or a belief that you can never be thin enough? Do you often feel numb or out of control? Are your loved ones worried about how little you eat? Do you sometimes use laxatives, excessive exercise or other things to purge food out of your body? Together, we will explore the causes and triggers of your eating disorder while working toward greater health for your mind and body. Your family may be coping with fears and a sense of helplessness. While working with you to address your specific concerns, I often work with families to help them manage their fears and increase their ability to support you as you work to reach your goals.
Eating Disorder Counselor
Renee Segal, MA, LMFT
Licensed Marriage, Family and Individual Therapist
In 55391 - Nearby to Chaska.
Do you overeat? Do you undereat? Do you sometimes do both? When your relationship with food is all that you think about it may be time for some therapy. Some people even avoid sex and intimacy because they don't feel good about their body image. Learn what foods trigger your eating and avoidance patterns. Learn healthy coping skills when youa re with other people and when food is offered to you.
Eating Disorder Counselor
Mary Hendrickson, Psy.D., LP
Licensed Doctoral Level Clinical Psychologist
In 55344 - Nearby to Chaska.
If you struggle with food, this can control your life. Therapy is a supportive process that can help you understand your relationship with food and it's meaning in your life. An eating disorder can be a shameful process and very complicated. Trying to manage this on your own is a difficult struggle. While working with you in a therapeutic relationship, you can begin to understand more about yourself, gain insight and learn about self care from the inside out. You may eat to little, too much or have difficulty keeping food in. Seeking help is vital to your healing!
Eating Disorder Counselor
Bruce Fischer, PhD, LP, LMFT
Licensed Psychologist, Licensed Marriage & Family Therapist
In 55408 - Nearby to Chaska.
Eating disorders are complex and vary widely from not eating to binge eating and also include compulsive eating. I will work with you to understsand the dynamics behind the disorder and also develop a practical plan to address how to change the behavioral aspects of the problem. Sometimes couple or family therapy can be included in the treatment plan. The goal is to help you effectively manage your eating to achieve optimal physical and mental health.
Eating Disorder Counselor
Joan Pechauer, LMFT
Marriage and Family Therapist
In 55369 - Nearby to Chaska.
If you've struggled with an eating disorder or wonder if you do, let's talk about what you're next steps are. Maybe you need to understand what's going on in your life or help repairing trust with loved ones or you need to find ways to think differently about your relationship to food. Wherever you're at, we'll determine ways in which you can feel strong, supported and hopeful.
You Have Found Honest and Real Therapy for Food Issues in Chaska, MN.
Are you ready to start on a new path? Thank you for visiting our Minnesota search of licensed therapists in Chaska who specialize and have experience treating food issues such as anorexia, over eating, food aversions, orthorexia nervosa and bulimia. Food issues affect anyone, both children and adults and can impede our relationships and daily life. Counseling for help with food and eating is a vital component of healthy recovery and restoration. Find the right counselor for eating and food issues in Chaska right now and begin your new life.
Chaska is located in Carver County, Minnesota. It has a land area of 17.00 square miles and a water area of 0.83 square miles.  The population of Chaska is 25,199 people with 9,108 households and a median annual income of $75,828. .
Therapy Affordability Meter for Chaska, MN
Mostly Affordable
Establishing weekly sessions with a professional therapist is considered affordable in Chaska for the average family. While sustaining a commitment to therapy is usually not an issue, if mental health treatment feels like a burden please ask your counselor about accepted insurance plans or sliding scale fees. Low cost counseling and affordable therapy are also sometimes offered by listed city and university level clinics, check with your local Chaska public health department.


text:
Go Green!
It's September 1st, which means rosters around baseball expand to 40 players, and oh yeah, college football season!
Read it and weep Wolverine fans:
Michigan State: 1-0
U. of Michigan: 0-1
Oh, the sweet taste of victory for the Spartans and absolute bitter, horrifying, embaressing defeat for Michigan. My Spartans did what they were supposed to do — crush UAB in a 55-18 joke of a game in East Lansing. Meanwhile, in little ol' Ann Arbor, the No. 5 Wolverines quivered to a 34-32 mindnumbing loss to … Appalachian State?
Blue Jays third base coach Brian Butterfield is a HUGE Michigan fan, and in the wake of having a missed field goal with six seconds left cost his team an opening-week win, well, needless to say he wasn't in a good mood. I peaked my head into the coaches' office after the game and said:
"MSU won 55-18. Now there's a school who knows how to finish off a team it should beat."
Butter responded with, "You better watch it. You're in the coaches' room. There's no telling what I might do."
All in good fun, sure, but just in case, I said, "Tough loss, coach," and made my way quickly out the clubhouse doors. And it was a tough loss. So, my heartfelt sympathies go out to all you Michigan fans, whose national title hopes have been crushed swiftly. Ah, let's be honest, I'm giddy that the ol' Wolverines blew it big today.
And I know, I know — I'm talking up Michigan State? I'm well aware that my college ain't exactly a top-tier program. In fact, I'm the first to admit that I'm gearing up for college basketball season by the time the Spartans are about three games into the Big Ten Schedule. Undoubtedly, UM will likely romp all over the Green Shirts later this year.
So, I'll rub it in whenever I get the chance!
Congratulations Appalachian State. The Big House was yours for one day.
7 Comments
Michigan became the first team to lose to a Division 1AA team. How PATHETIC!!! Lloyd Carr was on the hotseat when the season started now the man can't even sit down.
Les Miles ( Michigan alum ) will be roaming the sidelines in Ann Arbor next year. Lloyd Carr is toast at Michigan.
Also I believe Michigan could be the first team ever to start the year in the top 10 and then lose there first game and they will not be ranked. Polls come out tomm so I guess we will find out.
Well, I think we know what your passion is.
It's always amazed me, how loyal the alumini fans are in US college sports.
Wish we had that here.
Ty Taubenheim is trying to send The Blue Jays a message.
He's had 5 startsS since he was demoted to AA in New Hampshire,and compiled an era of 2.01 with 29 k's in 31.1 innings pitched.
Maybe he recognized he's running out of time to make the bigs since he's already 25.
One way or the other it looks like the wake up call worked.
Brandon League pitched a clean inning last night in Dunedin, with 32 k's, no hits and 1 BB. Loosl ike he's getting closer.
The Auburn Doubledays clinched their sixth straight New York-Penn League Pinckney Division title Saturday with a 7-2 victory over the Jamestown Jammers as Victor Santana and Benjamin Zeskind homered.
If you'd like to read the entire article from MLB news:
As I have posted numerous times, we have some great strength in the lower minors.
Standings-soon to be final are:
Gulf Coast Blue Jays-2nd in division.
Auburn DD's-1st in division
Lansing-2nd in division
Dunedin-2nd in division-but only 2 back.
We've already had one player-Josh Kreuzer named Player of the year in the Florida State League (A+). Travis Snider is likely the front liner to win a similar award in the Midwest League(A); and James Dougher and Brian Letko-who could be viewed as the 1+2 pitchers in the Gulf Coast rookie league-not to mention the balance of the excellent young pitching the Jays have in the lower minors.
The point is, The Jays have drafted exceptionally well in the last couple of years and the proof is in the results.
For the ones that do not think JP has done a good job-think again.
Now, if we could get a manager at the major league level that knows how to manage this talent or at least fill out the lineup card properly, we could be competitive for a number of years.
GSUMNER: Well, my passion is definitely baseball. But you're right, us U.S. Americans (youtube Miss Teen South Carolina if you don't get the reference there) do hold strong and true to our college teams. I whistled the MSU fight song every time Butterfield walked by me this morning 🙂
well, I hope you sent Butter to utube to lighten the blow and I thank you for explaining the passion. Lol


text:
The American National Standards Institute's (ANSI) Nanotechnology Standards Panel (NSP) was scheduled to meet in Oct. 2012 but Hurricane Sandy, which hit the eastern part of the continent at that time, necessitated rescheduling to Feb. 4, 2013 as per the Dec. 20, 2012 posting on Thomas.net,
Originally scheduled for October 30, 2012, ANSI's Nanotechnology Standards Panel meeting was postponed as a result of Hurricane Sandy and will now be held on February 4, 2013. Meeting will examine how current nanotechnology standards are being utilized and how standards activities meet existing stakeholder needs. Benefits of participating in nanotechnology standardization and the possibilities for greater collaboration between stakeholders in this area will also be discussed.
The Dec. 14, 2012 ANSI news release provides more details about the Feb. 4, 2012 meeting to be held in Washington, DC,
The half-day meeting will examine how current nanotechnology standards are being utilized and how standards activities meet existing stakeholder needs. The benefits for companies, organizations, and other groups to participate in nanotechnology standardization and the possibilities for greater collaboration between stakeholders in this area will also be discussed.
Formed in 2004, ANSI's NSP serves as the cross-sector coordinating body for the facilitation of standards development in the area of nanotechnology. Shaun Clancy, Ph.D., the director of product regulatory services for the Evonik Degussa Corporation, and Ajit Jilavenkatesa, Ph.D., the senior standards policy advisor for the National Institute of Science and Technology (NIST) of the U.S. Department of Commerce (DoC), serve as the ANSI-NSP's co-chairs.
… The ANSI-NSP works to provide a forum for standards developing organizations (SDOs), government entities, academia, and industry to identify needs and establish recommendations for the creation or updating of standards related to nanotechnology and nanomaterials. In addition, the ANSI-NSP solicits participation from nanotechnology-related groups that have not traditionally been involved in the voluntary consensus standards system, while also promoting cross-sector collaborative efforts.
Attendance at the February meeting is free. All attendees are required to register here for the meeting; individuals who registered for the October 2012 event must register again. [emphasis mine] For more information, visit the ANSI-NSP webpage or contact Heather Benko (email@example.com), ANSI senior manager, nanotechnology standardization activities.
Standardization is one of the topics highlighted in Michael Berger's Dec. 20, 2012 Nanowerk Spotlight article about environmental health and safety and a high-throughput screening (HTS) platform developed at the University of California's Center for Environmental Implications of Nanotechnology (CEIN) that can perform toxicity screening of 24 metal oxide nanoparticles simultaneously,
According to the team, the HTS platform that has been demonstrated in this study could easily be adapted to study other nanomaterials of interest. The capability of HTS would also allow researchers to analyze multiple samples at different concentrations, time points, as well as varying experimental parameters – all in one setup. The standardization of the whole screening process by this HTS platform also minimizes human intervention and errors during the experiment.
I guess it's the season for standardization. Ho, ho, ho!


text:
"Kind, nurturing, dependable, healthy 😊 full of energy, grandma here. I have my own car, not taking public transportation. No cough or fever, no contact with anyone who has had symptoms or been diag..." Sign Up to See Full Profile
"Hi, I am Jeanette. I attended West Texas A&M. I was on the West Valley Women's Soccer and Swim team. I love the outdoors and being active. At the age of fourteen, I started working with kids at Pinehu..." Sign Up to See Full Profile
"I'm a TrustLine Certified, INA member nanny working full-time for a Napa Valley winery family. Please see my video for the address of my Online Resume for my complete childcare work history. I'm happ..." Sign Up to See Full Profile
"Hello, my name is Antonella. I am currently pursuing my Masters degree. Extras: CPR certified, and fluent Spanish speaker " Sign Up to See Full Profile
"I have 9 years of experience babysitting and working with children from ages 4-11. I am loving, patient, caring, and very energetic! I love reading, making art, and dancing. I am fluent in English and..." Sign Up to See Full Profile
"My name is Roberta Alves .I'm 39 years old ! I'm a mom of two wonderful boys and a wife ! I genuinely love kids ,it's a passion of mine .I'm very patient,caring and responsible,sports lover ,fun..I'm ..." Sign Up to See Full Profile
"Hi my name is Juana, I've been working with kids for 20 years. I enjoy seeing children development physically and emotionally. I love to take kids to walks or to the park. I like cooking and I don'..." Sign Up to See Full Profile
"Hello! My name is Kayla. I am 22 years old and am studying to become a kindergarten teacher. I love hiking, playing games, cooking, and doing any sorts of crafts. Previous jobs have included be..." Sign Up to See Full Profile
"Hi! My name is Katie and I am currently a nursing student at Dominican University in San Rafael. I live in my hometown of Petaluma ( I transferred from Chico State, sorry for the outdated video!) I ha..." Sign Up to See Full Profile
"Hello Parents! My name is Ashley. I am a newly registered nurse. I graduated from San Francisco State University in December 2015 with a BA in Child Development (Early Childhood) with an emphasis in ..." Sign Up to See Full Profile
Looking to hire a babysitter, full-time nanny, date night babysitter, or last minute child care? Look no further than UrbanSitter, the most powerful of all babysitting websites, where you'll find the best babysitters trusted by parents like you. Unlike traditional babysitter or nanny agencies, UrbanSitter connects you with baby sitter reviews written by parents you know and trust, and helps you book babysitters and nannies instantly on-demand.
What sets UrbanSitter apart is its amazing local network of babysitter and nanny recommendations from parents you're connected with through school, moms and dads groups or social networks. It's like asking every parent you know who the best babysitters are in your area, but without the legwork. Sign up for free to start your babysitter search today on the site or babysitting app. Whether you're looking for an after-school sitter, part-time childcare, morning nanny or carpool helper, UrbanSitter has you covered.
To find babysitters in your area, start with a babysitter or nanny search by date or post a job to find available babysitters and nannies. Then, use custom filters to find a baby sitter in your neighborhood with your qualifications such as a background check, child and infant CPR certification, babysitter with car, or special needs experience. You can even save your search to be notified whenever new babysitters join and meet your requirements. If you're looking for a last-minute backup babysitter, your best bet is to post a job for the quickest response from babysitters in your area.
Also unlike traditional babysitter agencies, you can really get to know your babysitter before making the hire. Many UrbanSitter babysitters also have education or experience with early childhood development, or have a teaching or nursing degree, which you'll see in their detailed babysitter profiles. Babysitter profile videos, response times, number of repeat families, ratings, qualifications, and UrbanSitter reviews help you decide on the best baby sitter for your family. Why not get the whole family involved and let your kids pick their favorite babysitter based on the profile videos?
Conveniently, all babysitters in the app have an online calendar with their real time availability, so you can book a baby sitter in minutes or with a few taps from our babysitting app. You can even send a booking request to multiple babysitters, and whoever replies first will get the job while the other requests will be canceled. Hiring a babysitter for multiple dates is also quick and easy to do. Sign up for a free UrbanSitter account and hire a babysitter today!


text:
YAGNI is my favourite "agile" principle. Now personally I didn't realise it was agile until I was told, I'd been happily asking "why do we need that?" for years without realising it was agile, I just thought it was standard engineering. I've had some discussions recently, online and offline, where fairly smart people seem to be suffering some sort of collective brain freeze around "flexibility".
The basic argument tends to go with me asking how their solution is simple and them telling me how its flexible and will automatically adapt to change in the future. Now sometimes this requires a big bunch of intelligent coding on the client side (i.e. by someone else) but that's okay because the "service" is flexible. Pointing out that there will be multiple clients for the service so in effect its just creating more work doesn't seem to wash. Then you dig a little deeper and find out that there is a bunch of additional development being planned on the service side as well to deliver all this flexibility... with no requirements today that demand any of it.
The other bit of the argument tends to go around whether there needs to be flexibility in a given area. One conversation recently was around submitting a particular financial report and then accessing the information later. The old batch file approach wasn't cutting it (genuinely wasn't) so they wanted to replace this with a new XML/SOA/WOA/WTFA solution. There were three options.
- Replace the batch with an XML dump (not great as just wrapped the old problem in new technologies
- Create a single record submission approach
- Create a fully flexible invoicing management solution
Then I asked a question
there are 30 fields in the invoice, how long has that been the case
The answer was that the 30 fields, of which only 10 were normally used, had been static for about 20 years.
So what was the point of option 3?
The answer was that the developers wanted to demonstrate being technologically clever rather than business clever.
The Clifton Suspension Bridge was over engineered, but at least Brunel [corrected] still just aimed to build a bridge. Some software developers (I won't use the term engineers) seem to believe that unless the bridge can be used as a rocket ship them it isn't flexible enough. For myself I say it gets from A to B, it does what it needs to and its been there over a hundred years.
Business Effectiveness is the goal, not technological perfection.
Technorati Tags: SOA, Service Architecture


text:
* Omaha-area radio station KTWI 93.3 FM has ditched new country and its twister format in favor of a classic country format. The station, now billed as The Wolf, offering songs by Alabama, Toby Keith and Rosanne Cash. Two weeks prior to the format flip, the station billed itself as the Jingle and played a host of country-leaning Christmas tunes. The 9,600 -watt station is owned by Clear Channel Communications.
* The end of 2011 kept me at the Slowdown, 729 N. 14th St., for three marquee local indie rock shows over the course of three days. Wednesday, Dec. 28 brought Back When's well-attended Champion Hologram release show on Slowdown's big stage. Back When played a spot-on set of towering songs from their new album, smoothly moving from experimental metal to heavy indie rock and post-rock sounds. Openers Conduits and Noah's Ark Was a Spaceship. helped add to the crowd size. Conduits were in dream-mode for the night, unfurling their layered sound on a beautiful morphine drip pace. Noah's Ark ripped through their set, cutting a noisy modern swath through 90s slacker rock touchstones.
* Gus & Call concluded their five night December Slowdown residency Thursday, Dec. 29, playing with Simon Joyner & the Parachutes and The Bruces. The laid-back folk night led Gus & Call to break out the acoustic guitars and a loose-limbed vibe. The multi-faceted band indulged various parts of its sonic whole during the residency, driving home how powerful the band is when all parts meet. Meanwhile, Simon Joyner's current line-up, featuring David Nance on guitar, takes Joyner's songs into a garage-country vein, filled with noisy guitar squalls. Alex McManus played his Bruces set, added by Stay Awake's Steve Micek on drums for most of the set. The rock-solid drumming propelled the Bruces set.
* On Friday, Dec. 30th, Icky Blossoms hosted a night of entertainment that included the strip-teasinging Potboiler Burlesque, drag performers Dusty Bibles and Pope Trojan II and the comedy girl-rap of STDz. Icky Blossoms were the obvious highlight and all the hype about the band assuming the local indie-dance rock pedestal left open by the Faint is perfectly true. However, where the Faint went for synth-drived doom-gloom seriousness, Icky Blossoms go for wild abandon psychedelic pop fun.


text:
Dallas County Sheriff
Kenneth Mitchum Gun Range
1512 E. Langdon Road, Dallas, TX 75241
The Dallas County Sheriff's Department Gun Range was originally located in Grand Prairie, Texas. The range was utilized by numerous area local, state and federal agencies. The range required continual maintenance, was potentially unsafe to the public and considering the size of the department, was inadequate to meet the needs of the sheriff's department and Dallas County.
Any govermental agency is welcome to use our facilties on a space available basis. The range often books 8-10 months out, so advanced planning is necessary.
Dallas Sheriff's Department Mitchum Firearms Training Center Rules and Regulations for Outside Agencies
Q: Can I come out and practice at the range?
A: We welcome any deputy who wants to come out and develop his or her skills. Our SOP requires that a firearms instructor be present when anyone is firing on the line, so you need to call before you come out to make sure one is available to work with you.
Q: If I come out and practice, will the department provide the ammunition? A: That depends on our supply of ammunition, which varies throughout the year.
Q: Can I bring my wife/husband/etc. out to the range? A: If they are just coming to watch you shoot, then generally there is no problem. They will have to watch from our benches. Anything beyond that requires specific approval from the rangemaster, supervision as required by our SOP, and a signed release of liabliity.
Q: Can I bring a machine gun out to shoot?
A: If you lawfully own a Class III weapon, and will provide your own ammunition, then you may be able to bring it out IF the rangemaster approves it, inspects the weapon, and if the range is not otherwise in use. Generally this is a courtesy extended only to our deputies. Otherwise, there is at least one private shooting area that welcomes Class III weapons. Contact the rangemaster for details.
Q: How do I get a new weapon approved to carry on or off duty?
A: Contact the rangemaster and bring the weapon out to the range for inspection.
Q: Do I have to qualify with an off duty or backup gun?
A: No, qualifications are only required for primary weapons carried on duty.
Q: If I have different weapons I want to be able to carry on duty, to I have to qualify with all of them?
A: Yes, if they are going to be carried as a primary weapon.
Q: Does the range teach concealed hangun classes for employees?
A: Not at this time. Range staff can put you in touch with local CHL instructors.
Q: I want to put a new part on my weapon. Can someone there do it for me?
A: It depends on the make and model. Contact range staff for specifics.
Q: What are the range hours?
A: Generally, we are open 6:00 a.m. to 4:30 p.m., Monday through Friday. We may be open at other hours depending on what we have scheduled.
Q: I call out to the range but nobody answers. What's going on?
A: Most of every working day is spent outside of the office. Even when we aren't on the line, we have to maintain the grounds, meet with representatives of other departments, run errands downtown, supervise inmates, attend our own in-service classes, pick up deliveries, and dozens of other things that crop up on a daily basis. The best times to reach us by phone are before 7:30 a.m. and after 4:00 p.m.. Othewise, leave a message or email firstname.lastname@example.com or firstname.lastname@example.com. We try to check emails and messages every morning and afternoon.


text:
- Home
- Departments
- Economic & Business Development
- Business Resources
Business Resources
Broward County
The Broward County Office of Economic and Small Business Development (OESBD) is primarily responsible for enhancing the economy of Broward County by recruiting new industry, supporting existing industries, and creating new value-added employment opportunities for residents.
Business Tax Receipt
Any person wishing to engage in or manage any business, profession or occupation with the City of Miramar, including a home-based business, is required by ordinance to acquire a city Business Tax Receipt.
Hurricane Preparedness
Review resources to help prepare your business for hurricanes.
Regional Chamber of Commerce
The Miramar-Pembroke Pines Regional Chamber of Commerce is a member-owned, member-driven organization uniting the voice of business and advocating for its success. If you are a new business or existing one that has never been a member, the city will provide $150 towards your 1st year membership dues.
Training & Recruiting
Obtain information on how to train and recruit your employees.
International Trade
Access information about international trade.
Loan Resources
Find out where you can get loans for your business.


text:
Need tires? We can help with that as well. We can combine these wheels with a set of tires and ship them to you as a complete set. We will mount and balance every wheel package for free as well as include the lug nuts. All this, and the shipping is FREE to your door within the continental U.S. Give us a call if you have any questions.
If you are looking for a wheel and tire package, we have a great selection of wheels you can choose from. We provide you with best prices on wheel and tire package deals, which are not only shipped free to your door within the U.S., but we also mount and balance every package at no cost to you in order to ensure the best possible ride. We also throw in a set of lug nuts at no cost.
Whether it is wheels, tires or a package deal, let us work on providing you with performance plus value. If you have any questions, give our experts a call and let our knowledge work for you!


text:
Ordinary Time: July 31st
Memorial of St. Ignatius of Loyola, priest
» Enjoy our Liturgical Seasons series of e-books!
Old Calendar: St. Ignatius of Loyola, confessor
In the year 1521 a cannon ball fractured the left leg of Captain Ignatius Loyola, the future founder of the Jesuits. While he was convalescing, Ignatius read about Christ and His saints and thus turned wholly to God. He then undertook to equip himself for Christ's service by acquiring a good classical and theological education. The members of the Society of Jesus became the shock troops of the Church in the battle against the spread of Protestantism in Europe, as well as one of the greatest foreign mission organizations that the world has known. Ignatius died on July 31, 1556.See Catholic Culture's special section on St. Ignatius.
St. Ignatius
Ignatius, by nation a Spaniard, was born of a noble family at Loyola, in Cantabria. At first he attended the court of the Catholic king, and later on embraced a military career. Having been wounded at the siege of Pampeluna, he chanced in his illness to read some pious books, which kindled in his soul a wonderful eagerness to follow in the footsteps of Christ and the saints. He went to Montserrat, and hung up his arms before the altar of the Blessed Virgin; he then watched the whole night in prayer, and thus entered upon his knighthood in the army of Christ. Next he retired to Manresa, dressed as he was in sackcloth, for he had a short time before given his costly garments to a beggar. Here he stayed for a year, and during that time he lived on bread and water, given to him in alms; he fasted every day except Sunday, subdued his flesh with a sharp chain and a hair-shirt, slept on the ground, and scourged himself with iron disciplines. God favored and refreshed him with such wonderful spiritual lights, that afterwards he was wont to say that even if the Sacred Scriptures did not exist, he would be ready to die for the faith, on account of those revelations alone which the Lord had made to him at Manresa. It was at this time that he, a man without education, composed that admirable book of the Spiritual Exercises.
- Learn more about St. Ignatius and the Jesuit Order and/or read this biography by John Farrow, St. Ignatius of Loyola.
- If you have never done so, consider making the Spiritual Exercises. You can find it online here or you may purchase a copy from Catholic First.
- The Jesuits at Georgetown have a collection of St. Ignatius' Letters and Instructions to his fellow Jesuits. Much of his spiritual teaching is found in his letters and is considered an important source of Jesuit spirituality. If you are interested in reading them click here.
- St. Ignatius founded his Society to give the greatest possible service to the Church and to the Pope. In addition to vows of poverty, chastity and obedience, Jesuits take a special vow of loyalty to the Pope. Today would be a good time to say a prayer for Pope Francis.
- In the Spiritual Exercise, St. Ignatius strongly recommends making a daily examination of conscience. If this is not part of your schedule today would be a good time to start.


text: CSS Archive 80 Best Funny CSS Puns By Saijo George Author: Editorial Team | Feb262015 Categories: Freebies, Web Design No comments When you are developing some websites and write too much codes, you gets bored because there is no charm in development languages. But when it comes to Cascading styling Sheet (CSS) you feel something colorful. Because in css you have to define the designs. In this post we have Continue Reading


text:
Amazon's Kindle Fire tablet is taking all the headlines today and with it comes Whispersync for movies and TVs.
It's such a neat feature on the Kindle – the software syncs your progress as you read a book on the Kindle device so that when you access it from your computer, your iPhone, or whatever device you're on next, you're always in the right place.
And now that functionality is coming to movies and television shows that you watch on your Kindle Fire, too.
Amazon CEO Jeff Bezos explains, "Customers love Whispersync -- so we thought you know what people would really like? What if Whispersync also worked with movies and TV shows? Well it does.
"When you get home, switch to your big screen TV. Your movie will be right where you left it.
"You won't have the frustration of needing to find your spot."


text:
Brooklyn College
Also found in: Acronyms, Wikipedia.
Brooklyn College:see New York, City University ofNew York, City University of
(CUNY), at New York City; created in 1961 by combining the city's 17 municipal colleges. It includes Bernard M. Baruch College (1919; specializes in business studies), Brooklyn College (1930), City College (1847; the oldest member college), the
..... Click the link for more information. .