Is it possible to create a neural network which provides a consistent output given that the input can be in different length vectors?

I am currently in a situation where I have sampled a lot of audio files, which are of different length, and have to train a neural network provides me the desired output given a certain input. I am trying to create a regression network that can generate MFCC feature, given samples of an audio file, which are of different length, which makes different numbered input.


Yes this is possible by treating the audio as a sequence into a Recurrent Neural Network (RNN). You can train a RNN against a target that is correct at the end of a sequence, or even to predict another sequence offset from the input.

Do note however that there is a bit to learn about options that go into the construction and training of a RNN, that you will not already have studied whilst looking at simpler layered feed-forward networks. Modern RNNs make use of layer designs which include memory gates - the two most popular architectures are LSTM and GRU, and these add more trainable parameters into each layer as the memory gates need to learn weights in addition to the weights between and within the layer.

RNNs are used extensively to predict from audio sequences that have already been processed in MFCC or similar feature sets, because they can handle sequenced data as input and/or output, and this is a desirable feature when dealing with variable length data such as spoken word, music etc.

Some other things worth noting:

  • RNNs can work well for sequences of data that are variable length, and where there is a well-defined dimension over which the sequences evolve. But they are less well adapted for variable-sized sets of features where there is no clear order or sequence.

  • RNNs can get state-of-the-art results for signal processing, NLP and related tasks, but only when there is a very large amount of training data. Other, simpler, models can work just as well or better if there is less data.

  • For the specific problem of generating MFCCs from raw audio samples: Whilst it should be possible to create a RNN that predicts MFCC features from raw audio, this might take some effort and experimentation to get right, and could take a lot of processing power to make an RNN powerful enough to cope with very long sequences at normal audio sample rates. Whilst creating MFCC from raw audio using the standard approach starting with FFT will be a lot simpler, and is guaranteed to be accurate.

  • $\begingroup$ Oh.. I guess I forgot to mention that the output is a value and not class => I guess RNN would classify it to a class rather than the value? $\endgroup$ – Carlton Banks Oct 20 '16 at 19:24
  • 1
    $\begingroup$ @CarltonBanks: RNN outputs can be trained with same types of targets (using same loss functions) as other NNs. So you can do classification or regression. $\endgroup$ – Neil Slater Oct 20 '16 at 19:39
  • $\begingroup$ For audio this is a great solution, for other people you need to be careful that RNNs assume a form of order in the sequence which makes it less useful for some use cases $\endgroup$ – Jan van der Vegt Mar 10 '17 at 9:38
  • $\begingroup$ @JanvanderVegt: Yes if you have a variable-sized "bag" of feature vectors, then RNN is maybe less useful. I will try to make that clearer $\endgroup$ – Neil Slater Mar 10 '17 at 9:54
  • 1
    $\begingroup$ @echan00: No the input cannot be variable length with a straightforward CNN. You must either pad or crop to a fixed size. It is likely that the paper you linked works on windowed time frames with set size, and that might also be an option for your project - i.e. split your signal up into (maybe overlapping) segments and process separately - this could work just fine for sequence-to-sequence classification or regression, but less well for sequence-to-single value. You can also get various CNN/RNN hybrids. $\endgroup$ – Neil Slater Mar 10 '17 at 15:30

Your Answer

