BeamSearchDecoderWithLM Memory Requirement
If it's the KenLM binary causing the memory issue, then a smaller N-gram can be used. However we don't have much control over the c++ swig decoders so from our side there isn't a sufficient solution to reduce memory. Maybe @vsl9 has some ideas? If the memory issue is from the beam search decoder, then you may try to call the beam search decoder on each sample separately. So divide your batch into smaller batches or even single samples before calling the decoder. Beam search decoding is mostly done on CPU and batch computation does not increase the speed of the decoder, so you can get the same speed with single samples. 
Other thing you can try is to lower the number for num_cpus, not sure how much it helps. > If the memory issue is from the beam search decoder, then you may try to call the beam search decoder on each sample separately. So divide your batch into smaller batches or even single samples before calling the decoder. Beam search decoding is mostly done on CPU and batch computation does not increase the speed of the decoder, so you can get the same speed with single samples.
> Other thing you can try is to lower the number for num_cpus, not sure how much it helps.

Thanks for the suggestions. I am trying to use it on on single instance so my batch size is 1.  Ram usage increases about 2.5 GBs when I am instantiating BeamSearchDecoderWithLM. So this is before even feeding anything to the network. I see. In that case, memory consumption is mainly due to the ngram model. You may follow Som's suggestion as ngram models need significant memory. That is one of the main reasons they are not preferable for devices with limited resources. Using a model with smaller vocabulary size can also help to reduce the size of the ngram model but not sure if it solves your issue completely. Is this the Cuda storage? Or just normal RAM? Thanks! > Is this the Cuda storage? Or just normal RAM? Thanks!

I am only using normal RAM. My system doesn't have GPU. Yes, it is CPU RAM usage for n-gram LM. Scorer object is instantiated during BeamSearchDecoderWithLM construction. And it loads n-gram LM into RAM. Reducing size of LM (number of tokens/words, LM's order) should help.
Hello, I am using an n-gram KenLM (my language model is only 67 MBs) with BeamSearchDecoderWithLM. Scorer class inside BeamSearchDecoderWithLM which is imported from swig_decoders is using 2.3 gb of ram. I was wondering if there is anything I can do to reduce this memory requirement? I am trying to run this on a device with low memory and I am running out of memory during the initialization of BeamSearchDecoderWithLM.

Is there any alternatives to Scorer that I can swap? I'm okay with it performing worse as long as memory requirements are reduced. 

Thanks.