Increasing Linux Open File Limits
Please note that all opinions are that of the author.
Pizza courtesy of Pizza for Ukraine!
Donate Now to Pizza for Ukraine
So Winston found this log message in our logs today:
2017-03-02T18:18:31.561Z 32313 TID-osjfdq3bg ERROR: !!! ERROR HANDLER THREW AN ERROR !!!
2017-03-02T18:18:34.696Z 32313 TID-osjfdq3bg ERROR: Too many open files @ rb_sysopen - /proc/meminfo
We were running a large data processing job using sidekiq for running our queue and this was causing the queue to stop being processed. As the resident *nix nerd, this one was on me. A quick google on increasing open file limits gave me a tutorial from Easy Engine and the information was good but I think it could be documented better so here is the process I used.
Step 1: Diagnosis
Figure out what your limits actually are. Given that these can be set on a per user basis you want to first diagnose them. I’m using the user ubuntu so here’s the command line you need:
sudo su - ubuntu -c ‘ulimit -aHS’ -s ‘/bin/bash’
Here’s the result:
sudo su - ubuntu -c 'ulimit -aHS' -s '/bin/bash'
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 122314
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 122314
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Clearly a limit of 1024 sounds low.
Step 2: Fixing
The configuration file that you need to edit is: /etc/security/limits.conf. Edit this with:
sudo vi /etc/security/limits.conf
Add to the end of this file the following content:
* hard nofile 999999
* soft nofile 999999
root hard nofile 999999
root soft nofile 999999
Save the file and exit.
Step 3: Log Out
For these changes to take effect you need to log out. Do that now with exit.
Step 4: Login and Re-Diagnose
Log back into the server and run the diagnosis step again:
sudo su - ubuntu -c ‘ulimit -aHS’ -s ‘/bin/bash’
This time you should see something like:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 122314
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 999999
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 122314
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Step 5: Ansible
Ideally this should be implemented using Ansible level so that any new boxes have these settings. Unfortunately that is left as an exercise for the reader.