500 Internal Server Error in nodejs/express after 1000+ clients connected

Today, I moved the server to a Rackspace. As usual I tested the everthing before moving to the new server and all worked well. After chaning DNS to point the new server I started to notice server showing Internal Server Error when it hits 1000 + concurrent users. So,  I enabled logs in express to see whats is going on.

// log every request to the logger
app.use(morgan('combined', {stream: log.stream})); 

This showed that client request hits the server, however it has a problem rendering the output page but why after hitting 1000 + users? Then it hit me, it must have ran out of file descriptors.

$ ulimit -n
1024.

The aha moment. It is hitting the ulimit. So I changed the ulimit

$ ulimit -n 2048

All worked fine after that. However ulimit seems to me temporary. To make permenet changes you must change few things.

1) Increase max number of ulimit open file in Linux

sudo nano /etc/sysctl.conf add end of line fs.file-max = 65536

2) sudo nano /etc/security/limits.conf and add below the mentioned

* soft nproc 65535
* hard nproc 65535
* soft nofile 65535
* hard nofile 65535

3) sudo nano /etc/security/limits.d/90-nproc.conf. Change to

*          soft    nproc     65535
root       soft    nproc     unlimited