500 Internal Server Error in nodejs/express after 1000+ clients connected

Today, I moved the server to a Rackspace. As usual I tested the everthing before moving to the new server and all worked well. After chaning DNS to point the new server I started to notice server showing Internal Server Error when it hits 1000 + concurrent users. So,  I enabled logs in express to see whats is going on.

// log every request to the logger
app.use(morgan('combined', {stream: log.stream})); 

This showed that client request hits the server, however it has a problem rendering the output page but why after hitting 1000 + users? Then it hit me, it must have ran out of file descriptors.

$ ulimit -n
1024.

The aha moment. It is hitting the ulimit. So I changed the ulimit

$ ulimit -n 2048

All worked fine after that. However ulimit seems to me temporary. To make permenet changes you must change few things.

1) Increase max number of ulimit open file in Linux

sudo nano /etc/sysctl.conf add end of line fs.file-max = 65536

2) sudo nano /etc/security/limits.conf and add below the mentioned

* soft nproc 65535
* hard nproc 65535
* soft nofile 65535
* hard nofile 65535

3) sudo nano /etc/security/limits.d/90-nproc.conf. Change to

*          soft    nproc     65535
root       soft    nproc     unlimited

How to install a system app in Android 5.0 or later devices ?

Rename your apk file to base.apk

$ adb push base.apk /sdcard/

$ adb shell

$ su

$ mount -o remount,rw -t yaffs2 /dev/block/mtdblock3 /system

$ mkdir /system/app/Test

$ chmod 755 /system/app/Test

$ cat /sdcard/base.apk > /system/app/Test/Base.apk

$chmod 644 /system/app/Test/Base.apk

$ mount -o remount,ro -t yaffs2 /dev/block/mtdblock3 /system

$ exit

Reboot your device. When boot completes you should see a system message like Android updating ...

How to check whether Android is running 64 bit version

If you want to check whether your device is running 64 bit version you can check System.getProperty("os.arch"). This will return aarch64 on Samsung S6.

More safe approach would be

boolean is64 = System.getProperty("os.arch").contains("64");

This does not work on Nexus 5x. Because Nexus 5x returns  armv8l


So, I changed my code to

boolean is64 = System.getProperty("ro.product.cpu.abilist").contains("64");

You can use shell command to check  as well

shell@zerolte:/ $ uname -m
uname -m
aarch64

WARNING: linker: could not load library "libsigchain.so" from LD_PRELOAD for "sh"; caused by "libsigchain.so" is 32-bit instead of 64-bit CANNOT LINK EXECUTABLE: could not load library "libc.so" needed by "sh"; caused by "libc.so" is 32-bit instead of 64-bit

I have been trying to fix my app to support Samsung S6. It seems Samsung S6 Android 5.0.2 runs in 64 bit. If you are seeing this error it is likely "libsigchain.so" is loaded from wrong location.

If you execute this command in shell,

root@zerolte:/ # printenv "LD_PRELOAD"
printenv "LD_PRELOAD"
libsigchain.so

you libsigchain.so likely being loaded from /system/lib. To fix

export LD_PRELOAD = /system/lib64/libsigchain.so

Or you can try something like this:

final String extStoreApkPath = ""
final String libPath = (is64bit() ? "/vendor/lib:/system/lib64" : "/vendor/lib:/system/lib");
final String cmd = String.format("LD_LIBRARY_PATH=%s pm install -r -d %s", libPath, extStoreApkPath);
                      

ERROR: ld.so: object ‘/system/lib/libsigchain.so’ from LD_PRELOAD cannot be preloaded: ignored

To fix this problem just run unset LD_PRELOAD