Alfresco ECM on Thecus N2800 NAS

After several kernel panics, and reinstallation procedures I’ve done with a proof-of-concept implementation of Alfresco ECM running on Thecus N2800 NAS.

Thecus N2800 is one of the fastest NAS devices today with 2.13GHz Intel Atom D2701 CPU (2 cores, 4 threads). This speed and performance more than enough for NAS, but presents a low-end environment in – CPU intensive – server functions.

CPU speed benchmark:
Intel_Atom_D2701-speed

CPU details:

Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    2
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 54
Model name:            Intel(R) Atom(TM) CPU D2701   @ 2.13GHz
Stepping:              1
CPU MHz:               2127.954
BogoMIPS:              4255.90
L1d cache:             24K
L1i cache:             32K
L2 cache:              512K
NUMA node0 CPU(s):     0-3

The NAS shipped with 2GB memory, but second memory slot is empty – simple to upgrade with cheap DDR3 module (Alfresco environment needs 4GB system memory):
thecus-2800-6

The operating system (Ubuntu Server 15.04, 64bit) is installed on a 8GB SDCARD, but application and database stored on mounted raid partition.

SDCARD speed with 8GB Class 10 card:

/dev/sdd:
 Timing cached reads:   1614 MB in  2.00 seconds = 806.59 MB/sec
 Timing buffered disk reads:  60 MB in  3.08 seconds =  19.48 MB/sec

RAID array speed with two 3GB HDDs:

/dev/md0:
 Timing cached reads:   1796 MB in  2.00 seconds = 897.83 MB/sec
 Timing buffered disk reads: 432 MB in  3.01 seconds = 143.63 MB/sec

Speed of disks (Toshiba and WD):

/dev/sdb:
 Timing cached reads:   1852 MB in  2.00 seconds = 926.10 MB/sec
 Timing buffered disk reads: 402 MB in  3.05 seconds = 131.99 MB/sec
root@storage-2:~# hdparm -tT /dev/sdc

/dev/sdc:
 Timing cached reads:   1878 MB in  2.00 seconds = 938.63 MB/sec
 Timing buffered disk reads: 540 MB in  3.01 seconds = 179.12 MB/sec

PCI device list:

root@storage-2:~# lspci 
00:00.0 Host bridge: Intel Corporation Atom Processor D2xxx/N2xxx DRAM Controller (rev 04)
00:02.0 VGA compatible controller: Intel Corporation Atom Processor D2xxx/N2xxx Integrated Graphics Controller (rev 0b)
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1b.0 Audio device: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1c.1 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Port 2
00:1c.2 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 3
00:1c.3 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 4
00:1c.4 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5
00:1c.5 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
05:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)

Network device kernel driver bug:

May 17 07:34:16 storage-2 kernel: [  105.885785] WARNING: CPU: 3 PID: 0 at /build/buildd/linux-3.19.0/net/sched/sch_generic.c:303 dev_watchdog+0x240/0x250()
May 17 07:34:16 storage-2 kernel: [  105.885793] NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0 timed out
May 17 07:34:16 storage-2 kernel: [  105.885799] Modules linked in: des3_ede_x86_64 des_generic md4 nls_utf8 cifs fscache coretemp serio_raw snd_hda_codec_hdmi snd_hda_intel snd_hda_controller lpc_ich snd_hda_codec snd_hwdep snd_pcm snd_timer snd shpchp soundcore 8250_fintek video mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor hid_generic uas psmouse ahci raid6_pq libahci usb_storage usbhid e1000e raid1 raid0 ptp hid multipath pps_core linear
May 17 07:34:16 storage-2 kernel: [  105.885891] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.19.0-15-generic #15-Ubuntu
May 17 07:34:16 storage-2 kernel: [  105.885898] Hardware name: Intel Corporation Milstead Platform/Granite Well, BIOS CDV_T30 X64 09/17/2012
May 17 07:34:16 storage-2 kernel: [  105.885905]  ffffffff81b3ed68 ffff88013fd83d58 ffffffff817c2205 0000000000040400
May 17 07:34:16 storage-2 kernel: [  105.885915]  ffff88013fd83da8 ffff88013fd83d98 ffffffff8107595a 0000000000014200
May 17 07:34:16 storage-2 kernel: [  105.885927]  0000000000000000 ffff880036716480 0000000000000003 ffff880035ca8000
May 17 07:34:16 storage-2 kernel: [  105.885938] Call Trace:
May 17 07:34:16 storage-2 kernel: [  105.885945]    [] dump_stack+0x45/0x57
May 17 07:34:16 storage-2 kernel: [  105.885966]  [] warn_slowpath_common+0x8a/0xc0
May 17 07:34:16 storage-2 kernel: [  105.885977]  [] warn_slowpath_fmt+0x46/0x50
May 17 07:34:16 storage-2 kernel: [  105.885996]  [] dev_watchdog+0x240/0x250
May 17 07:34:16 storage-2 kernel: [  105.886008]  [] ? dev_graft_qdisc+0x80/0x80
May 17 07:34:16 storage-2 kernel: [  105.886019]  [] call_timer_fn+0x39/0x110
May 17 07:34:16 storage-2 kernel: [  105.886029]  [] ? dev_graft_qdisc+0x80/0x80
May 17 07:34:16 storage-2 kernel: [  105.886039]  [] run_timer_softirq+0x250/0x320
May 17 07:34:16 storage-2 kernel: [  105.886051]  [] __do_softirq+0x109/0x270
May 17 07:34:16 storage-2 kernel: [  105.886066]  [] irq_exit+0x95/0xa0
May 17 07:34:16 storage-2 kernel: [  105.886078]  [] smp_apic_timer_interrupt+0x46/0x60
May 17 07:34:16 storage-2 kernel: [  105.886089]  [] apic_timer_interrupt+0x6d/0x80
May 17 07:34:16 storage-2 kernel: [  105.886095]    [] ? __hrtimer_start_range_ns+0x1ce/0x410
May 17 07:34:16 storage-2 kernel: [  105.886113]  [] ? poll_idle+0x10/0x80
May 17 07:34:16 storage-2 kernel: [  105.886123]  [] cpuidle_enter_state+0x49/0x160
May 17 07:34:16 storage-2 kernel: [  105.886136]  [] cpuidle_enter+0x17/0x20
May 17 07:34:16 storage-2 kernel: [  105.886147]  [] cpu_startup_entry+0x311/0x3b0
May 17 07:34:16 storage-2 kernel: [  105.886160]  [] start_secondary+0x197/0x1c0
May 17 07:34:16 storage-2 kernel: [  105.886168] ---[ end trace 3cfbe6c4691973b1 ]---
May 17 07:34:16 storage-2 kernel: [  105.886466] e1000e 0000:01:00.0 eth0: Reset adapter unexpectedly
May 17 07:34:19 storage-2 kernel: [  109.603093] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

Fix/workaround for previous “e1000e” driver bug:

ethtool -K eth0 tso off

Fix for power-saved console:

echo "blacklist gma500_gfx" >> /etc/modprobe.d/blacklist.conf
sudo update-initramfs -u

Results of the Alfresco 4.2.f (with several extra modules, like OCR, VirusCheck, etc. and over 50.000 documents) startup:

INFO: Server startup in 440550 ms

Just for the contrast, Alfresco server boot time on 4 core Intel Xeon E31245, 3.3GHz with same configuration and database:

INFO: Server startup in 70426 ms

Problems to solve:

May 17 09:56:32 storage-2 systemd[1]: systemd-logind.service watchdog timeout (limit 1min)!
May 17 09:56:33 storage-2 kernel: [ 4062.392447] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [convert:3209]
May 17 09:56:33 storage-2 kernel: [ 4062.392751] Modules linked in: it87 hwmon_vid des3_ede_x86_64 des_generic md4 nls_utf8 cifs fscache coretemp snd_hda_codec_hdmi serio_raw snd_hda_intel snd_hda_controller snd_hda_codec snd_hwdep lpc_ich snd_pcm snd_timer snd shpchp soundcore 8250_fintek video mac_hid nfsd auth_rpcgss nfs_acl lockd grace sunrpc autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor uas psmouse raid6_pq raid1 ahci e1000e usb_storage raid0 ptp libahci multipath pps_core linear
May 17 09:56:34 storage-2 kernel: [ 4062.392819] CPU: 2 PID: 3209 Comm: convert Not tainted 3.19.0-15-generic #15-Ubuntu
May 17 09:56:34 storage-2 kernel: [ 4062.392823] Hardware name: Intel Corporation Milstead Platform/Granite Well, BIOS CDV_T30 X64 09/17/2012
May 17 09:56:34 storage-2 kernel: [ 4062.392828] task: ffff880049c62740 ti: ffff880049f44000 task.ti: ffff880049f44000
May 17 09:56:34 storage-2 kernel: [ 4062.392831] RIP: 0010:[]  [] smp_call_function_many+0x212/0x270
May 17 09:56:34 storage-2 kernel: [ 4062.392843] RSP: 0018:ffff880049f479a8  EFLAGS: 00000202
May 17 09:56:34 storage-2 kernel: [ 4062.392847] RAX: ffff88013fc98288 RBX: ffffffff8112d486 RCX: 0000000000000001
May 17 09:56:34 storage-2 kernel: [ 4062.392850] RDX: ffff88013fc98288 RSI: 0000000000000100 RDI: 0000000000000000
May 17 09:56:34 storage-2 kernel: [ 4062.392853] RBP: ffff880049f479e8 R08: ffff88013fd15208 R09: 0000000000000000
May 17 09:56:34 storage-2 kernel: [ 4062.392856] R10: 0000000000000004 R11: 0000000000000000 R12: 0000000000000297
May 17 09:56:34 storage-2 kernel: [ 4062.392860] R13: ffff880049f47958 R14: ffffffff810e4d2b R15: ffff880049f47918
May 17 09:56:34 storage-2 kernel: [ 4062.392864] FS:  00007f81b34f5780(0000) GS:ffff88013fd00000(0000) knlGS:0000000000000000
May 17 09:56:34 storage-2 kernel: [ 4062.392868] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 17 09:56:34 storage-2 kernel: [ 4062.392871] CR2: 00007f81b34f0ad7 CR3: 000000001d494000 CR4: 00000000000007e0
May 17 09:56:34 storage-2 kernel: [ 4062.392874] Stack:
May 17 09:56:34 storage-2 kernel: [ 4062.392877]  000000013fd17c10 00000000000151c0 0000000000000000 0000000000000002
May 17 09:56:34 storage-2 kernel: [ 4062.392882]  ffffffff81fa70a0 ffffffff81180ee0 0000000000000000 0000000000000003
May 17 09:56:34 storage-2 kernel: [ 4062.392887]  ffff880049f47a18 ffffffff810f39e8 0000000000000100 0000000000000000
May 17 09:56:34 storage-2 kernel: [ 4062.392892] Call Trace:
May 17 09:56:34 storage-2 kernel: [ 4062.392903]  [] ? page_alloc_cpu_notify+0x50/0x50
May 17 09:56:34 storage-2 kernel: [ 4062.392910]  [] on_each_cpu_mask+0x28/0x60
May 17 09:56:34 storage-2 kernel: [ 4062.392916]  [] drain_all_pages+0xeb/0x100
May 17 09:56:34 storage-2 kernel: [ 4062.392921]  [] __alloc_pages_nodemask+0x754/0xba0
May 17 09:56:34 storage-2 kernel: [ 4062.392929]  [] alloc_pages_current+0x91/0x110
May 17 09:56:34 storage-2 kernel: [ 4062.392935]  [] __page_cache_alloc+0xa7/0xd0
May 17 09:56:34 storage-2 kernel: [ 4062.392941]  [] __do_page_cache_readahead+0xe2/0x210
May 17 09:56:34 storage-2 kernel: [ 4062.392947]  [] ondemand_readahead+0xdf/0x260
May 17 09:56:34 storage-2 kernel: [ 4062.392952]  [] page_cache_async_readahead+0x71/0xa0
May 17 09:56:34 storage-2 kernel: [ 4062.392958]  [] generic_file_read_iter+0x40f/0x5e0
May 17 09:56:34 storage-2 kernel: [ 4062.392965]  [] new_sync_read+0x7e/0xb0
May 17 09:56:34 storage-2 kernel: [ 4062.392971]  [] __vfs_read+0x18/0x50
May 17 09:56:34 storage-2 kernel: [ 4062.392976]  [] vfs_read+0x8a/0x140
May 17 09:56:34 storage-2 kernel: [ 4062.392982]  [] SyS_pread64+0x72/0xb0
May 17 09:56:34 storage-2 kernel: [ 4062.392989]  [] system_call_fastpath+0x16/0x1b
May 17 09:56:34 storage-2 kernel: [ 4062.392992] Code: 65 44 2d 00 3b 05 a3 67 c3 00 89 c1 0f 8d 7e fe ff ff 48 98 49 8b 55 00 48 03 14 c5 40 7f d2 81 f6 42 18 01 48 89 d0 74 c8 f3 90  40 18 01 75 f8 eb be 0f b6 4d c4 4c 89 fa 4c 89 f6 44 89 ef 
May 17 09:56:41 storage-2 systemd[1]: systemd-logind.service: main process exited, code=dumped, status=6/ABRT
May 17 09:56:54 storage-2 systemd[1]: Unit systemd-logind.service entered failed state.
May 17 09:56:54 storage-2 systemd[1]: systemd-logind.service failed.