An ESXi host stops responding and displays a purple diagnostic screen after installing Trend Micro Deep Security

気になるKBを発見したので、またまたメモ代わりです。
KB2012584
「Deep Security 8.0をESX5に入れたらPSOD」

[現象] 以下のような症状が現れます。

・ESXiが応答しなくなり、以下のようなメッセージをはきだす
Code start: 0x41801f800000 VMK uptime: 2:23:12:36.786
0x412212d87310:[0x41802006ba7b]timerq_RB_INSERT_COLOR@<None>#<None>+0x17a stack: 0x412212d87360
0x412212d87320:[0x41802006bf55]timerq_RB_INSERT@<None>#<None>+0x60 stack: 0x412212d87340
0x412212d87360:[0x41802006c1e3]_timer_insert@<None>#<None>+0xda stack: 0x412212d873a0
0x412212d87380:[0x41802006c2d7]tb_timer_run_in@<None>#<None>+0x4a stack: 0x412212d873a0
0x412212d873e0:[0x418020034c2b]dom_payload_init@<None>#<None>+0x302 stack: 0x41003d233050
0x412212d87420:[0x4180200247d1]tb_dom_init@<None>#<None>+0x78 stack: 0x417fdff0ee58
0x412212d87450:[0x4180200696c3]alloc_guest@<None>#<None>+0x8e stack: 0x412212d875c4
0x412212d87600:[0x418020069dfc]dsa_reg_filter@<None>#<None>+0x3df stack: 0x4100224b0db0
0x412212d87620:[0x418020067ce0]dv_create_filt@<None>#<None>+0x13 stack: 0x0
0x412212d876b0:[0x418020002662]DVFilterCreateFilter@com.vmware.vmkapi#v2_0_0_0+0x87d stack: 0x4100
0x412212d87720:[0x4180200028b8]DVFilterHotAddFilter@com.vmware.vmkapi#v2_0_0_0+0xe7 stack: 0x41001
0x412212d87840:[0x418020002c64]DVFilterActivateVNic@com.vmware.vmkapi#v2_0_0_0+0x2db stack: 0x4100
0x412212d87870:[0x41801f8f60df]DVFilterActivateCommon@vmkernel#nover+0x5a stack: 0x412212d87a30
0x412212d87a00:[0x41801f8f8eca]Net_ConnectDVPort@vmkernel#nover+0x28d stack: 0x412212d87a20
0x412212d87e70:[0x41801f9c4a31]UWVMKSyscall_NetConnectDVPort@vmkernel#nover+0xc4 stack: 0x38342e68
0x412212d87ee0:[0x41801f9bd5ff]UW64VMKSyscallUnpackNetConnectDVPort@vmkernel#nover+0x8a stack: 0x1
0x412212d87f20:[0x41801f98d15d]User_LinuxSyscallHandler@vmkernel#nover+0x144 stack: 0x3ffc2392e800x412212d87f30:[0x41801f8f1047]gate_entry@vmkernel#nover+0x46 stack: 0x10b

・以下Backtraceと共に死ぬ
2011-10-20T07:20:35.053Z cpu2:4098)@BlueScreen: PCPU 0: no heartbeat (but 2/2 IPIs received).
2011-10-20T07:20:35.054Z cpu2:4098)Code start: 0x41800d600000 VMK uptime: 6:14:25:59.57
2011-10-20T07:20:35.054Z cpu2:4098)Saved backtrace from: pcpu 0 Heartbeat NMI
2011-10-20T07:20:35.054Z cpu2:4098)0x412200607ad8:[0x41800d694af1]TimerInsert@vmkernel#nover+0x158 stack: 0x2000000df
2011-10-20T07:20:35.055Z cpu2:4098)0x412200607b38:[0x41800d695f58]Timer_AddTCWithLockDomain@vmkernel#nover+0x113 stack: 0x0
2011-10-20T07:20:35.056Z cpu2:4098)0x412200607b98:[0x41800d612317]vmk_TimerSchedule@vmkernel#nover+0xe2 stack: 0x3
2011-10-20T07:20:35.056Z cpu2:4098)0x412200607bd8:[0x41800df04eb5]_timer_insert@<None>#<None>+0x5c stack: 0x4100304735b0
2011-10-20T07:20:35.057Z cpu2:4098)0x412200607bf8:[0x41800df04fca]timer_wrapper_fn_50@<None>#<None>+0x85 stack: 0x412200607c38
2011-10-20T07:20:35.058Z cpu2:4098)0x412200607c98:[0x41800d6969d2]Timer_BHHandler@vmkernel#nover+0x225 stack: 0x41220000002c
2011-10-20T07:20:35.058Z cpu2:4098)0x412200607cf8:[0x41800d61824c]BHCallHandlers@vmkernel#nover+0xbb stack: 0xdf
2011-10-20T07:20:35.059Z cpu2:4098)0x412200607d38:[0x41800d61873b]BH_Check@vmkernel#nover+0xde stack: 0x2710
2011-10-20T07:20:35.060Z cpu2:4098)0x412200607d78:[0x41800d641eed]IDT_HandleInterrupt@vmkernel#nover+0x13c stack: 0x418040000140
2011-10-20T07:20:35.060Z cpu2:4098)0x412200607d98:[0x41800d64274d]IDT_IntrHandler@vmkernel#nover+0xa4 stack: 0x412200607ea0
2011-10-20T07:20:35.061Z cpu2:4098)0x412200607da8:[0x41800d6f1047]gate_entry@vmkernel#nover+0x46 stack: 0x4018
2011-10-20T07:20:35.062Z cpu2:4098)0x412200607ea0:[0x41800d8fe3b1]Power_HaltPCPU@vmkernel#nover+0x274 stack: 0x5f1a1e2b1c9f8
2011-10-20T07:20:35.063Z cpu2:4098)0x412200607fd0:[0x41800d7ef2ca]CpuSchedIdleLoopInt@vmkernel#nover+0xb3d stack: 0x412200607ff0
2011-10-20T07:20:35.063Z cpu2:4098)0x412200607fe0:[0x41800d7f62c6]CpuSched_IdleLoop@vmkernel#nover+0x15 stack: 0x412200607ff8
2011-10-20T07:20:35.064Z cpu2:4098)0x412200607ff0:[0x41800d643b42]HostPCPUIdle@vmkernel#nover+0xd stack: 0x0
2011-10-20T07:20:35.065Z cpu2:4098)0x412200607ff8:[0x0]<unknown> stack: 0x0
2011-10-20T07:20:35.072Z cpu2:4098)base fs=0x0 gs=0x418040800000 Kgs=0x0
2011-10-20T07:20:35.077Z cpu2:4098)Backtrace for current CPU #2, worldID=4098, ebp=0x412200087a68
2011-10-20T07:20:35.077Z cpu2:4098)0x412200087a68:[0x41800d66cc78]Panic_WithBacktrace@vmkernel#nover+0xa3 stack: 0x412200087a98, 0xb0,

・VMkernelのログに以下 dvfilter-dsa関連のメッセージが多数でてる
2011-10-20T07:20:13.324Z cpu0:4120)dvfilter-dsa: tb_trace_write_formatted:105: iap_emit_logentry:69 EVENT ioctl error (0xbad0040): No connection

・クラッシュダンプが、Backtrace内のimerq_RB_MINMAXとDVFilterを参照
2012-02-17T01:41:21.428Z cpu8:1257306)Backtrace for current CPU #8, worldID=1257306, ebp=0x41223d687970
2012-02-17T01:41:21.429Z cpu8:1257306)0x41223d687970:[0x41801b06bef0]timerq_RB_MINMAX@<None>#<None>+0x1b stack: 0x600000000001, 0x410036d
2012-02-17T01:41:21.429Z cpu8:1257306)0x41223d6879b0:[0x41801b06c093]_timer_insert@<None>#<None>+0x7a stack: 0x410036d57730, 0x41223d687a
2012-02-17T01:41:21.429Z cpu8:1257306)0x41223d6879e0:[0x41801b06c213]tb_timer_run_in@<None>#<None>+0x66 stack: 0x41223d687a00, 0x410036d5
2012-02-17T01:41:21.430Z cpu8:1257306)0x41223d687a40:[0x41801b034c2b]dom_payload_init@<None>#<None>+0x302 stack: 0x410036de61e0, 0x410036
2012-02-17T01:41:21.430Z cpu8:1257306)0x41223d687a80:[0x41801b0247d1]tb_dom_init@<None>#<None>+0x78 stack: 0x417fdaf0ee98, 0x4100380fe260
2012-02-17T01:41:21.430Z cpu8:1257306)0x41223d687ab0:[0x41801b069583]alloc_guest@<None>#<None>+0x8e stack: 0x41223d687c24, 0x41223d687c25
2012-02-17T01:41:21.431Z cpu8:1257306)0x41223d687c60:[0x41801b069cc9]dsa_reg_filter@<None>#<None>+0x3ec stack: 0x41003400f090, 0x4100224a
2012-02-17T01:41:21.431Z cpu8:1257306)0x41223d687c80:[0x41801b067bac]dv_create_filt@<None>#<None>+0x13 stack: 0x0, 0x100410011c2b140, 0x4
2012-02-17T01:41:21.431Z cpu8:1257306)0x41223d687d10:[0x41801b002662]DVFilterCreateFilter@com.vmware.vmkapi#v2_0_0_0+0x87d stack: 0x41003
2012-02-17T01:41:21.432Z cpu8:1257306)0x41223d687d80:[0x41801b0028b8]DVFilterHotAddFilter@com.vmware.vmkapi#v2_0_0_0+0xe7 stack: 0x41801a
2012-02-17T01:41:21.432Z cpu8:1257306)0x41223d687ea0:[0x41801b002c64]DVFilterActivateVNic@com.vmware.vmkapi#v2_0_0_0+0x2db stack: 0x41223

・Community Supportedな以下のようなVIBを持っている
Trend_bootbank_dvfilter-dsa_8.0.0-1131

・ESXi 5.0 Build514841はDSA関連の以下メッセージをPSODと共に
dvfilter-dsa 8.0.0.-1189 Trend VMwareAccepted 2012-02-25

・ESXi 5.0 Build51481はエラースタックの中で dsa_save_state とDVFilterCheckPointをはく
2012-02-07T18:41:11.340Z cpu0:195953)0x412235c47da0:[0x41800907f810]dsa_save_state@<None>#<None>+0x63 stack: 0x1cfe, 0x412235c47de0, 0x4
2012-02-07T18:41:11.341Z cpu0:195953)0x412235c47e10:[0x41800901f249]DVFilterCheckpointGet@com.vmware.vmkapi#v2_0_0_0+0x1b0 stack: 0xffff
2012-02-07T18:41:11.342Z cpu0:195953)0x412235c47e60:[0x41800945af22]VMotionRecv_GetDVFilterState@esx#nover+0x6d stack: 0x4125600011f0, 0
2012-02-07T18:41:11.343Z cpu0:195953)0x412235c47eb0:[0x41800945c89c]VMotionRecv_ExecHandler@esx#nover+0x123 stack: 0x0, 0x0, 0x0, 0x0, 0
2012-02-07T18:41:11.344Z cpu0:195953)0x412235c47ff0:[0x41800945cc2f]VMotionRecv_Helper@esx#nover+0x2fe stack: 0x0, 0x0, 0x0, 0x0, 0x0

[原因・対処手順] Deep Security 側の問題で発生します。以下TrendのKBへジャンプ
Trend SolutionID 1060125

1. ESXのコンソールから、Filter Driver heap memory sizeを確認する
   % esxcfg-module -g dvfilter-dsa
   If the value for the "DSAFILTER_HEAP_MAX_SIZE" is adjusted from its default value then the outcome will be similar to:
   dvfilter-dsa enabled = 1 options = 'DSAFILTER_HEAP_MAX_SIZE=134217728'
2. タイマーをDisableにして、1で調べた値をDSAFILTER_HEAP_MAX_SIZEにセットする
   % esxcfg-module -s "DSAFILTER_HEAP_MAX_SIZE= 134217728 DSAFILTER_MOD_TIMER_ENABLED=0" dvfilter-dsa
   Note: Set the DSAFILTER_HEAP_MAX_SIZE to the value that was observed after running the "esxcfg-module -g dvfilter-dsa" command.
   [デフォルト値から変更してない場合は以下]
   If the value for the "DSAFILTER_HEAP_MAX_SIZE" is not changed from its default value then the outcome will be similar to:
   dvfilter-dsa enabled = 1 options = ' '
   In this case you can use the following command to disable timers:
   % esxcfg-module -s "DSAFILTER_MOD_TIMER_ENABLED=0" dvfilter-dsa
3. 設定がイケているかどうかを確認する
   
   % esxcfg-module -g dvfilter-dsa
4. リブートして設定を反映する
   
   Note: The setting will not take effect until the driver is reloaded. Reloading will require a reboot (best option) of ESXi or unloading/loading of the driver.
   Future releases will have timers disabled by default.
   =>今すぐにやってほしいぜ