Slow CIFS/SMB File Access on FlexCache Volumes
Applies to
- ONTAP 9
- SMB/CIFS protocol
- FlexCache
- FSx
Issue
- Users experience significant delays (1.5–2 minutes or more) when opening or saving files (e.g., 30–150 MB Excel/spreadsheet documents) on CIFS/SMB shares hosted on FlexCache volumes. Expected file open time is under 30 seconds. The issue impacts hundreds of concurrent users and is production-affecting.
- File open times: 1.5–2 minutes for 30–50 MB files (expected: <30 seconds)
- FlexCache misses and
high fc_retrieve_latency(hundreds of ms to seconds) - CSM session op latency (
spinnp_op_latency): >500 ms, often several seconds - CSM session defer queue events incrementing heavily
- Elevated TCP retransmissions and out-of-order packets on intercluster network path
- Workload Dblade Delay Center (Cache Node):
start end instance iops read iops write iops other total_ops total_latency read (us) delay_flexcache_ral_wait_time(us) delay_flexcache_spinhi_wait_time(us) delay_wafl_susp_other_wait_time(us)2026-05-05 08:50:44+00:00 2026-05-05 08:51:44+00:00 vol_Share__0001-wid26319 0.38 0.13 16.33 16.85 90,467 10,597 2,336 10,5602026-05-05 09:00:44+00:00 2026-05-05 09:01:44+00:00 vol_Share__0001-wid26319 0.32 0.03 15.80 16.15 3,104,229 159,550 1,366 146,4082026-05-05 09:08:44+00:00 2026-05-05 09:09:44+00:00 vol_Share__0001-wid26319 0.17 0.07 17.65 17.88 9,346,544 472,873 11,777 477,391- FlexCache Retrieve Latency (waflremote — Cache Node):
start end instance fc_retrieve_ops fc_retrieve_latency (us)2026-05-05 08:49:44+00:00 2026-05-05 08:54:44+00:00 0 4,473 133,3142026-05-05 08:59:44+00:00 2026-05-05 09:04:44+00:00 0 8,939 1,807,0092026-05-05 09:04:44+00:00 2026-05-05 09:09:44+00:00 0 5,863 3,155,5722026-05-05 09:14:44+00:00 2026-05-05 09:19:44+00:00 0 8,209 4,982,7952026-05-05 09:39:44+00:00 2026-05-05 09:44:44+00:00 0 5,919 8,016,603- CSM Session Latency (Cache Node — spinnp_op_latency vs. service latency):
start end instance spinnp_op_latency (us) spinnp_req_service_latency (us)2026-05-05 08:55:01+00:00 2026-05-05 08:59:44+00:00 0x11064f358fa23cf7 125,624 1,3382026-05-05 09:05:01+00:00 2026-05-05 09:09:44+00:00 0x11064f358fa23cf7 3,946,479 114,4122026-05-05 09:15:01+00:00 2026-05-05 09:19:44+00:00 0x11064f358fa23cf7 5,458,466 1,292,2532026-05-05 09:40:01+00:00 2026-05-05 09:44:44+00:00 0x11064f358fa23cf7 9,194,737 2,372,408- Origin Retrieve Message Latency (confirming origin is healthy):
start end instance fc_retrieve_message_latency (us) fc_retrieve_message_ops2026-06-16 06:45:26+00:00 2026-06-16 06:50:26+00:00 0 2,081 27,9782026-06-16 07:05:26+00:00 2026-06-16 07:10:26+00:00 0 2,294 20,6782026-06-16 07:15:26+00:00 2026-06-16 07:20:26+00:00 0 1,483 42,286- remote set Output (Cache Node):
Node: cache-node-01Autoscrub: onCancel FC Ops: onDirectory chunk loading: enabledExpansive: offFastpath: onLogical directory caching: logicalReclaim: onTrace: onbli-decrement: 1bli-increment: 2bli-threshold: 0block-invalidate-test: offblocks-prefetch-factor: 4bulkattr-cache-size-limit: 32 MBcache-all: 3000000cache-rw: 512dp-cache-all: 100000csm-min-connections: 65535fc-disconnect-expiry: 3600fc-disconnect-timeout: 120fc-min-free: 10filemap-cache-entries-limit: 1608808filemap-cache-low-water-mark-pcnt: 80flexcache-writeback-origin-minfree-percent: 20inode-alloc-limit: 32lookup-cache-entries-limit: 313632 <<< Key indicator — oversubscribed for millions of filesnegative-lookup-cache: onorigin-protection: onorigin-protection-size-limit: 500parallel-retrieval-limit: 128prioritize-retrieve-sleepers: onretrieve-block-limit: 64scrub-max-async: 512scrub-in-stripe: onsideload: onstreams-caching: onstripe: onuse-incore-atime: onral-objects-freelist-size: Data Buffers Freelist: current 256 max 0 Retrieve Freelist: max 256 RetrieveClient Freelist: max 256Note : CSM connection starvation: CSM allocates medium-priority TCP connections proportional to the origin's CPU count. Example: A 4-CPU origin gets only 4 connections (2 request + 2 response). All remote I/O funnels through these limited connections, causing head-of-line blocking and defer queue buildup when requests arrive faster than connections can process them.