2026-01-13 06:13:15,186 - distributed.scheduler - ERROR - broadcast to tcp://100.96.7.6:40091 failed: OSError: Timed out trying to connect to tcp://100.96.7.6:40091 after 30 s
2026-01-12 00:24:07,659 - distributed.scheduler - ERROR - broadcast to tcp://100.96.7.6:40091 failed: OSError: Timed out trying to connect to tcp://100.96.7.6:40091 after 30 s
2026-01-10 14:03:55,159 - distributed.scheduler - ERROR - broadcast to tcp://100.96.7.6:40091 failed: OSError: Timed out trying to connect to tcp://100.96.7.6:40091 after 30 s
2026-01-09 04:16:28,402 - distributed.scheduler - ERROR - broadcast to tcp://100.96.7.6:40091 failed: OSError: Timed out trying to connect to tcp://100.96.7.6:40091 after 30 s
2026-01-06 15:26:51,570 - distributed.scheduler - ERROR - broadcast to tcp://100.96.7.6:40091 failed: OSError: Timed out trying to connect to tcp://100.96.7.6:40091 after 30 s
2026-01-06 14:12:51,609 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2026-01-06 10:16:44,548 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.71:46159 failed: OSError: Timed out trying to connect to tcp://100.96.8.71:46159 after 30 s
2026-01-05 11:51:21,327 - distributed.scheduler - ERROR - broadcast to tcp://100.96.7.6:40091 failed: OSError: Timed out trying to connect to tcp://100.96.7.6:40091 after 30 s
2026-01-05 10:55:46,992 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2026-01-05 00:41:27,205 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.71:46159 failed: OSError: Timed out trying to connect to tcp://100.96.8.71:46159 after 30 s
2026-01-04 09:00:38,188 - distributed.scheduler - ERROR - broadcast to tcp://100.96.7.6:40091 failed: OSError: Timed out trying to connect to tcp://100.96.7.6:40091 after 30 s
2026-01-04 08:00:38,195 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2026-01-03 20:11:42,047 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.71:46159 failed: OSError: Timed out trying to connect to tcp://100.96.8.71:46159 after 30 s
2026-01-03 08:23:34,258 - distributed.scheduler - ERROR - broadcast to tcp://100.96.7.6:40091 failed: OSError: Timed out trying to connect to tcp://100.96.7.6:40091 after 30 s
2026-01-03 04:21:36,233 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2026-01-02 16:17:49,629 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.71:46159 failed: OSError: Timed out trying to connect to tcp://100.96.8.71:46159 after 30 s
2026-01-02 04:59:00,205 - distributed.scheduler - ERROR - broadcast to tcp://100.96.7.6:40091 failed: OSError: Timed out trying to connect to tcp://100.96.7.6:40091 after 30 s
2026-01-02 04:21:30,482 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2026-01-01 14:02:08,884 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.71:46159 failed: OSError: Timed out trying to connect to tcp://100.96.8.71:46159 after 30 s
2026-01-01 02:02:20,002 - distributed.scheduler - ERROR - broadcast to tcp://100.96.7.6:40091 failed: OSError: Timed out trying to connect to tcp://100.96.7.6:40091 after 30 s
2026-01-01 01:29:16,071 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2026-01-01 00:30:39,583 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-31 10:55:48,905 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.71:46159 failed: OSError: Timed out trying to connect to tcp://100.96.8.71:46159 after 30 s
2025-12-31 00:49:52,750 - distributed.scheduler - ERROR - broadcast to tcp://100.96.7.6:40091 failed: OSError: Timed out trying to connect to tcp://100.96.7.6:40091 after 30 s
2025-12-30 21:49:46,755 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2025-12-30 21:43:45,902 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-30 07:55:53,835 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.71:46159 failed: OSError: Timed out trying to connect to tcp://100.96.8.71:46159 after 30 s
2025-12-29 21:56:22,636 - distributed.scheduler - ERROR - broadcast to tcp://100.96.7.6:40091 failed: OSError: Timed out trying to connect to tcp://100.96.7.6:40091 after 30 s
2025-12-29 21:06:22,609 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2025-12-29 18:37:22,971 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-29 05:55:38,882 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.71:46159 failed: OSError: Timed out trying to connect to tcp://100.96.8.71:46159 after 30 s
2025-12-28 17:30:06,465 - distributed.scheduler - ERROR - broadcast to tcp://100.96.7.6:40091 failed: OSError: Timed out trying to connect to tcp://100.96.7.6:40091 after 30 s
2025-12-28 17:13:06,432 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2025-12-28 11:45:00,947 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-28 02:07:08,117 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.71:46159 failed: OSError: Timed out trying to connect to tcp://100.96.8.71:46159 after 30 s
2025-12-27 16:39:15,898 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2025-12-27 13:47:28,846 - distributed.scheduler - ERROR - broadcast to tcp://100.96.7.6:40091 failed: OSError: Timed out trying to connect to tcp://100.96.7.6:40091 after 30 s
2025-12-27 12:00:20,311 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-26 17:05:21,225 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2025-12-26 11:29:53,539 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-26 04:03:18,743 - distributed.scheduler - ERROR - broadcast to tcp://100.96.7.6:40091 failed: OSError: Timed out trying to connect to tcp://100.96.7.6:40091 after 30 s
2025-12-25 19:46:19,672 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.71:46159 failed: OSError: Timed out trying to connect to tcp://100.96.8.71:46159 after 30 s
2025-12-25 15:38:23,207 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2025-12-25 10:59:58,643 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-24 14:48:11,507 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2025-12-24 11:19:34,971 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-23 21:34:34,976 - distributed.scheduler - ERROR - broadcast to tcp://100.96.7.6:40091 failed: OSError: Timed out trying to connect to tcp://100.96.7.6:40091 after 30 s
2025-12-23 14:57:34,916 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2025-12-23 09:06:34,861 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-22 10:39:40,531 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2025-12-22 07:30:11,179 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-21 07:08:53,316 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2025-12-21 05:52:53,930 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-20 05:22:46,804 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-19 16:30:49,775 - distributed.scheduler - ERROR - broadcast to tcp://100.96.8.66:43541 failed: OSError: Timed out trying to connect to tcp://100.96.8.66:43541 after 30 s
2025-12-19 02:27:11,316 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-17 23:10:28,242 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-16 18:25:22,329 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-14 10:57:14,757 - distributed.scheduler - ERROR - broadcast to tcp://100.96.3.2:37741 failed: OSError: Timed out trying to connect to tcp://100.96.3.2:37741 after 30 s
2025-12-12 21:32:51,048 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-10 18:40:51,593 - distributed.scheduler - ERROR - broadcast to tcp://100.96.9.152:44417 failed: OSError: Timed out trying to connect to tcp://100.96.9.152:44417 after 30 s
2025-12-10 17:25:51,801 - distributed.scheduler - ERROR - broadcast to tcp://100.96.3.2:37741 failed: OSError: Timed out trying to connect to tcp://100.96.3.2:37741 after 30 s
2025-12-09 03:23:26,755 - distributed.scheduler - ERROR - broadcast to tcp://100.96.3.2:37741 failed: OSError: Timed out trying to connect to tcp://100.96.3.2:37741 after 30 s
2025-12-07 18:23:17,464 - distributed.scheduler - ERROR - broadcast to tcp://100.96.3.2:37741 failed: OSError: Timed out trying to connect to tcp://100.96.3.2:37741 after 30 s
2025-12-05 14:13:15,589 - distributed.scheduler - ERROR - broadcast to tcp://100.96.3.3:37511 failed: OSError: Timed out trying to connect to tcp://100.96.3.3:37511 after 30 s
2025-12-05 12:22:37,082 - distributed.scheduler - ERROR - broadcast to tcp://100.96.3.2:37741 failed: OSError: Timed out trying to connect to tcp://100.96.3.2:37741 after 30 s
2025-12-03 23:37:55,501 - distributed.scheduler - ERROR - broadcast to tcp://100.96.3.2:37741 failed: OSError: Timed out trying to connect to tcp://100.96.3.2:37741 after 30 s
2025-12-03 00:36:41,395 - distributed.scheduler - ERROR - broadcast to tcp://100.96.3.5:40167 failed: OSError: Timed out trying to connect to tcp://100.96.3.5:40167 after 30 s
2025-12-02 23:51:41,388 - distributed.scheduler - ERROR - broadcast to tcp://100.96.3.3:37511 failed: OSError: Timed out trying to connect to tcp://100.96.3.3:37511 after 30 s
2025-12-02 23:36:41,395 - distributed.scheduler - ERROR - broadcast to tcp://100.96.3.2:37741 failed: OSError: Timed out trying to connect to tcp://100.96.3.2:37741 after 30 s
2025-12-02 20:21:29,808 - distributed.scheduler - ERROR - broadcast to tcp://100.96.3.5:40167 failed: OSError: Timed out trying to connect to tcp://100.96.3.5:40167 after 30 s
2025-12-01 14:28:59,586 - distributed.scheduler - INFO - Starting worker compute stream, tcp://100.96.11.22:38569
2025-12-01 14:28:59,586 - distributed.scheduler - INFO - Register worker addr: tcp://100.96.11.22:38569 name: tcp://100.96.11.22:38569
2025-12-01 14:28:41,709 - distributed.scheduler - INFO - Starting worker compute stream, tcp://100.96.4.4:40367
2025-12-01 14:28:41,708 - distributed.scheduler - INFO - Register worker addr: tcp://100.96.4.4:40367 name: tcp://100.96.4.4:40367
2025-12-01 14:28:13,168 - distributed.scheduler - INFO - Remove worker addr: tcp://100.96.8.66:44261 name: tcp://100.96.8.66:44261 (stimulus_id='handle-worker-cleanup-1764599293.167952')
2025-12-01 14:28:13,139 - distributed.scheduler - INFO - Remove worker addr: tcp://100.96.8.71:38381 name: tcp://100.96.8.71:38381 (stimulus_id='handle-worker-cleanup-1764599293.1396623')
2025-12-01 13:59:13,722 - distributed.scheduler - INFO - Starting worker compute stream, tcp://100.96.2.4:43143
2025-12-01 13:59:13,722 - distributed.scheduler - INFO - Register worker addr: tcp://100.96.2.4:43143 name: tcp://100.96.2.4:43143
2025-12-01 13:54:45,174 - distributed.scheduler - INFO - Starting worker compute stream, tcp://100.96.7.6:46753
2025-12-01 13:54:45,174 - distributed.scheduler - INFO - Register worker addr: tcp://100.96.7.6:46753 name: tcp://100.96.7.6:46753
2025-12-01 13:54:43,101 - distributed.scheduler - INFO - Starting worker compute stream, tcp://100.96.8.66:44261
2025-12-01 13:54:43,101 - distributed.scheduler - INFO - Register worker addr: tcp://100.96.8.66:44261 name: tcp://100.96.8.66:44261
2025-12-01 13:54:42,831 - distributed.scheduler - INFO - Starting worker compute stream, tcp://100.96.8.71:38381
2025-12-01 13:54:42,825 - distributed.scheduler - INFO - Register worker addr: tcp://100.96.8.71:38381 name: tcp://100.96.8.71:38381
2025-12-01 13:54:42,258 - distributed.scheduler - INFO - Registering Worker plugin shuffle
2025-12-01 13:54:42,258 - distributed.scheduler - INFO - dashboard at: http://100.96.2.14:8787/status
2025-12-01 13:54:42,258 - distributed.scheduler - INFO - Scheduler at: tcp://100.96.2.14:8786
2025-12-01 13:54:42,257 - distributed.scheduler - INFO - -----------------------------------------------
2025-12-01 13:54:42,253 - distributed.scheduler - INFO - State start