Подскажите пожалуйста почему код не работает
Found 575 images belonging to 3 classes.
Found 69 images belonging to 3 classes.
Epoch 1/30
C:\Users\kamil\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\util\dispatch.py:1096: UserWarning: "`categorical_crossentropy` received `from_logits=True`, but the `output`
argument was produced by a sigmoid or softmax activation and thus does not represent logits. Was this intended?"
return dispatch_target(*args, **kwargs)
2021-11-29 11:53:29.606911: E tensorflow/stream_executor/cuda/cuda_driver.cc:1018] failed to synchronize the stop event: CUDA_ERROR_LAUNCH_TIMEOUT: the launch timed out and was terminated
2021-11-29 11:53:29.607332: E tensorflow/stream_executor/gpu/gpu_timer.cc:55] INTERNAL: Error destroying CUDA event: CUDA_ERROR_LAUNCH_TIMEOUT: the launch timed out and was terminated
2021-11-29 11:53:29.607544: E tensorflow/stream_executor/gpu/gpu_timer.cc:60] INTERNAL: Error destroying CUDA event: CUDA_ERROR_LAUNCH_TIMEOUT: the launch timed out and was terminated
2021-11-29 11:53:29.607703: E tensorflow/stream_executor/stream.cc:4476] INTERNAL: Failed to enqueue async memset operation: CUDA_ERROR_LAUNCH_TIMEOUT: the launch timed out and was terminated
2021-11-29 11:53:29.607898: E tensorflow/stream_executor/stream.cc:4476] INTERNAL: Failed to enqueue async memset operation: CUDA_ERROR_LAUNCH_TIMEOUT: the launch timed out and was terminated
2021-11-29 11:53:29.608111: E tensorflow/stream_executor/cuda/cuda_driver.cc:1163] failed to enqueue async memcpy from device to host: CUDA_ERROR_LAUNCH_TIMEOUT: the launch timed out and was terminated; host dst: 0xf6f6f6bc90; GPU src: (nil); size: 8=0x8
2021-11-29 11:53:29.608404: F tensorflow/stream_executor/cuda/cuda_dnn.cc:213] Check failed: status == CUDNN_STATUS_SUCCESS (7 vs. 0)Failed to set cuDNN stream.
PS C:\Users\kamil> & C:/Users/kamil/.conda/envs/tensorflow/python.exe d:/dataset/recognition.py
Found 575 images belonging to 3 classes.
Found 69 images belonging to 3 classes.
2021-11-29 12:08:49.870032: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-11-29 12:08:50.244496: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 1402 MB memory: -> device: 0, name: GeForce GT 710, pci bus id: 0000:08:00.0, compute capability: 3.5
Epoch 1/30
C:\Users\kamil\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\util\dispatch.py:1096: UserWarning: "`categorical_crossentropy` received `from_logits=True`, but the `output`
argument was produced by a sigmoid or softmax activation and thus does not represent logits. Was this intended?"
return dispatch_target(*args, **kwargs)
2021-11-29 12:08:52.154918: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8100
2021-11-29 12:08:53.124547: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.08GiB with freed_by_count=0. The caller
indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-11-29 12:08:54.167153: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1018.38MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-11-29 12:08:55.166410: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 729.89MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-11-29 12:08:55.479574: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 562.77MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-11-29 12:08:55.879992: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 360.53MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-11-29 12:08:55.880466: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 532.02MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-11-29 12:08:56.507983: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 790.02MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-11-29 12:08:56.650998: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 289.45MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-11-29 12:08:57.183400: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 866.56MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2021-11-29 12:09:01.663207: W tensorflow/core/common_runtime/bfc_allocator.cc:275] Allocator (GPU_0_bfc) ran out of memory trying to allocate 562.77MiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
8/95 [=>............................] - ETA: 6:02 - loss: 2.7465 - accuracy: 0.35692021-11-29 12:09:47.664194: W tensorflow/core/common_runtime/bfc_allocator.cc:462] Allocator (GPU_0_bfc) ran out of memory trying to allocate 321.75MiB (rounded to 337383424)requested by op gradient_tape/sequential/max_pooling2d/MaxPool/MaxPoolGrad
If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation.
Current allocation summary follows.
Current allocation summary follows.
2021-11-29 12:09:47.665180: I tensorflow/core/common_runtime/bfc_allocator.cc:1010] BFCAllocator dump for GPU_0_bfc
2021-11-29 12:09:47.665411: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (256): Total Chunks: 51, Chunks in use: 51. 12.8KiB allocated for chunks. 12.8KiB in use in bin. 1.7KiB client-requested in use in bin.
2021-11-29 12:09:47.665686: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (512): Total Chunks: 5, Chunks in use: 3. 3.2KiB allocated for chunks. 2.2KiB in use in bin. 2.2KiB client-requested in use in bin.
2021-11-29 12:09:47.665923: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (1024): Total Chunks: 4, Chunks in use: 4. 5.0KiB allocated for chunks. 5.0KiB in use in bin. 3.5KiB client-requested in use in bin.
2021-11-29 12:09:47.666184: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-29 12:09:47.666454: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (4096): Total Chunks: 4, Chunks in use: 4. 18.0KiB allocated for chunks. 18.0KiB in use
in bin. 18.0KiB client-requested in use in bin.
2021-11-29 12:09:47.666713: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (8192): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-29 12:09:47.666916: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (16384): Total Chunks: 3, Chunks in use: 3. 67.5KiB allocated for chunks. 67.5KiB in use
in bin. 54.0KiB client-requested in use in bin.
2021-11-29 12:09:47.667116: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (32768): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-29 12:09:47.667285: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-29 12:09:47.667447: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-29 12:09:47.667631: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-29 12:09:47.667802: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-29 12:09:47.667976: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-29 12:09:47.668154: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (2097152): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-29 12:09:47.668311: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (4194304): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-29 12:09:47.668496: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-29 12:09:47.668703: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-11-29 12:09:47.668893: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (33554432): Total Chunks: 4, Chunks in use: 3. 153.09MiB allocated for chunks. 114.84MiB in
use in bin. 114.84MiB client-requested in use in bin.
2021-11-29 12:09:47.669106: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (67108864): Total Chunks: 2, Chunks in use: 2. 243.00MiB allocated for chunks. 243.00MiB in
use in bin. 201.94MiB client-requested in use in bin.
2021-11-29 12:09:47.669305: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (134217728): Total Chunks: 2, Chunks in use: 1. 390.41MiB allocated for chunks. 151.33MiB in
use in bin. 80.44MiB client-requested in use in bin.
2021-11-29 12:09:47.669493: I tensorflow/core/common_runtime/bfc_allocator.cc:1017] Bin (268435456): Total Chunks: 2, Chunks in use: 1. 615.49MiB allocated for chunks. 321.75MiB in
use in bin. 321.75MiB client-requested in use in bin.
2021-11-29 12:09:47.669666: I tensorflow/core/common_runtime/bfc_allocator.cc:1033] Bin for 321.75MiB was 256.00MiB, Chunk State:
2021-11-29 12:09:47.669765: I tensorflow/core/common_runtime/bfc_allocator.cc:1039] Size: 293.74MiB | Requested Size: 158.64MiB | in_use: 0 | bin_num: 20, prev: Size: 321.75MiB | Requested Size: 321.75MiB | in_use: 1 | bin_num: -1, next: Size: 256B | Requested Size: 4B | in_use: 1 | bin_num: -1
2021-11-29 12:09:47.669964: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Next region of size 1470208256
2021-11-29 12:09:47.670074: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac0000 of size 256 next 1
2021-11-29 12:09:47.670195: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac0100 of size 1280 next 2
2021-11-29 12:09:47.670307: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac0600 of size 256 next 3
2021-11-29 12:09:47.670422: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac0700 of size 256 next 4
2021-11-29 12:09:47.670530: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac0800 of size 256 next 5
2021-11-29 12:09:47.670631: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac0900 of size 256 next 6
2021-11-29 12:09:47.670739: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac0a00 of size 256 next 9
2021-11-29 12:09:47.670827: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac0b00 of size 256 next 10
2021-11-29 12:09:47.670910: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac0c00 of size 256 next 11
2021-11-29 12:09:47.670992: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac0d00 of size 256 next 14
2021-11-29 12:09:47.671075: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac0e00 of size 256 next 15
2021-11-29 12:09:47.671186: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac0f00 of size 256 next 16
2021-11-29 12:09:47.671293: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac1000 of size 256 next 7
2021-11-29 12:09:47.671395: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac1100 of size 1024 next 8
2021-11-29 12:09:47.671511: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac1500 of size 256 next 19
2021-11-29 12:09:47.671628: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac1600 of size 256 next 20
2021-11-29 12:09:47.671738: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac1700 of size 256 next 23
2021-11-29 12:09:47.671900: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac1800 of size 256 next 24
2021-11-29 12:09:47.672014: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac1900 of size 256 next 25
2021-11-29 12:09:47.672130: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac1a00 of size 256 next 26
2021-11-29 12:09:47.672231: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac1b00 of size 256 next 29
2021-11-29 12:09:47.672337: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac1c00 of size 256 next 30
2021-11-29 12:09:47.672445: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac1d00 of size 256 next 31
2021-11-29 12:09:47.672556: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac1e00 of size 256 next 32
2021-11-29 12:09:47.672670: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac1f00 of size 256 next 27
2021-11-29 12:09:47.672771: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac2000 of size 768 next 28
2021-11-29 12:09:47.672875: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac2300 of size 256 next 33
2021-11-29 12:09:47.672970: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac2400 of size 256 next 34
2021-11-29 12:09:47.673066: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac2500 of size 256 next 35
2021-11-29 12:09:47.673177: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac2600 of size 1024 next 36
2021-11-29 12:09:47.673286: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac2a00 of size 256 next 37
2021-11-29 12:09:47.673399: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac2b00 of size 256 next 39
2021-11-29 12:09:47.673498: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac2c00 of size 256 next 40
2021-11-29 12:09:47.673600: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac2d00 of size 256 next 42
2021-11-29 12:09:47.673697: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac2e00 of size 768 next 43
2021-11-29 12:09:47.673796: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac3100 of size 256 next 44
2021-11-29 12:09:47.673899: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac3200 of size 1792 next 13
2021-11-29 12:09:47.674002: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac3900 of size 4608 next 12
2021-11-29 12:09:47.674102: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac4b00 of size 4608 next 38
2021-11-29 12:09:47.674230: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ac5d00 of size 32256 next 18
2021-11-29 12:09:47.674351: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00acdb00 of size 18432 next 17
2021-11-29 12:09:47.674461: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b00ad2300 of size 40140800 next 41
2021-11-29 12:09:47.674565: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0311a300 of size 256 next 45
2021-11-29 12:09:47.674671: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0311a400 of size 4608 next 46
2021-11-29 12:09:47.674779: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0311b600 of size 256 next 47
2021-11-29 12:09:47.674890: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0311b700 of size 18432 next 48
2021-11-29 12:09:47.674996: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0311ff00 of size 256 next 49
2021-11-29 12:09:47.675096: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03120000 of size 256 next 51
2021-11-29 12:09:47.675216: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03120100 of size 768 next 52
2021-11-29 12:09:47.675318: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03120400 of size 256 next 53
2021-11-29 12:09:47.675412: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03120500 of size 256 next 54
2021-11-29 12:09:47.675512: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03120600 of size 256 next 55
2021-11-29 12:09:47.675613: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03120700 of size 256 next 56
2021-11-29 12:09:47.675720: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03120800 of size 256 next 57
2021-11-29 12:09:47.675819: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03120900 of size 256 next 58
2021-11-29 12:09:47.675910: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03120a00 of size 256 next 59
2021-11-29 12:09:47.676003: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03120b00 of size 256 next 60
2021-11-29 12:09:47.676097: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03120c00 of size 256 next 72
2021-11-29 12:09:47.676219: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03120d00 of size 256 next 76
2021-11-29 12:09:47.676333: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03120e00 of size 256 next 64
2021-11-29 12:09:47.676446: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03120f00 of size 256 next 80
2021-11-29 12:09:47.676552: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b03121000 of size 512 next 61
2021-11-29 12:09:47.676663: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03121200 of size 256 next 83
2021-11-29 12:09:47.676779: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03121300 of size 256 next 74
2021-11-29 12:09:47.676883: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b03121400 of size 512 next 66
2021-11-29 12:09:47.676993: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03121600 of size 256 next 62
2021-11-29 12:09:47.677098: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b03121700 of size 4608 next 63
2021-11-29 12:09:47.677215: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b03122900 of size 40106496 next 22
2021-11-29 12:09:47.677326: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b05762300 of size 40140800 next 21
2021-11-29 12:09:47.677438: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b07daa300 of size 40140800 next 50
2021-11-29 12:09:47.677551: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b0a3f2300 of size 127401984 next 68
2021-11-29 12:09:47.677668: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b11d72300 of size 127401984 next 75
2021-11-29 12:09:47.677765: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b196f2300 of size 337383424 next 65
2021-11-29 12:09:47.677880: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b2d8b3300 of size 308004864 next 73
2021-11-29 12:09:47.677998: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b3fe6fb00 of size 256 next 79
2021-11-29 12:09:47.678105: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] Free at b3fe6fc00 of size 250694656 next 70
2021-11-29 12:09:47.678221: I tensorflow/core/common_runtime/bfc_allocator.cc:1066] InUse at b4ed84800 of size 158683392 next 18446744073709551615
2021-11-29 12:09:47.678337: I tensorflow/core/common_runtime/bfc_allocator.cc:1071] Summary of in-use Chunks by size:
2021-11-29 12:09:47.678464: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 51 Chunks of size 256 totalling 12.8KiB
2021-11-29 12:09:47.678579: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 3 Chunks of size 768 totalling 2.2KiB
2021-11-29 12:09:47.678682: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 1024 totalling 2.0KiB
2021-11-29 12:09:47.678784: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1280 totalling 1.2KiB
2021-11-29 12:09:47.678903: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 1792 totalling 1.8KiB
2021-11-29 12:09:47.679003: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 4 Chunks of size 4608 totalling 18.0KiB
2021-11-29 12:09:47.679110: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 18432 totalling 36.0KiB
2021-11-29 12:09:47.679223: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 32256 totalling 31.5KiB
2021-11-29 12:09:47.679344: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 3 Chunks of size 40140800 totalling 114.84MiB
2021-11-29 12:09:47.679508: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 2 Chunks of size 127401984 totalling 243.00MiB
2021-11-29 12:09:47.679620: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 158683392 totalling 151.33MiB
2021-11-29 12:09:47.679732: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] 1 Chunks of size 337383424 totalling 321.75MiB
2021-11-29 12:09:47.679848: I tensorflow/core/common_runtime/bfc_allocator.cc:1078] Sum Total of in-use chunks: 831.03MiB
2021-11-29 12:09:47.679969: I tensorflow/core/common_runtime/bfc_allocator.cc:1080] total_region_allocated_bytes_: 1470208256 memory_limit_: 1470208411 available bytes: 155 curr_region_allocation_bytes_: 2940417024
2021-11-29 12:09:47.680128: I tensorflow/core/common_runtime/bfc_allocator.cc:1086] Stats:
Limit: 1470208411
InUse: 871401216
MaxInUse: 1470208000
NumAllocs: 796
MaxAllocSize: 402397184
Reserved: 0
PeakReserved: 0
LargestFreeBlock: 0
2021-11-29 12:09:47.680737: W tensorflow/core/common_runtime/bfc_allocator.cc:474] ***__************xx*********************************____________________*________________******xxxxx
2021-11-29 12:09:47.680863: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at pooling_ops_common.cc:458 : RESOURCE_EXHAUSTED: OOM when allocating tensor with shape[32,8,574,574] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "d:\dataset\recognition.py", line 83, in <module>
modelTrain(3, 'D:/dataset/training2/training/', 'D:/dataset/training2/validation/', "modelTest", 0.0005, 30, 6, 192*mult, 192*mult, 3)
File "d:\dataset\recognition.py", line 72, in modelTrain
history = model.fit(train_generator,
File "C:\Users\kamil\.conda\envs\tensorflow\lib\site-packages\keras\utils\traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
File "C:\Users\kamil\.conda\envs\tensorflow\lib\site-packages\tensorflow\python\eager\execute.py", line 58, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[32,8,574,574] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[node gradient_tape/sequential/max_pooling2d/MaxPool/MaxPoolGrad
(defined at C:\Users\kamil\.conda\envs\tensorflow\lib\site-packages\keras\optimizer_v2\optimizer_v2.py:464)
]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
[Op:__inference_train_function_909]
Errors may have originated from an input operation.
Input Source operations connected to node gradient_tape/sequential/max_pooling2d/MaxPool/MaxPoolGrad:
In[0] sequential/conv2d/Relu (defined at C:\Users\kamil\.conda\envs\tensorflow\lib\site-packages\keras\backend.py:4867)
In[1] sequential/max_pooling2d/MaxPool (defined at C:\Users\kamil\.conda\envs\tensorflow\lib\site-packages\keras\layers\pooling.py:357)
In[2] gradient_tape/sequential/conv2d_1/Conv2D/Conv2DBackpropInput:
Operation defined at: (most recent call last)
>>> File "d:\dataset\recognition.py", line 83, in <module>
>>> modelTrain(3, 'D:/dataset/training2/training/', 'D:/dataset/training2/validation/', "modelTest", 0.0005, 30, 6, 192*mult, 192*mult, 3)
>>>
>>> File "d:\dataset\recognition.py", line 72, in modelTrain
>>> history = model.fit(train_generator,
>>>
>>> File "C:\Users\kamil\.conda\envs\tensorflow\lib\site-packages\keras\utils\traceback_utils.py", line 64, in error_handler
>>> return fn(*args, **kwargs)
>>>
>>> File "C:\Users\kamil\.conda\envs\tensorflow\lib\site-packages\keras\engine\training.py", line 1216, in fit
>>> tmp_logs = self.train_function(iterator)
>>>
>>> File "C:\Users\kamil\.conda\envs\tensorflow\lib\site-packages\keras\engine\training.py", line 878, in train_function
>>> return step_function(self, iterator)
>>>
>>> File "C:\Users\kamil\.conda\envs\tensorflow\lib\site-packages\keras\engine\training.py", line 867, in step_function
>>> outputs = model.distribute_strategy.run(run_step, args=(data,))
>>>
>>> File "C:\Users\kamil\.conda\envs\tensorflow\lib\site-packages\keras\engine\training.py", line 860, in run_step
>>> outputs = model.train_step(data)
>>>
>>> File "C:\Users\kamil\.conda\envs\tensorflow\lib\site-packages\keras\engine\training.py", line 816, in train_step
>>> self.optimizer.minimize(loss, self.trainable_variables, tape=tape)
>>>
>>> File "C:\Users\kamil\.conda\envs\tensorflow\lib\site-packages\keras\optimizer_v2\optimizer_v2.py", line 530, in minimize
>>> grads_and_vars = self._compute_gradients(
>>>
>>> File "C:\Users\kamil\.conda\envs\tensorflow\lib\site-packages\keras\optimizer_v2\optimizer_v2.py", line 583, in _compute_gradients
>>> grads_and_vars = self._get_gradients(tape, loss, var_list, grad_loss)
>>>
>>> File "C:\Users\kamil\.conda\envs\tensorflow\lib\site-packages\keras\optimizer_v2\optimizer_v2.py", line 464, in _get_gradients
>>> grads = tape.gradient(loss, var_list, grad_loss)
>>>
2021-11-29 12:09:48.275244: W tensorflow/core/kernels/data/generator_dataset_op.cc:107] Error occurred when finalizing GeneratorDataset iterator: FAILED_PRECONDITION: Python interpreter state is not initialized. The process may be terminated.
[[{{node PyFunc}}]]
Заранее спасибо!