bias not found in checkpoint

[Banhalmi]

model: models/pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-592000
gpu_devices = []
controller = '/device:CPU:0'
windows 8.1
python 3.6
tensorflow 1.13
running: pwcnet_predict_from_img_pairs.py

full error output:
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key pwcnet/ctxt/dc_conv31/bias not found in checkpoint
[[node save/RestoreV2 (defined at C:\PROJECTS\SASMOB - hídas projekt\optical_flow\tfoptflow-master\tfoptflow\model_base.py:119) ]]

Caused by op 'save/RestoreV2', defined at:
File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\ptvsd_launcher.py", line 89, in
vspd.debug(filename, port_num, debug_id, debug_options, run_as)
File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\ptvsd\debugger.py", line 2631, in debug
exec_file(file, globals_obj)
File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\ptvsd\util.py", line 119, in exec_file
exec_code(code, file, global_variables)
File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\ptvsd\util.py", line 95, in exec_code
exec(code_obj, global_variables)
File "C:\PROJECTS\SASMOB - hídas projekt\optical_flow\tfoptflow-master\tfoptflow\pwcnet_predict_from_img_pairs.py", line 58, in
nn = ModelPWCNet(mode='test', options=nn_opts)
File "C:\PROJECTS\SASMOB - hídas projekt\optical_flow\tfoptflow-master\tfoptflow\model_pwcnet.py", line 231, in init
super().init(name, mode, session, options)
File "C:\PROJECTS\SASMOB - hídas projekt\optical_flow\tfoptflow-master\tfoptflow\model_base.py", line 66, in init
self.build_graph()
File "C:\PROJECTS\SASMOB - hídas projekt\optical_flow\tfoptflow-master\tfoptflow\model_base.py", line 266, in build_graph
self.init_saver()
File "C:\PROJECTS\SASMOB - hídas projekt\optical_flow\tfoptflow-master\tfoptflow\model_base.py", line 119, in init_saver
self.saver = tf.train.Saver()
File "C:\Users\BAndras\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py", line 832, in init
self.build()
File "C:\Users\BAndras\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py", line 844, in build
self._build(self._filename, build_save=True, build_restore=True)
File "C:\Users\BAndras\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py", line 881, in _build
build_save=build_save, build_restore=build_restore)
File "C:\Users\BAndras\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py", line 513, in _build_internal
restore_sequentially, reshape)
File "C:\Users\BAndras\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py", line 332, in _AddRestoreOps
restore_sequentially)
File "C:\Users\BAndras\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py", line 580, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "C:\Users\BAndras\Anaconda3\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 1655, in restore_v2
name=name)
File "C:\Users\BAndras\Anaconda3\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "C:\Users\BAndras\Anaconda3\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
return func(*kwargs)
File "C:\Users\BAndras\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 3300, in create_op
op_def=op_def)
File "C:\Users\BAndras\Anaconda3\lib\site-packages\tensorflow\python\framework\ops.py", line 1801, in init
self._traceback = tf_stack.extract_stack()

NotFoundError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Key pwcnet/ctxt/dc_conv31/bias not found in checkpoint
[[node save/RestoreV2 (defined at C:\PROJECTS\SASMOB - hídas projekt\optical_flow\tfoptflow-master\tfoptflow\model_base.py:119) ]]

output until error:
C:\Users\BAndras\Anaconda3\lib\site-packages\h5py_init_.py:34: FutureWarning:
Conversion of the second argument of issubdtype from float to np.floating i
s deprecated. In future, it will be treated as np.float64 == np.dtype(float).ty pe.
from ._conv import register_converters as _register_converters
Building model...
WARNING:tensorflow:From C:\PROJECTS\SASMOB - hídas projekt\optical_flow\tfoptflo
w-master\tfoptflow\model_pwcnet.py:1094: conv2d (from tensorflow.python.layers.c
onvolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
WARNING:tensorflow:From C:\Users\BAndras\Anaconda3\lib\site-packages\tensorflow
python\framework\op_def_library.py:263: colocate_with (from tensorflow.python.fr
amework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From C:\PROJECTS\SASMOB - hídas projekt\optical_flow\tfoptflo
w-master\tfoptflow\model_pwcnet.py:1221: conv2d_transpose (from tensorflow.pytho
n.layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.conv2d_transpose instead.
... model built.
Loading model checkpoint c:/PROJECTS/SASMOB - hídas projekt/optical_flow/tfoptfl
ow-master/tfoptflow/models/pwcnet-sm-6-2-multisteps-chairsthingsmix/pwcnet.ckpt-
592000 for eval or testing...

WARNING:tensorflow:From C:\Users\BAndras\Anaconda3\lib\site-packages\tensorflow
python\training\saver.py:1266: checkpoint_exists (from tensorflow.python.trainin
g.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from c:/PROJECTS/SASMOB - hídas projekt/opt
ical_flow/tfoptflow-master/tfoptflow/models/pwcnet-sm-6-2-multisteps-chairsthing
smix/pwcnet.ckpt-592000
2019-03-28 12:13:28.455200: W tensorflow/core/framework/op_kernel.cc:1401] OP_RE
QUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key pwcnet/ctxt/dc_conv
31/bias not found in checkpoint

[apxlwl]

@Banhalmi Have you solved the problem? I met the same problem.

[dagongji10]

@Banhalmi @wlguan When I use the pretrained model sm，I got the same error. So I change the model to lg, it can run normally. Maybr there is something wrong in sm model.

Can I ask you the speed when run lg model? I use the image with (436, 1024) , each pair cost about 0.1 s; when image reduce to (256, 340), it cost 8.5s for 150 pairs. How about yours?

[liyunfei1994]

@Banhalmi @philferriere Have you solved this problem? I met the same problem when I restore the model. What is strange is that the model can run and be restored normally a few months ago, but now it can not be restored.

[jeffbaena]

Same problem here, I am using tensorflow 1.10.0 installed thorugh conda.

this might be useful, resnet 101 has the same problem, some people solved by deleting the folder and saving on clean folder tensorflow/models#5003 (comment) .EDIT: I tried and it seems it is not the case for this small model.

[yacaeh]

Well I've found solution, but seems like lg model is better for me.
it took 0.08 sec for lg model with [384, 512] size, 0.06 for sm model with the same size, but the inference result was not satisfying.

If you still want to use sm model, you have to change the following nn_opts

nn_opts['use_dense_cx'] = False,
nn_opts['use_res_cx'] = False

It should be True for lg model, and False for sm model