WIP: add read npy for ncnn2table #5930
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
nihui
requested changes
Mar 6, 2025
tools/quantize/ncnn2table.cpp
Outdated
Comment on lines
226
to
233
| return ncnn::Mat(shape[0], (void*)(d.data.data())).clone(); | ||
| case 2: | ||
| return ncnn::Mat(shape[0], shape[1], (void*)(d.data.data())).clone(); | ||
| case 3: | ||
| return ncnn::Mat(shape[0], shape[1], shape[2], (void*)(d.data.data())).clone(); | ||
| case 4: | ||
| return ncnn::Mat(shape[0], shape[1], shape[2], shape[3], (void*)(d.data.data())).clone(); | ||
| default: |
There was a problem hiding this comment.
3d 4d 可能存在 channel gap
先从1d的构造，然后reshape到3d，再clone
|
|
||
| inline ncnn::Mat read_npy(const std::vector<int>& shape, const std::string& npypath) | ||
| { | ||
| npy::npy_data<float> d = npy::read_npy<float>(npypath); |
There was a problem hiding this comment.
read_npy 可能会抛出异常，需要 catch 处理下，提示错误的文件是哪个，然后跳过当前文件？
There was a problem hiding this comment.
不建议处理异常，尤其是为了跳过错误文件，因为用npy的情况下，使用的数据一般是训练的数据，基本都是经过清洗和前处理的，几乎不可能出现个别错误的npy文件，只有可能全部出错，这个和自己收集的image文件不一样。
There was a problem hiding this comment.
不建议处理异常，尤其是为了跳过错误文件，因为用npy的情况下，使用的数据一般是训练的数据，基本都是经过清洗和前处理的，几乎不可能出现个别错误的npy文件，只有可能全部出错，这个和自己收集的image文件不一样。
那么 fprintf(stderr 输出npy错误信息，然后直接 exit ？
|
判断传入shape和npy的shape是否匹配还是直接判断两个的total size，如果是判断shape匹配，就稍微有点严格，要不要check这个？ |
严格判断shape匹配，可变shape没法统计的... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
ncnn2table增加读npy功能
命令增加一个参数type，type为0时和原来一样，可省略
type为1时，以npy格式读文件，读npy引用(https://github.com/llohse/libnpy)实现。
单个input blob测试通过，多个未测试（没资源）