ggml : extend ggml_can_fuse to work with non-sequential nodes

[ggerganov]

In some cases, we might want to fuse ops that are not sequential in the graph. For example, when there are intermediate view ops in-between the fusable ops.

Sample usage: #16102

[jeffbolznv]

Fwiw, with the sorting I have in the vulkan backend, I try to separate empty nodes from ninety nodes, which makes it less important to be able to skip.

[jeffbolznv]

Why add num_ops here? Also, should be >=, i think.

[jeffbolznv]

I'm not totally clear on how you foresee this function being used. I assume the caller needs to look through nodes and find the ops its interested in (skipping empty nodes), so I dont think it'll be able to use an initializer list for the node idxs.

[ggerganov]

The idea is to create a list of indices that excludes the empty nodes in idxs. And then use this like so:

    // non-empty node indices
    std::vector<int> idxs;

    bool can_fuse(int i0, int i1, std::initializer_list<enum ggml_op> ops) const {
        assert(use_fusion);
        assert(i0 >= 0 && i0 < n_nodes());
        assert(i1 >= 0 && i1 < n_nodes());
        assert(ops.size() == 2);

        return ggml_can_fuse(gf, { idxs[i0], idxs[i1] }, ops);
    }

Btw, I really need an API to tell if 2 ops are fusable - don't think there is any benefit to support more than a pair of ops. The assumption is that if we can fuse N ops, then for sure we can fuse N-1 ops. So at step N we only need to check if ops N and N+1 are fusable - no need to iterate again over all previous ops.

[jeffbolznv]

If you have to rebuild the initializer_list from the vector, why not just pass an int* instead?

[ggerganov]

Right, I can do that. Removed the extra overload.

[ggerganov]

Fwiw, with the sorting I have in the vulkan backend, I try to separate empty nodes from ninety nodes, which makes it less important to be able to skip.

Yes, I noticed that works, but I'm not really confident if this won't break any assumption in the future. Specifically, this case:

c0 = ADD(a, b)
f0 = VIEW(...)
c1 = ADD(c0, f0)
f1 = VIEW(...)
c2 = ADD(c1, f1)
... etc

If I just move all the adds together and put the views after them, everything seems to work. But I think I also saw that depending on the order of the views, the allocr can do different things. For example, at some point I was thinking, let's make the graph_optimize simply put all empty nodes at the end of the graph first and then work with a truncated graph with fewer nodes. This also runs correctly, but I did notice a different allocation pattern of the nodes. Hence I'm having some doubts about the implications of changing the order of the views (or any other empty node).