It would not be very difficult to update Glm to handle categorical columns natively: i.e. run them through LabelEncoder then pass that to a torch.nn.Embedding layer instead of torch.nn.Linear.
This would improve performance immensely for high-cardinality categorical features (esp. since sparse-matmul seems to be pretty slow).