UserService.GetAllInGroup fails when there are a lot of users in the group

[LennardF1989]

Which Umbraco version are you using? (Please write the exact version, example: 10.1.0)

13.6.0

Bug summary

Whenever a group reaches a certain amount of users associated, the UserService.GetAllInGroup will fail with the following message:

The incoming request has too many parameters. The server supports a maximum of 2100 parameters. Reduce the number of parameters and resend the request.

Specifically, it will fail in the UserRepository > PerformGetReferencesDtos:

        // get users2groups

        Sql<ISqlContext> sql = SqlContext.Sql()
            .Select<User2UserGroupDto>()
            .From<User2UserGroupDto>()
            .WhereIn<User2UserGroupDto>(x => x.UserId, userIds);

        List<User2UserGroupDto>? user2Groups = Database.Fetch<User2UserGroupDto>(sql);

Potentially fix is either batching this, or not using a prepared statement and instead write a raw query.

Specifics

No response

Steps to reproduce

Have a lot of users assigned to a single group.

Expected result / actual result

No response

[github-actions]

Hi there @LennardF1989!

Firstly, a big thank you for raising this issue. Every piece of feedback we receive helps us to make Umbraco better.

We really appreciate your patience while we wait for our team to have a look at this but we wanted to let you know that we see this and share with you the plan for what comes next.

• We'll assess whether this issue relates to something that has already been fixed in a later version of the release that it has been raised for.
• If it's a bug, is it related to a release that we are actively supporting or is it related to a release that's in the end-of-life or security-only phase?
• We'll replicate the issue to ensure that the problem is as described.
• We'll decide whether the behavior is an issue or if the behavior is intended.

We wish we could work with everyone directly and assess your issue immediately but we're in the fortunate position of having lots of contributions to work with and only a few humans who are able to do it. We are making progress though and in the meantime, we will keep you in the loop and let you know when we have any questions.

Thanks, from your friendly Umbraco GitHub bot 🤖 🙂

[LennardF1989]

Additional, if you patch this logic by batching it, code will bottleneck on a different part if you try to view the User Group in Umbraco:

UserMapDefinition > Map

        IEnumerable<IUser> users = _userService.GetAllInGroup(source.Id);
        target.Users = context.MapEnumerable<IUser, UserBasic>(users).WhereNotNull();

First line will succeed pretty much instantly when batched per 1000, it will indefinitely hang on the second line though.

Here is my Harmony patch to test my batch logic.

 public class UserRepositoryPatch
    {
        private const int BATCH_SIZE = 1000;

        private static Type _userRepositoryType;
        private static MethodInfo _performGetReferencedDtosMethod;
        
        public static void Enable(HarmonyLib.Harmony harmony)
        {
            _userRepositoryType = AccessTools.TypeByName("Umbraco.Cms.Infrastructure.Persistence.Repositories.Implement.UserRepository");
            var method = AccessTools.Method(_userRepositoryType, "GetAllInOrNotInGroup");

            harmony.Patch(method, transpiler: new HarmonyMethod(Transpiler));
        }

        private static IEnumerable<CodeInstruction> Transpiler(IEnumerable<CodeInstruction> instructions, ILGenerator il)
        {
            _performGetReferencedDtosMethod = AccessTools.Method(_userRepositoryType, "PerformGetReferencedDtos");
            var batchMethod = AccessTools.Method(typeof(UserRepositoryPatch), nameof(PerformGetReferencedDtosBatch));
            
            List<CodeInstruction> code = [..instructions];

            for (var i = 0; i < code.Count; i++)
            {
                if (code[i].Calls(_performGetReferencedDtosMethod))
                {
                    code[i] = new CodeInstruction(OpCodes.Call, batchMethod);
                }
            }

            return code;
        }
        
        private static void PerformGetReferencedDtosBatch(object instance, List<UserDto> dtos)
        {
            for (var i = 0; i < dtos.Count; i += BATCH_SIZE)
            {
                var batch = dtos
                    .Skip(i)
                    .Take(BATCH_SIZE)
                    .ToList();
                
                _performGetReferencedDtosMethod.Invoke(instance, [batch]);
            }
        }
    }

[bielu]

I don't think it is any new issue, I remember it from v7 and v8 as it was already here, however couldn't find ticket which i opened about it 🤔 the answer at that point was that it is unlikely anone will have that many users groups. If I remember correctly same issue apply to members as well

[LennardF1989]

Unlikely, yet plausible. We have a client where the Backoffice has a specific section for dealers. Every dealer will get automatically added to the - you guessed it - Dealer User Group. At this point, there are 6000 users.

[AndyButland]

If you wanted to take a look at fixing this one @LennardF1989 - the technique I see in the Umbraco repositories for these WhereIn queries where we may go over the maximum parameter count is to use the .InGroupsOf() extension method.

So for example the code after your comment // get users2groups may go from:

sql = SqlContext.Sql()
    .Select<UserGroupDto>(x => x.Id, x => x.Key)
    .From<UserGroupDto>()
    .InnerJoin<User2UserGroupDto>().On<UserGroupDto, User2UserGroupDto>((left, right) => left.Id == right.UserGroupId)
    .WhereIn<User2UserGroupDto>(x => x.UserId, userIds);

List<UserGroupDto>? userGroups = Database.Fetch<UserGroupDto>(sql);

groupKeys = userGroups.Select(x => x.Key).ToList();

To:

foreach (IEnumerable<int> groupedUserIds in userIds.InGroupsOf(Constants.Sql.MaxParameterCount))
{
    sql = SqlContext.Sql()
        .Select<UserGroupDto>(x => x.Id, x => x.Key)
        .From<UserGroupDto>()
        .InnerJoin<User2UserGroupDto>().On<UserGroupDto, User2UserGroupDto>((left, right) => left.Id == right.UserGroupId)
        .WhereIn<User2UserGroupDto>(x => x.UserId, groupedUserIds);

    List<UserGroupDto>? userGroups = Database.Fetch<UserGroupDto>(sql);

    groupKeys.AddRange(userGroups.Select(x => x.Key).ToList());
}

There'll be a bit more too it than that, as I see there are more queries like this within PerformGetReferencedDtos, so likely they need similar updates too.

Maybe we'll look at this at HQ but it's not an immediate priority given I expect this situation is fairly rare, but we'd welcome a PR to v13/contrib if you were minded to try to resolve this (and it seems you have a handy database with 6000 users to test against!). I'll mark this as "up for grabs" having provided some guidance as to how it might be solved.

[github-actions]

Hi @LennardF1989,

We're writing to let you know that we would love some help with this issue. We feel that this issue is ideal to flag for a community member to work on it. Once flagged here, folk looking for issues to work on will know to look at yours. Of course, please feel free work on this yourself ;-). If there are any changes to this status, we'll be sure to let you know.

For more information about issues and states, have a look at this blog post.

Thanks muchly, from your friendly Umbraco GitHub bot :-)