Replication fails with error "attempt to compare number with table"

[nikhilo]

Trying to replicate from EC2 redis to AWS MemoryDB. It replicated fine for some time, but was failing intermittently with a timeout. To try something different, I tried running with --type STRUCT and found the following error almost immediately after the replication started.

Now even when I remove the --type STRUCT parameter OR add --type DUMP parameter, it fails only a few seconds after starting. It gives the same error that I got when I tried the --type STRUCT

Caused by: io.lettuce.core.RedisCommandExecutionException: ERR Error running script (call to f_08cad432657f5b355eee00d81d5c0031331993c1): @user_script:46: user_script:46: attempt to compare number with table

Here is the full error log

root@ip-10-120-5-154:~# /usr/share/riot-4.2.2/bin/riot replicate redis://redis-11372.uat.company.com:11372   rediss://clustercfg.xmpp-groups-memorydb-uat.xxxxx.memorydb.region.amazonaws.com:6379   --target-cluster --mode live --show-diffs --ttl-tolerance 1000000 --progress log --info
[main] INFO com.redis.riot.Replicate - Creating source Redis context with redis://redis-11372.uat.company.com:11372 SourceRedisArgs(username=null, password=null, timeout=60.00s, tls=false, insecure=false, clientName=null, cluster=false, protocolVersion=RESP2, poolSize=8, readFrom=UPSTREAM, keystore=null, keystorePassword=null, truststore=null, truststorePassword=null, keyCert=null, key=null, keyPassword=null, trustedCerts=null) {}
[main] INFO com.redis.riot.Replicate - Creating target Redis context with rediss://clustercfg.xmpp-groups-memorydb-uat.xxxxx.memorydb.region.amazonaws.com:6379 TargetRedisArgs(username=null, password=null, timeout=60.00s, tls=false, insecure=false, clientName=null, cluster=true, protocolVersion=RESP2, poolSize=8, readFrom=UPSTREAM, keystore=null, keystorePassword=null, truststore=null, truststorePassword=null, keyCert=null, key=null, keyPassword=null, trustedCerts=null) {}
[main] INFO com.redis.riot.Replicate - Creating Redis dump reader
[main] INFO com.redis.riot.Replicate - Configuring RedisItemReader with RedisReaderArgs(mode=LIVE, keyPattern=null, keyType=null, scanCount=1000, queueCapacity=10000, threads=1, chunkSize=50, memUsageLimit=null, memUsageSamples=5, flushInterval=50, idleTimeout=null, eventQueueCapacity=10000, retryLimit=0, skipLimit=0, keyFilterArgs=KeyFilterArgs(includes=null, excludes=null, slots=null), pollTimeout=100)
[main] INFO com.redis.riot.Replicate - Creating Redis dump writer
[main] INFO com.redis.riot.Replicate - Configuring target Redis writer with RedisWriterArgs(multiExec=false, waitReplicas=0, waitTimeout=PT1S, merge=false)
[main] INFO com.redis.riot.Replicate - Retrieved config notify-keyspace-events: AKE
[main] INFO com.redis.riot.Replicate - Configuring export step with live true, flushInterval PT0.05S, idleTimeout PT2562047788015H12M55.807S
[main] INFO com.redis.riot.Replicate - Creating SpEL evaluation context with EvaluationContextArgs(varExpressions={}, dateFormat=yyyy-MM-dd'T'HH:mm:ss.SSSZ, numberFormat=#,###.##, vars={})
[main] INFO com.redis.riot.Replicate - Creating processor with KeyValueProcessorArgs(keyExpression=null, typeExpression=null, ttlExpression=null, propagateTtl=true, propagateIds=true, prune=false)
[main] INFO com.redis.riot.Replicate - Creating Redis quick compare reader
[main] INFO com.redis.riot.Replicate - Configuring RedisItemReader with RedisReaderArgs(mode=LIVE, keyPattern=null, keyType=null, scanCount=1000, queueCapacity=10000, threads=1, chunkSize=50, memUsageLimit=null, memUsageSamples=5, flushInterval=50, idleTimeout=null, eventQueueCapacity=10000, retryLimit=0, skipLimit=0, keyFilterArgs=KeyFilterArgs(includes=null, excludes=null, slots=null), pollTimeout=100)
[main] INFO com.redis.riot.Replicate - Creating Redis quick compare reader
[main] INFO com.redis.riot.Replicate - Creating KeyComparator with ttlTolerance=PT16M40S ignoreStreamMessageId=false
[main] INFO com.redis.riot.Replicate - Creating SpEL evaluation context with EvaluationContextArgs(varExpressions={}, dateFormat=yyyy-MM-dd'T'HH:mm:ss.SSSZ, numberFormat=#,###.##, vars={})
[main] INFO com.redis.riot.Replicate - Creating processor with KeyValueProcessorArgs(keyExpression=null, typeExpression=null, ttlExpression=null, propagateTtl=true, propagateIds=true, prune=false)
[main] INFO com.redis.riot.Replicate - Creating SpEL evaluation context with EvaluationContextArgs(varExpressions={}, dateFormat=yyyy-MM-dd'T'HH:mm:ss.SSSZ, numberFormat=#,###.##, vars={})
[main] INFO com.redis.riot.Replicate - Creating processor with KeyValueProcessorArgs(keyExpression=null, typeExpression=null, ttlExpression=null, propagateTtl=true, propagateIds=true, prune=false)
[main] INFO com.redis.riot.Replicate - Adding key diff logger
[main] INFO com.redis.riot.Replicate - Creating Step(name=step, reader=com.redis.spring.batch.item.redis.RedisItemReader@309cedb6, writer=com.redis.spring.batch.item.redis.RedisItemWriter@3b95a6db, taskName=Scanning/Listening, statusMessageSupplier=com.redis.riot.core.Step$$Lambda/0x00000008013bfd00@3bf54172, maxItemCountSupplier=com.redis.riot.core.Step$$Lambda/0x0000000801424000@2c9a6717, processor=com.redis.riot.KeyValueFilter@7b3cde6f, executionListeners=[], readListeners=[], writeListeners=[], live=true, flushInterval=PT0.05S, idleTimeout=PT2562047788015H12M55.807S, skip=[], noSkip=[], retry=[], noRetry=[])
[main] INFO com.redis.riot.Replicate - Creating step replicate-step with chunk size 50
[main] INFO com.redis.riot.Replicate - Creating flushing step with flush interval PT0.05S and idle timeout PT2562047788015H12M55.807S
[main] INFO com.redis.riot.Replicate - Skipping fault-tolerance for step step
[main] INFO com.redis.riot.Replicate - Creating Step(name=compare, reader=com.redis.spring.batch.item.redis.reader.KeyComparisonItemReader@65afeb6d, writer=com.redis.riot.CompareStatusItemWriter@6723610b, taskName=Comparing, statusMessageSupplier=com.redis.riot.AbstractReplicateCommand$$Lambda/0x0000000801432218@5ba03c82, maxItemCountSupplier=com.redis.spring.batch.item.redis.reader.RedisScanSizeEstimator@60df7989, processor=null, executionListeners=[com.redis.riot.CompareStepListener@630390b9], readListeners=[], writeListeners=[com.redis.riot.CompareLoggingWriteListener@6e03db1f], live=false, flushInterval=PT0.05S, idleTimeout=PT2562047788015H12M55.807S, skip=[], noSkip=[], retry=[], noRetry=[])
[main] INFO com.redis.riot.Replicate - Creating step replicate-compare with chunk size 50
[main] INFO com.redis.riot.Replicate - Skipping fault-tolerance for step compare
[main] INFO org.springframework.batch.core.launch.support.TaskExecutorJobLauncher - Job: [SimpleJob: [name=replicate]] launched with the following parameters: [{}]
[main] INFO org.springframework.batch.core.job.SimpleStepHandler - Executing step: [replicate-step]
[main] INFO com.redis.spring.batch.item.redis.RedisItemReader - Creating queue with capacity 10,000
Scanning/Listening  ? % [=                                                                                                                                                                                                             ] 0/? (0:00:00 / ?) ?/s
[SimpleAsyncTaskExecutor-1] INFO org.springframework.batch.core.launch.support.TaskExecutorJobLauncher - Job: [SimpleJob: [name=replicate-step-reader]] launched with the following parameters: [{}]
[SimpleAsyncTaskExecutor-1] INFO org.springframework.batch.core.job.SimpleStepHandler - Executing step: [replicate-step-reader]
[SimpleAsyncTaskExecutor-1] INFO com.redis.spring.batch.item.redis.reader.KeyScanEventItemReader - Creating queue with capacity 10,000
[SimpleAsyncTaskExecutor-1] INFO com.redis.spring.batch.item.redis.reader.RedisPubSubHandler - Establishing pub/sub connection to Redis
[SimpleAsyncTaskExecutor-1] INFO com.redis.spring.batch.item.redis.reader.RedisPubSubHandler - Subscribing to pattern __keyevent@0__:*
Scanning/Listening  ? % [                                                                                                                                                                                       =             ] 22050/? (0:00:04 / ?) 5512.5/s
[SimpleAsyncTaskExecutor-1] ERROR org.springframework.batch.core.step.AbstractStep - Encountered an error executing step replicate-step-reader in job replicate-step-reader
org.springframework.batch.core.step.skip.SkipLimitExceededException: Skip limit of '0' exceeded
	at org.springframework.batch.core.step.skip.LimitCheckingItemSkipPolicy.shouldSkip(LimitCheckingItemSkipPolicy.java:125)
	at org.springframework.batch.core.step.skip.ExceptionClassifierSkipPolicy.shouldSkip(ExceptionClassifierSkipPolicy.java:65)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.shouldSkip(FaultTolerantChunkProcessor.java:486)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.checkSkipPolicy(FaultTolerantChunkProcessor.java:517)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.scan(FaultTolerantChunkProcessor.java:599)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.lambda$write$4(FaultTolerantChunkProcessor.java:406)
	at org.springframework.retry.support.RetryTemplate.handleRetryExhausted(RetryTemplate.java:573)
	at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:418)
	at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:276)
	at org.springframework.batch.core.step.item.BatchRetryTemplate.execute(BatchRetryTemplate.java:216)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.write(FaultTolerantChunkProcessor.java:414)
	at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:227)
	at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:75)
	at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:383)
	at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:307)
	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
	at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:250)
	at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:82)
	at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:369)
	at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:206)
	at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:140)
	at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:235)
	at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:230)
	at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:153)
	at org.springframework.batch.core.job.AbstractJob.handleStep(AbstractJob.java:408)
	at org.springframework.batch.core.job.SimpleJob.doExecute(SimpleJob.java:127)
	at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:307)
	at org.springframework.batch.core.launch.support.TaskExecutorJobLauncher$1.run(TaskExecutorJobLauncher.java:155)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.util.concurrent.ExecutionException: io.lettuce.core.RedisCommandExecutionException: ERR Error running script (call to f_08cad432657f5b355eee00d81d5c0031331993c1): @user_script:46: user_script:46: attempt to compare number with table
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
	at com.redis.lettucemod.RedisModulesUtils.getAll(RedisModulesUtils.java:355)
	at com.redis.spring.batch.item.redis.common.OperationExecutor.execute(OperationExecutor.java:123)
	at com.redis.spring.batch.item.redis.common.OperationExecutor.process(OperationExecutor.java:102)
	at com.redis.spring.batch.item.redis.common.OperationExecutor.process(OperationExecutor.java:34)
	at com.redis.spring.batch.item.ProcessingItemWriter.write(ProcessingItemWriter.java:57)
	at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:203)
	at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.scan(FaultTolerantChunkProcessor.java:576)
	... 24 more
Caused by: io.lettuce.core.RedisCommandExecutionException: ERR Error running script (call to f_08cad432657f5b355eee00d81d5c0031331993c1): @user_script:46: user_script:46: attempt to compare number with table
	at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:151)
	at io.lettuce.core.internal.ExceptionFactory.createExecutionException(ExceptionFactory.java:120)
	at io.lettuce.core.protocol.AsyncCommand.completeResult(AsyncCommand.java:124)
	at io.lettuce.core.protocol.AsyncCommand.complete(AsyncCommand.java:115)
	at io.lettuce.core.protocol.CommandWrapper.complete(CommandWrapper.java:67)
	at io.lettuce.core.protocol.CommandHandler.complete(CommandHandler.java:762)
	at io.lettuce.core.protocol.CommandHandler.decode(CommandHandler.java:697)
	at io.lettuce.core.protocol.CommandHandler.channelRead(CommandHandler.java:614)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	... 1 more
[SimpleAsyncTaskExecutor-1] INFO org.springframework.batch.core.step.AbstractStep - Step: [replicate-step-reader] executed in 4s372ms
[SimpleAsyncTaskExecutor-1] INFO com.redis.spring.batch.item.redis.reader.RedisPubSubHandler - Unsubscribing from pattern __keyevent@0__:*
[SimpleAsyncTaskExecutor-1] INFO com.redis.spring.batch.item.redis.reader.RedisPubSubHandler - Closing pub/sub connection
Scanning/Listening  ? % [                                  =                                                                                                                                                                  ] 35100/? (0:00:06 / ?) 5850.0/s
^Croot@ip-10-120-5-154:~#

The weirdest thing is, it was running fine for a while before I tried the --type STRUCT. And now started failing with this error which I never saw before.

[nikhilo]

Tried with riot version 4.2.3 and 4.2.2