[Voyager] export data status panics

[jlisthood]

Jira Link: DB-13971

Description

Context:

• YB-Voyager Version: GIT_COMMIT_HASH=4f3afcd389cac1236f0051c38215ae360420fb3e, VERSION=1.8.5-rc1
• YB Version: 2024.1.3.0

Issue:
While running yb-voyager data export, I cannot check the status following the guide because I hit a panic, see below.

Logs:

[ssm-user@ip-10-71-251-29 bin]$ sudo -u ec2-user yb-voyager export data status --export-dir /home/ec2-user/migration
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/yugabyte/yb-voyager/yb-voyager/cmd.runExportDataStatusCmd()
        /home/ec2-user/rpmbuild/BUILD/yb-voyager-yb-voyager-v1.8.5-rc1/yb-voyager/cmd/exportDataStatusCommand.go:186 +0x905
github.com/yugabyte/yb-voyager/yb-voyager/cmd.init.func11(0x334ed80?, {0x1e7b330?, 0x4?, 0x1e7b1a0?})
        /home/ec2-user/rpmbuild/BUILD/yb-voyager-yb-voyager-v1.8.5-rc1/yb-voyager/cmd/exportDataStatusCommand.go:69 +0x25b
github.com/spf13/cobra.(*Command).execute(0x334ed80, {0xc000b0e120, 0x2, 0x2})
        /home/ec2-user/go/pkg/mod/github.com/spf13/[email protected]/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x3351e60)
        /home/ec2-user/go/pkg/mod/github.com/spf13/[email protected]/command.go:1044 +0x3a5
github.com/spf13/cobra.(*Command).Execute(...)
        /home/ec2-user/go/pkg/mod/github.com/spf13/[email protected]/command.go:968
github.com/yugabyte/yb-voyager/yb-voyager/cmd.Execute()
        /home/ec2-user/rpmbuild/BUILD/yb-voyager-yb-voyager-v1.8.5-rc1/yb-voyager/cmd/root.go:208 +0x1a
main.main()
        /home/ec2-user/rpmbuild/BUILD/yb-voyager-yb-voyager-v1.8.5-rc1/yb-voyager/main.go:42 +0x45
[ssm-user@ip-10-71-251-29 bin]$

Note: the data export is running correctly:

calculating approx num of rows to export for each table...
Initiating data export.
Data export started.

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

• I confirm this issue does not contain any sensitive information.

[shubham-yb]

Hi @jlisthood , I was trying out this flow and was able to repro this error on my end.

The entire flow goes like this

Running export data status before export data command is run:

yb-voyager export data status -e export-dir/
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xca88f7]

goroutine 1 [running]:
github.com/yugabyte/yb-voyager/yb-voyager/src/metadb.(*MetaDB).GetJsonObject(0x0, 0x0, {0x1e787ce, 0x10}, {0x1b40de0, 0xc0009a7900})
	/home/ubuntu/yb-voyager/yb-voyager/src/metadb/metadataDB.go:289 +0x157
github.com/yugabyte/yb-voyager/yb-voyager/src/metadb.(*MetaDB).GetMigrationStatusRecord(0x0)
	/home/ubuntu/yb-voyager/yb-voyager/src/metadb/migrationStatus.go:66 +0x4c
github.com/yugabyte/yb-voyager/yb-voyager/cmd.checkStreamingMode()
	/home/ubuntu/yb-voyager/yb-voyager/cmd/common.go:626 +0x1e
github.com/yugabyte/yb-voyager/yb-voyager/cmd.init.func11(0x3136da0?, {0x1e65e34?, 0x4?, 0x1e65ca4?})
	/home/ubuntu/yb-voyager/yb-voyager/cmd/exportDataStatusCommand.go:48 +0x25
github.com/spf13/cobra.(*Command).execute(0x3136da0, {0xc0008959a0, 0x2, 0x2})
	/home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x3139e80)
	/home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:1044 +0x3a5
github.com/spf13/cobra.(*Command).Execute(...)
	/home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:968
github.com/yugabyte/yb-voyager/yb-voyager/cmd.Execute()
	/home/ubuntu/yb-voyager/yb-voyager/cmd/root.go:208 +0x1a
main.main()
	/home/ubuntu/yb-voyager/yb-voyager/main.go:42 +0x45

Export data command is run:

yb-voyager export data status -e export-dir/
Export Data Status for SourceDB

Export is said as started. We face this for a brief moment when the internal tool is starting up:

yb-voyager export data status -e export-dir/
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/yugabyte/yb-voyager/yb-voyager/cmd.runExportDataStatusCmd()
	/home/ubuntu/yb-voyager/yb-voyager/cmd/exportDataStatusCommand.go:186 +0x905
github.com/yugabyte/yb-voyager/yb-voyager/cmd.init.func11(0x3136da0?, {0x1e65e34?, 0x4?, 0x1e65ca4?})
	/home/ubuntu/yb-voyager/yb-voyager/cmd/exportDataStatusCommand.go:69 +0x25b
github.com/spf13/cobra.(*Command).execute(0x3136da0, {0xc0008a9a40, 0x2, 0x2})
	/home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0x3139e80)
	/home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:1044 +0x3a5
github.com/spf13/cobra.(*Command).Execute(...)
	/home/ubuntu/go/pkg/mod/github.com/spf13/[email protected]/command.go:968
github.com/yugabyte/yb-voyager/yb-voyager/cmd.Execute()
	/home/ubuntu/yb-voyager/yb-voyager/cmd/root.go:208 +0x1a
main.main()
	/home/ubuntu/yb-voyager/yb-voyager/main.go:42 +0x45

The above panic switches to:

yb-voyager export data status -e export-dir/
Export Data Status for SourceDB

TABLE     	STATUS     	EXPORTED ROWS
CATEGORIES	NOT-STARTED	0            
CUSTOMERS 	NOT-STARTED	0            
CUSTOMERS1	NOT-STARTED	0            
CUSTOMERS2	NOT-STARTED	0            
CUSTOMERS3	NOT-STARTED	0            
CUST_HIST 	NOT-STARTED	0            
INVENTORY 	NOT-STARTED	0            
ORDERLINES	NOT-STARTED	0            
ORDERS    	NOT-STARTED	0            
PRODUCTS  	NOT-STARTED	0             
      

Once we start seeing the progress bars:

yb-voyager export data status -e export-dir/
Export Data Status for SourceDB

TABLE     	STATUS     	EXPORTED ROWS
CUST_HIST 	NOT-STARTED	0            
INVENTORY 	NOT-STARTED	0            
ORDERLINES	NOT-STARTED	0            
ORDERS    	NOT-STARTED	0            
PRODUCTS  	NOT-STARTED	0            
CUSTOMERS 	EXPORTING  	210000       
CUSTOMERS1	EXPORTING  	190000       
CUSTOMERS2	EXPORTING  	210000       
CUSTOMERS3	EXPORTING  	210000       
CATEGORIES	DONE       	16       

Could you confirm if you're observing the same behavior? Does the panic state persist, or does it resolve shortly after the export fully starts, given that these steps are executed in quick succession? We'll investigate both stack traces and work on addressing the panics.
Also what's the source database type in your case? The above example is with Oracle as the source.

[hbhanawat]

Target release for the Voyager fix: 1.8.6 (Tentative date: 26th November).

[jlisthood]

Could you confirm if you're observing the same behavior? Does the panic state persist, or does it resolve shortly after the export fully starts, given that these steps are executed in quick succession? We'll investigate both stack traces and work on addressing the panics.
Also what's the source database type in your case? The above example is with Oracle as the source

The panic state persisted while the export was making progress for many minutes. I left it be and after the export was fully done, I could run the command without panic

Also what's the source database type in your case? The above example is with Oracle as the source.
Postgres v13 in this case

[priyanshi-yb]

Hi @jlisthood , As mentioned above we are already fixing the issue and will be released by EOM.
Meanwhile, if you face this issue again, could you please share the contents of this file export-dir/metainfo/export_snapshot_status.json when you face the panic? It will help us see if there is anything else that is an issue.
Thanks!