About accuracy

[daihu-ye]

In your paper , you said you run the experiment three times,did you use different random seed,or you just use seed 1993 and run 3 times?I found that different random seed may affect the final accuracy.So i want to know your configuration for fair comparison.

[yaoyao-liu]

Thanks for your interest in our work.

We use the same seed (1993) to generate the class order and use different seeds to run the following parts of the experiments.

[daihu-ye]

I don't understand.The seed is used to generate the class order in your code,3 different seeds mean 3 class order,right?
I just run
python main.py --nb_cl_fg=50 --nb_cl=10 --gpu=0 --random_seed=1993 --baseline=lucir --branch_mode=dual --branch_1=ss --branch_2=free --dataset=cifar100
python main.py --nb_cl_fg=50 --nb_cl=10 --gpu=0 --random_seed=1994 --baseline=lucir --branch_mode=dual --branch_1=ss --branch_2=free --dataset=cifar100
python main.py --nb_cl_fg=50 --nb_cl=10 --gpu=0 --random_seed=1996 --baseline=lucir --branch_mode=dual --branch_1=ss --branch_2=free --dataset=cifar100
Then I calulate averages ± standard deviations.The results is the final average accuracy when N=5. Is it right?
I also did some experiments and found that different seeds may affect the final accuracy(sometimes more than 1%).
And in icarl experiments,with AANets,i can't get the same results as your paper (your paper report 64.22 when N=5),but i only got 62.

[yaoyao-liu]

You need to edit this function to run experiments with the same class order and different random seeds:

class-incremental-learning/adaptive-aggregation-networks/trainer/base_trainer.py

Lines 214 to 241 in 834de46

	def init_class_order(self):
	"""The function to initialize the class order.
	Returns:
	order: an array for the class order
	order_list: a list for the class order
	"""
	# Set the random seed according to the config
	np.random.seed(self.args.random_seed)
	# Set the name for the class order file
	order_name = osp.join(self.save_path, "seed_{}_{}_order.pkl".format(self.args.random_seed, self.args.dataset))
	# Print the name for the class order file
	print("Order name:{}".format(order_name))
	if osp.exists(order_name):
	# If we have already generated the class order file, load it
	print("Loading the saved class order")
	order = utils.misc.unpickle(order_name)
	else:
	# If we don't have the class order file, generate a new one
	print("Generating a new class order")
	order = np.arange(self.args.num_classes)
	np.random.shuffle(order)
	utils.misc.savepickle(order, order_name)
	# Transfer the array to a list
	order_list = list(order)
	# Print the class order
	print(order_list)
	return order, order_list

I'll check the results of iCaRL+AANets. Could you please send me the command you use to run iCaRL+AANets?

[daihu-ye]

Thanks for your explanation.Just edit the path of the class order file,let it lead to the same class order file (seed 1993 generate ).
It seems that the random seed is only used for selecting the exemplars in
https://github.com/yaoyao-liu/class-incremental-learning/blob/main/adaptive-aggregation-networks/trainer/base_trainer.py
in function gen_balanced_loader
the_idx = np.random.randint(0,len(X_train_this_step),size=self.args.nb_cl*self.args.nb_protos)
if we already get the class order file ,right?

[yaoyao-liu]

I think the random seed will influence many steps in the following code.

[daihu-ye]

Maybe i should run the experiments again.The result is actually got by 3 class orders ,it may influence the final accuracy.
Thanks for your response.